home-kubernetes/plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md

Pod-creation latency tracing with OpenTelemetry

Context

Goal: measure end-to-end latency of kubectl apply → pod running, broken down by component. The approach is to enable native OpenTelemetry tracing in Kubernetes 1.32 components (kube-apiserver, kubelet, etcd) and ship spans to a Jaeger backend running outside the cluster on docker-29. With all three reporting under the same trace id (object-resource correlation), Jaeger will let us see how much time is spent in admission/etcd, on the wire, and inside the kubelet's pod sync loop.

Constraints / known gaps:

• kube-scheduler has no native OTLP tracing yet; there will be a gap between "object stored" and "kubelet sees assignment". We can infer it from event timestamps (kubectl get events) or .status.conditions PodScheduled/Initialized/ContainersReady/Ready timestamps.
• Cluster is K8s 1.32, kubeadm-bootstrapped via Terraform/cloud-init. Per Phase-3 decision, the canonical source-of-truth is the Terraform cloud-init; changes there cover future rebuilds. To activate tracing on the currently running nodes we also need a one-time manual application step (documented at the bottom).

Backend: Jaeger on docker-29

New directory docker-29/tracing/ containing docker-compose.yaml. Uses Jaeger all-in-one v1 with Badger on-disk storage and the built-in OTLP receiver enabled.

# docker-29/tracing/docker-compose.yaml
services:
  jaeger:
    image: jaegertracing/all-in-one:1.62
    container_name: jaeger
    restart: unless-stopped
    environment:
      SPAN_STORAGE_TYPE: badger
      BADGER_EPHEMERAL: "false"
      BADGER_DIRECTORY_VALUE: /badger/data
      BADGER_DIRECTORY_KEY: /badger/key
      BADGER_SPAN_STORE_TTL: 4h       # auto-expire traces older than 4h
      BADGER_MAINTENANCE_INTERVAL: 5m # GC cycle that reclaims disk
      COLLECTOR_OTLP_ENABLED: "true"
    volumes:
      - ./badger:/badger
    ports:
      - "4317:4317"    # OTLP gRPC (apiserver + kubelet exporters)
      - "4318:4318"    # OTLP HTTP (optional)
      - "16686:16686"  # Jaeger UI

Endpoint that the cluster will send to: 192.168.0.29:4317. Jaeger UI: http://192.168.0.29:16686.

The OTLP receiver in Jaeger all-in-one accepts plaintext OTLP/gRPC by default — matches endpoint: ...:4317 on the K8s side with no TLS. Acceptable for a home lab on a private LAN.

Storage sizing and retention

Two complementary levers control how much disk Badger uses:

1. TTL — BADGER_SPAN_STORE_TTL=4h deletes spans older than 4 hours; Badger's value-log GC reclaims disk on BADGER_MAINTENANCE_INTERVAL (default 5m). This is the hard upper bound on retention.
2. Sampling rate — controls how many spans get written in the first place. Set in the apiserver/kubelet TracingConfiguration (samplingRatePerMillion).

Rough estimate for this cluster (1 master + 2 workers, mostly idle, Flux reconciling continuously):

Sampling	Span ingest rate	Disk footprint at 4h TTL
100% (1000000)	~500–3000 spans/s	~6–40 GB steady-state
1% (10000)	~5–30 spans/s	~60–400 MB steady-state
0.1% (1000)	~0.5–3 spans/s	~6–40 MB steady-state

Note: 100% sampling on this cluster is sized for short experiments, not long-term operation. The 4h TTL prevents runaway growth if I forget to lower it, but Badger has no built-in size cap — only TTL + GC. If a hard size cap is needed, mount ./badger/ from a fixed-size LVM volume or zfs dataset with a quota.

Recommended progression:

• Start with samplingRatePerMillion: 1000000 (100%) and TTL=4h so the first measurements are easy to find.
• After the first round of experiments, drop sampling to 10000 (1%) for steady-state and adjust TTL upward if a longer history is useful.

kube-apiserver tracing

Two changes, both in the Terraform cloud-init / referenced manifest:

1. Write the TracingConfiguration file (master.tf cloud-init)

Add to write_files in kubernetes-kvm-terraform/master.tf:

- path: /etc/kubernetes/tracing-config.yaml
  content: |
    apiVersion: apiserver.config.k8s.io/v1beta1
    kind: TracingConfiguration
    endpoint: 192.168.0.29:4317
    samplingRatePerMillion: 1000000  # 100% sampling for measurement work

The apiserver static pod already mounts /etc/kubernetes read-only via the k8s-config volume, so the file becomes visible inside the container automatically — no new volume mount needed.

2. Add the apiserver flag

Edit kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml — add one new flag in the command: list (alphabetical placement after --tls-private-key-file works; ordering isn't enforced):

- --tracing-config-file=/etc/kubernetes/tracing-config.yaml

APIServerTracing is GA in 1.32 — no feature-gate flag required.

kubelet tracing

Add a tracing: block to the existing KubeletConfiguration document inside the kubeadm-config heredoc in kubernetes-kvm-terraform/master.tf (around line 158):

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
tracing:
  endpoint: 192.168.0.29:4317
  samplingRatePerMillion: 1000000

KubeletTracing is beta and on by default in 1.32 — no feature gate needed.

Kubeadm stores this KubeletConfiguration in the kubelet-config ConfigMap in kube-system, so workers joining via kubeadm join pick the tracing block up automatically. The two existing workers (kube-node-32, kube-node-33) need a one-time manual nudge (see below) because they were already joined before the change.

etcd tracing (optional, second-step)

etcd supports OTLP via flags --experimental-enable-distributed-tracing, --experimental-distributed-tracing-address=192.168.0.29:4317, and --experimental-distributed-tracing-sampling-rate=1000000. Enabling these requires editing the etcd static-pod manifest on the master — propose deferring until after the apiserver+kubelet path is verified, so we keep the first iteration narrow.

Files to change

File	Change
docker-29/tracing/docker-compose.yaml	new — Jaeger all-in-one + Badger
kubernetes-kvm-terraform/master.tf	add tracing-config.yaml to write_files; add tracing: block to KubeletConfiguration
kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml	add --tracing-config-file=/etc/kubernetes/tracing-config.yaml
kubernetes-kvm-terraform/files/tracing-config.yaml	new — reference copy of the TracingConfiguration (mirrors the inline content in master.tf, matches the existing auth-config.yaml convention)

No changes to nodes-on-homer.tf / nodes-on-beelink.tf are needed — workers receive kubelet config from the cluster ConfigMap on join.

Applying to the currently running cluster

The Terraform changes cover future rebuilds. To activate tracing now without rebuilding:

1. Start Jaeger on docker-29: cd docker-29/tracing && docker compose up -d (creates ./badger/ for storage).
2. On the master (kube-master-31, 192.168.0.31):
   • scp or write /etc/kubernetes/tracing-config.yaml with the TracingConfiguration shown above.
   • Edit /etc/kubernetes/manifests/kube-apiserver.yaml to add --tracing-config-file=/etc/kubernetes/tracing-config.yaml. Kubelet will detect the manifest change and restart the apiserver static pod within seconds.
   • Watch: crictl ps | grep apiserver, then kubectl get --raw /livez.
3. Update the kubelet-config ConfigMap in kube-system:
   • kubectl -n kube-system edit cm kubelet-config and add the same tracing: block under kubelet:.
4. On every node (master + both workers): edit /var/lib/kubelet/config.yaml to append the tracing: block, then systemctl restart kubelet. Pods will not be evicted by a kubelet restart, but expect a brief NotReady blip per node — stagger the restarts.

Verification

End-to-end test once everything is live:

1. UI reachable: open http://192.168.0.29:16686 and check "Services" dropdown populates with apiserver (it will appear only after the first traced request).
2. Generate traffic that exercises the full path:

   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   

3. In Jaeger UI, search service apiserver, operation KubernetesAPI (or similar) — you should see a trace covering admission/etcd write. Search service kubelet, operation syncPod — kubelet-side pod sync spans should be visible.
4. Cross-check the gap (scheduler) by comparing pod metadata.creationTimestamp, the PodScheduled condition lastTransitionTime, and the kubelet span start. The diff between those values quantifies the scheduler portion that we cannot trace natively.
5. Confirm Badger persistence: docker compose restart jaeger, reload UI — earlier traces should still be present.

Notes / risks

• 100% sampling rate is fine for measurement; once we're done we should drop to e.g. samplingRatePerMillion: 10000 (1%) so storage doesn't explode.
• Badger volume ./badger/ on docker-29 will grow with span volume; size budget worth watching but not urgent.
• tracing.endpoint is plaintext OTLP/gRPC and traverses the LAN — fine for a home lab but explicitly not encrypted.
• If apiserver fails to come back after the manifest edit, the most common cause is a typo in the new flag; revert is straightforward — keep a copy of the original /etc/kubernetes/manifests/kube-apiserver.yaml in /root/ before editing.