home-kubernetes/docs/kubernetes-tracing.md

Kubernetes component tracing

OpenTelemetry tracing for kube-apiserver and kubelet, shipping OTLP spans to Jaeger on docker-29. Useful for measuring pod-creation latency broken down by component.

Related: plans/k8s-pod-creation-tracing.md — setup rationale, storage sizing, etcd tracing notes.

---

Where the toggles live

Component	Config on node	Restart trigger
kube-apiserver	/etc/kubernetes/manifests/kube-apiserver.yaml (flag) + /etc/kubernetes/tracing-config.yaml	kubelet detects manifest change, auto-restarts static pod (~5–10 s)
kubelet	/var/lib/kubelet/config.yaml (tracing: block) + kube-system/kubelet-config CM	systemctl restart kubelet

Nodes: kube-master-31 @ 192.168.0.31, kube-node-32 @ 192.168.0.32, kube-node-33 @ 192.168.0.33.

---

Quick suppress (no component restart)

Stop the Jaeger backend — components keep building spans but have nowhere to send them. Lowest-risk; use this between measurement sessions.

ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose stop jaeger'

Resume:

ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose start jaeger'

Caveat: components still pay the (small) cost of building spans. For true zero-overhead, use Full disable below.

---

Full enable (live cluster)

1. Start Jaeger on docker-30

ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose up -d'

2. kube-apiserver (master only — 192.168.0.31)

ssh novakj@192.168.0.31 bash -s <<'EOF'
# TracingConfiguration file
sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'YAML'
apiVersion: apiserver.config.k8s.io/v1beta1
kind: TracingConfiguration
endpoint: 192.168.0.29:4317
samplingRatePerMillion: 1000000
YAML

# Back up the manifest before touching it
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing

# Add the flag (after --tls-private-key-file line)
sudo sed -i '/--tls-private-key-file=/a\    - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' \
    /etc/kubernetes/manifests/kube-apiserver.yaml

# Wait for static pod to restart
sleep 10
sudo crictl ps | grep apiserver
kubectl get --raw /livez
EOF

3. kubelet (all three nodes)

Run on each node in turn — wait for Ready before moving to the next to avoid simultaneous NotReady:

for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
  echo "==> $NODE"
  ssh ubuntu@$NODE bash -s <<'EOF'
grep -q '^tracing:' /var/lib/kubelet/config.yaml && echo "already enabled, skipping" && exit 0
sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'YAML'
tracing:
  endpoint: 192.168.0.29:4317
  samplingRatePerMillion: 1000000
YAML
sudo systemctl restart kubelet
EOF
  # wait for node Ready before proceeding
  kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
done

4. Update the cluster ConfigMap (so future joins/reboots pick it up)

kubectl -n kube-system edit cm kubelet-config
# under the `kubelet:` key, add:
#   tracing:
#     endpoint: 192.168.0.29:4317
#     samplingRatePerMillion: 1000000

---

Full disable (live cluster)

kube-apiserver (master only)

ssh novakj@192.168.0.31 bash -s <<'EOF'
# Restore the pre-tracing manifest (exact revert)
sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
# Optional: remove the config file
sudo rm -f /etc/kubernetes/tracing-config.yaml
EOF

Static pod restarts automatically. If the backup wasn't taken, remove the flag in-place instead:

sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml

kubelet (all three nodes)

for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
  echo "==> $NODE"
  ssh novakj@$NODE bash -s <<'EOF'
sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
EOF
  kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
done

Remove the tracing block from the cluster ConfigMap too:

kubectl -n kube-system edit cm kubelet-config
# remove the `tracing:` block under `kubelet:`

---

Verification

After enable

1. Open Jaeger UI: http://192.168.0.29:16686 — Services dropdown should show apiserver and kubelet (appear after first traced request).

2. Generate a full pod-lifecycle trace:

   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   

3. In Jaeger UI, search:

   • Service apiserver — look for spans covering the create/watch flow for trace-probe.
   • Service kubelet — look for syncPod spans tied to the same pod.

4. The gap between apiserver spans and kubelet spans is scheduler time (kube-scheduler has no OTLP tracing in 1.32). Cross-check with kubectl get events or pod condition timestamps.

After disable

Repeat step 2 above, then confirm no new spans appear in Jaeger for apiserver or kubelet within ~1 minute.

---

Sampling rate cheatsheet

Change samplingRatePerMillion in /etc/kubernetes/tracing-config.yaml (apiserver) and /var/lib/kubelet/config.yaml (kubelet) then restart each component.

Value	Effective rate	Use case
1000000	100%	Active measurement
10000	1%	Steady-state background
0	off	Disable without removing config