Jan Novak
d43ffd488e
Wire kube-apiserver and kubelet tracing to a Jaeger collector on docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and stand up Mimir + Loki on docker-30 as their backing stores. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
181 lines
5.5 KiB
Markdown
181 lines
5.5 KiB
Markdown
# Kubernetes component tracing
|
||
|
||
OpenTelemetry tracing for kube-apiserver and kubelet, shipping OTLP spans to Jaeger on docker-29. Useful for measuring pod-creation latency broken down by component.
|
||
|
||
Related: [plans/k8s-pod-creation-tracing.md](../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md) — setup rationale, storage sizing, etcd tracing notes.
|
||
|
||
---
|
||
|
||
## Where the toggles live
|
||
|
||
| Component | Config on node | Restart trigger |
|
||
|---|---|---|
|
||
| kube-apiserver | `/etc/kubernetes/manifests/kube-apiserver.yaml` (flag) + `/etc/kubernetes/tracing-config.yaml` | kubelet detects manifest change, auto-restarts static pod (~5–10 s) |
|
||
| kubelet | `/var/lib/kubelet/config.yaml` (`tracing:` block) + `kube-system/kubelet-config` CM | `systemctl restart kubelet` |
|
||
|
||
Nodes: kube-master-31 @ 192.168.0.31, kube-node-32 @ 192.168.0.32, kube-node-33 @ 192.168.0.33.
|
||
|
||
---
|
||
|
||
## Quick suppress (no component restart)
|
||
|
||
Stop the Jaeger backend — components keep building spans but have nowhere to send them. Lowest-risk; use this between measurement sessions.
|
||
|
||
```bash
|
||
ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose stop jaeger'
|
||
```
|
||
|
||
Resume:
|
||
|
||
```bash
|
||
ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose start jaeger'
|
||
```
|
||
|
||
Caveat: components still pay the (small) cost of building spans. For true zero-overhead, use **Full disable** below.
|
||
|
||
---
|
||
|
||
## Full enable (live cluster)
|
||
|
||
### 1. Start Jaeger on docker-30
|
||
|
||
```bash
|
||
ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose up -d'
|
||
```
|
||
|
||
### 2. kube-apiserver (master only — 192.168.0.31)
|
||
|
||
```bash
|
||
ssh novakj@192.168.0.31 bash -s <<'EOF'
|
||
# TracingConfiguration file
|
||
sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'YAML'
|
||
apiVersion: apiserver.config.k8s.io/v1beta1
|
||
kind: TracingConfiguration
|
||
endpoint: 192.168.0.29:4317
|
||
samplingRatePerMillion: 1000000
|
||
YAML
|
||
|
||
# Back up the manifest before touching it
|
||
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing
|
||
|
||
# Add the flag (after --tls-private-key-file line)
|
||
sudo sed -i '/--tls-private-key-file=/a\ - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' \
|
||
/etc/kubernetes/manifests/kube-apiserver.yaml
|
||
|
||
# Wait for static pod to restart
|
||
sleep 10
|
||
sudo crictl ps | grep apiserver
|
||
kubectl get --raw /livez
|
||
EOF
|
||
```
|
||
|
||
### 3. kubelet (all three nodes)
|
||
|
||
Run on each node in turn — wait for `Ready` before moving to the next to avoid simultaneous NotReady:
|
||
|
||
```bash
|
||
for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
|
||
echo "==> $NODE"
|
||
ssh ubuntu@$NODE bash -s <<'EOF'
|
||
grep -q '^tracing:' /var/lib/kubelet/config.yaml && echo "already enabled, skipping" && exit 0
|
||
sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'YAML'
|
||
tracing:
|
||
endpoint: 192.168.0.29:4317
|
||
samplingRatePerMillion: 1000000
|
||
YAML
|
||
sudo systemctl restart kubelet
|
||
EOF
|
||
# wait for node Ready before proceeding
|
||
kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
|
||
done
|
||
```
|
||
|
||
### 4. Update the cluster ConfigMap (so future joins/reboots pick it up)
|
||
|
||
```bash
|
||
kubectl -n kube-system edit cm kubelet-config
|
||
# under the `kubelet:` key, add:
|
||
# tracing:
|
||
# endpoint: 192.168.0.29:4317
|
||
# samplingRatePerMillion: 1000000
|
||
```
|
||
|
||
---
|
||
|
||
## Full disable (live cluster)
|
||
|
||
### kube-apiserver (master only)
|
||
|
||
```bash
|
||
ssh novakj@192.168.0.31 bash -s <<'EOF'
|
||
# Restore the pre-tracing manifest (exact revert)
|
||
sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
|
||
# Optional: remove the config file
|
||
sudo rm -f /etc/kubernetes/tracing-config.yaml
|
||
EOF
|
||
```
|
||
|
||
Static pod restarts automatically. If the backup wasn't taken, remove the flag in-place instead:
|
||
|
||
```bash
|
||
sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml
|
||
```
|
||
|
||
### kubelet (all three nodes)
|
||
|
||
```bash
|
||
for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
|
||
echo "==> $NODE"
|
||
ssh novakj@$NODE bash -s <<'EOF'
|
||
sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
|
||
sudo systemctl restart kubelet
|
||
EOF
|
||
kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
|
||
done
|
||
```
|
||
|
||
Remove the tracing block from the cluster ConfigMap too:
|
||
|
||
```bash
|
||
kubectl -n kube-system edit cm kubelet-config
|
||
# remove the `tracing:` block under `kubelet:`
|
||
```
|
||
|
||
---
|
||
|
||
## Verification
|
||
|
||
### After enable
|
||
|
||
1. Open Jaeger UI: `http://192.168.0.29:16686` — Services dropdown should show `apiserver` and `kubelet` (appear after first traced request).
|
||
|
||
2. Generate a full pod-lifecycle trace:
|
||
|
||
```bash
|
||
kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
|
||
kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
|
||
kubectl delete pod trace-probe
|
||
```
|
||
|
||
3. In Jaeger UI, search:
|
||
- Service `apiserver` — look for spans covering the create/watch flow for `trace-probe`.
|
||
- Service `kubelet` — look for `syncPod` spans tied to the same pod.
|
||
|
||
4. The gap between apiserver spans and kubelet spans is scheduler time (kube-scheduler has no OTLP tracing in 1.32). Cross-check with `kubectl get events` or pod condition timestamps.
|
||
|
||
### After disable
|
||
|
||
Repeat step 2 above, then confirm no new spans appear in Jaeger for `apiserver` or `kubelet` within ~1 minute.
|
||
|
||
---
|
||
|
||
## Sampling rate cheatsheet
|
||
|
||
Change `samplingRatePerMillion` in `/etc/kubernetes/tracing-config.yaml` (apiserver) and `/var/lib/kubelet/config.yaml` (kubelet) then restart each component.
|
||
|
||
| Value | Effective rate | Use case |
|
||
|---|---|---|
|
||
| `1000000` | 100% | Active measurement |
|
||
| `10000` | 1% | Steady-state background |
|
||
| `0` | off | Disable without removing config |
|