observability: add k8s API/kubelet tracing, Alloy, Mimir and Loki
Wire kube-apiserver and kubelet tracing to a Jaeger collector on docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and stand up Mimir + Loki on docker-30 as their backing stores. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
180
docs/kubernetes-tracing.md
Normal file
180
docs/kubernetes-tracing.md
Normal file
@@ -0,0 +1,180 @@
|
||||
# Kubernetes component tracing
|
||||
|
||||
OpenTelemetry tracing for kube-apiserver and kubelet, shipping OTLP spans to Jaeger on docker-29. Useful for measuring pod-creation latency broken down by component.
|
||||
|
||||
Related: [plans/k8s-pod-creation-tracing.md](../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md) — setup rationale, storage sizing, etcd tracing notes.
|
||||
|
||||
---
|
||||
|
||||
## Where the toggles live
|
||||
|
||||
| Component | Config on node | Restart trigger |
|
||||
|---|---|---|
|
||||
| kube-apiserver | `/etc/kubernetes/manifests/kube-apiserver.yaml` (flag) + `/etc/kubernetes/tracing-config.yaml` | kubelet detects manifest change, auto-restarts static pod (~5–10 s) |
|
||||
| kubelet | `/var/lib/kubelet/config.yaml` (`tracing:` block) + `kube-system/kubelet-config` CM | `systemctl restart kubelet` |
|
||||
|
||||
Nodes: kube-master-31 @ 192.168.0.31, kube-node-32 @ 192.168.0.32, kube-node-33 @ 192.168.0.33.
|
||||
|
||||
---
|
||||
|
||||
## Quick suppress (no component restart)
|
||||
|
||||
Stop the Jaeger backend — components keep building spans but have nowhere to send them. Lowest-risk; use this between measurement sessions.
|
||||
|
||||
```bash
|
||||
ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose stop jaeger'
|
||||
```
|
||||
|
||||
Resume:
|
||||
|
||||
```bash
|
||||
ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose start jaeger'
|
||||
```
|
||||
|
||||
Caveat: components still pay the (small) cost of building spans. For true zero-overhead, use **Full disable** below.
|
||||
|
||||
---
|
||||
|
||||
## Full enable (live cluster)
|
||||
|
||||
### 1. Start Jaeger on docker-30
|
||||
|
||||
```bash
|
||||
ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose up -d'
|
||||
```
|
||||
|
||||
### 2. kube-apiserver (master only — 192.168.0.31)
|
||||
|
||||
```bash
|
||||
ssh novakj@192.168.0.31 bash -s <<'EOF'
|
||||
# TracingConfiguration file
|
||||
sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'YAML'
|
||||
apiVersion: apiserver.config.k8s.io/v1beta1
|
||||
kind: TracingConfiguration
|
||||
endpoint: 192.168.0.29:4317
|
||||
samplingRatePerMillion: 1000000
|
||||
YAML
|
||||
|
||||
# Back up the manifest before touching it
|
||||
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing
|
||||
|
||||
# Add the flag (after --tls-private-key-file line)
|
||||
sudo sed -i '/--tls-private-key-file=/a\ - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' \
|
||||
/etc/kubernetes/manifests/kube-apiserver.yaml
|
||||
|
||||
# Wait for static pod to restart
|
||||
sleep 10
|
||||
sudo crictl ps | grep apiserver
|
||||
kubectl get --raw /livez
|
||||
EOF
|
||||
```
|
||||
|
||||
### 3. kubelet (all three nodes)
|
||||
|
||||
Run on each node in turn — wait for `Ready` before moving to the next to avoid simultaneous NotReady:
|
||||
|
||||
```bash
|
||||
for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
|
||||
echo "==> $NODE"
|
||||
ssh ubuntu@$NODE bash -s <<'EOF'
|
||||
grep -q '^tracing:' /var/lib/kubelet/config.yaml && echo "already enabled, skipping" && exit 0
|
||||
sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'YAML'
|
||||
tracing:
|
||||
endpoint: 192.168.0.29:4317
|
||||
samplingRatePerMillion: 1000000
|
||||
YAML
|
||||
sudo systemctl restart kubelet
|
||||
EOF
|
||||
# wait for node Ready before proceeding
|
||||
kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
|
||||
done
|
||||
```
|
||||
|
||||
### 4. Update the cluster ConfigMap (so future joins/reboots pick it up)
|
||||
|
||||
```bash
|
||||
kubectl -n kube-system edit cm kubelet-config
|
||||
# under the `kubelet:` key, add:
|
||||
# tracing:
|
||||
# endpoint: 192.168.0.29:4317
|
||||
# samplingRatePerMillion: 1000000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full disable (live cluster)
|
||||
|
||||
### kube-apiserver (master only)
|
||||
|
||||
```bash
|
||||
ssh novakj@192.168.0.31 bash -s <<'EOF'
|
||||
# Restore the pre-tracing manifest (exact revert)
|
||||
sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
|
||||
# Optional: remove the config file
|
||||
sudo rm -f /etc/kubernetes/tracing-config.yaml
|
||||
EOF
|
||||
```
|
||||
|
||||
Static pod restarts automatically. If the backup wasn't taken, remove the flag in-place instead:
|
||||
|
||||
```bash
|
||||
sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml
|
||||
```
|
||||
|
||||
### kubelet (all three nodes)
|
||||
|
||||
```bash
|
||||
for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
|
||||
echo "==> $NODE"
|
||||
ssh novakj@$NODE bash -s <<'EOF'
|
||||
sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
|
||||
sudo systemctl restart kubelet
|
||||
EOF
|
||||
kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
|
||||
done
|
||||
```
|
||||
|
||||
Remove the tracing block from the cluster ConfigMap too:
|
||||
|
||||
```bash
|
||||
kubectl -n kube-system edit cm kubelet-config
|
||||
# remove the `tracing:` block under `kubelet:`
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
### After enable
|
||||
|
||||
1. Open Jaeger UI: `http://192.168.0.29:16686` — Services dropdown should show `apiserver` and `kubelet` (appear after first traced request).
|
||||
|
||||
2. Generate a full pod-lifecycle trace:
|
||||
|
||||
```bash
|
||||
kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
|
||||
kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
|
||||
kubectl delete pod trace-probe
|
||||
```
|
||||
|
||||
3. In Jaeger UI, search:
|
||||
- Service `apiserver` — look for spans covering the create/watch flow for `trace-probe`.
|
||||
- Service `kubelet` — look for `syncPod` spans tied to the same pod.
|
||||
|
||||
4. The gap between apiserver spans and kubelet spans is scheduler time (kube-scheduler has no OTLP tracing in 1.32). Cross-check with `kubectl get events` or pod condition timestamps.
|
||||
|
||||
### After disable
|
||||
|
||||
Repeat step 2 above, then confirm no new spans appear in Jaeger for `apiserver` or `kubelet` within ~1 minute.
|
||||
|
||||
---
|
||||
|
||||
## Sampling rate cheatsheet
|
||||
|
||||
Change `samplingRatePerMillion` in `/etc/kubernetes/tracing-config.yaml` (apiserver) and `/var/lib/kubelet/config.yaml` (kubelet) then restart each component.
|
||||
|
||||
| Value | Effective rate | Use case |
|
||||
|---|---|---|
|
||||
| `1000000` | 100% | Active measurement |
|
||||
| `10000` | 1% | Steady-state background |
|
||||
| `0` | off | Disable without removing config |
|
||||
Reference in New Issue
Block a user