# Kubernetes component tracing

OpenTelemetry tracing for kube-apiserver and kubelet, shipping OTLP spans to Jaeger on docker-29. Useful for measuring pod-creation latency broken down by component.

Related: [plans/k8s-pod-creation-tracing.md](../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md) — setup rationale, storage sizing, etcd tracing notes.

---

## Where the toggles live

| Component | Config on node | Restart trigger |
|---|---|---|
| kube-apiserver | `/etc/kubernetes/manifests/kube-apiserver.yaml` (flag) + `/etc/kubernetes/tracing-config.yaml` | kubelet detects manifest change, auto-restarts static pod (~5–10 s) |
| kubelet | `/var/lib/kubelet/config.yaml` (`tracing:` block) + `kube-system/kubelet-config` CM | `systemctl restart kubelet` |

Nodes: kube-master-31 @ 192.168.0.31, kube-node-32 @ 192.168.0.32, kube-node-33 @ 192.168.0.33.

---

## Quick suppress (no component restart)

Stop the Jaeger backend — components keep building spans but have nowhere to send them. Lowest-risk; use this between measurement sessions.

```bash
ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose stop jaeger'
```

Resume:

```bash
ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose start jaeger'
```

Caveat: components still pay the (small) cost of building spans. For true zero-overhead, use **Full disable** below.

---

## Full enable (live cluster)

### 1. Start Jaeger on docker-30

```bash
ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose up -d'
```

### 2. kube-apiserver (master only — 192.168.0.31)

```bash
ssh novakj@192.168.0.31 bash -s <<'EOF'
# TracingConfiguration file
sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'YAML'
apiVersion: apiserver.config.k8s.io/v1beta1
kind: TracingConfiguration
endpoint: 192.168.0.29:4317
samplingRatePerMillion: 1000000
YAML

# Back up the manifest before touching it
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing

# Add the flag (after --tls-private-key-file line)
sudo sed -i '/--tls-private-key-file=/a\    - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' \
    /etc/kubernetes/manifests/kube-apiserver.yaml

# Wait for static pod to restart
sleep 10
sudo crictl ps | grep apiserver
kubectl get --raw /livez
EOF
```

### 3. kubelet (all three nodes)

Run on each node in turn — wait for `Ready` before moving to the next to avoid simultaneous NotReady:

```bash
for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
  echo "==> $NODE"
  ssh ubuntu@$NODE bash -s <<'EOF'
grep -q '^tracing:' /var/lib/kubelet/config.yaml && echo "already enabled, skipping" && exit 0
sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'YAML'
tracing:
  endpoint: 192.168.0.29:4317
  samplingRatePerMillion: 1000000
YAML
sudo systemctl restart kubelet
EOF
  # wait for node Ready before proceeding
  kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
done
```

### 4. Update the cluster ConfigMap (so future joins/reboots pick it up)

```bash
kubectl -n kube-system edit cm kubelet-config
# under the `kubelet:` key, add:
#   tracing:
#     endpoint: 192.168.0.29:4317
#     samplingRatePerMillion: 1000000
```

---

## Full disable (live cluster)

### kube-apiserver (master only)

```bash
ssh novakj@192.168.0.31 bash -s <<'EOF'
# Restore the pre-tracing manifest (exact revert)
sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
# Optional: remove the config file
sudo rm -f /etc/kubernetes/tracing-config.yaml
EOF
```

Static pod restarts automatically. If the backup wasn't taken, remove the flag in-place instead:

```bash
sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml
```

### kubelet (all three nodes)

```bash
for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
  echo "==> $NODE"
  ssh novakj@$NODE bash -s <<'EOF'
sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
EOF
  kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
done
```

Remove the tracing block from the cluster ConfigMap too:

```bash
kubectl -n kube-system edit cm kubelet-config
# remove the `tracing:` block under `kubelet:`
```

---

## Verification

### After enable

1. Open Jaeger UI: `http://192.168.0.29:16686` — Services dropdown should show `apiserver` and `kubelet` (appear after first traced request).

2. Generate a full pod-lifecycle trace:

   ```bash
   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   ```

3. In Jaeger UI, search:
   - Service `apiserver` — look for spans covering the create/watch flow for `trace-probe`.
   - Service `kubelet` — look for `syncPod` spans tied to the same pod.

4. The gap between apiserver spans and kubelet spans is scheduler time (kube-scheduler has no OTLP tracing in 1.32). Cross-check with `kubectl get events` or pod condition timestamps.

### After disable

Repeat step 2 above, then confirm no new spans appear in Jaeger for `apiserver` or `kubelet` within ~1 minute.

---

## Sampling rate cheatsheet

Change `samplingRatePerMillion` in `/etc/kubernetes/tracing-config.yaml` (apiserver) and `/var/lib/kubelet/config.yaml` (kubelet) then restart each component.

| Value | Effective rate | Use case |
|---|---|---|
| `1000000` | 100% | Active measurement |
| `10000` | 1% | Steady-state background |
| `0` | off | Disable without removing config |