# Tracing enable/disable runbook for K8s components

## Context

The pod-creation tracing plan ([plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md](../../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md)) covers the *one-time setup* of OTLP tracing across kube-apiserver and kubelet, plus the Jaeger backend on docker-29. What's missing is a short operational doc for **toggling tracing on/off as quickly as possible** on the running cluster — useful for short measurement experiments where we want overhead only when we're actively recording.

The Terraform/cloud-init changes already in flight (`master.tf`, `files/manifests/kube-apiserver.yaml`, `files/tracing-config.yaml`) cover *rebuilds*. This runbook covers the *live* cluster: kube-master-31 + kube-node-32 + kube-node-33, K8s 1.32.

Output of this plan: a single new file `docs/kubernetes-tracing.md` containing three runbooks (quick suppress, full enable, full disable) and a verification section. No code or manifest changes — pure documentation.

## What "as quickly as possible" means

There are two speeds of toggling:

1. **Quick suppress** — leave the tracing flags/config in place on the components; just stop Jaeger so span exports go nowhere. No K8s component restart, takes seconds, but components still pay the (small) cost of building spans and attempting to export.
2. **Full enable / full disable** — edit configs and restart components. apiserver static-pod restart is automatic via kubelet manifest watch (~5–10s); kubelet itself needs `systemctl restart kubelet` on each node (brief NotReady blip per node, no pod evictions). This is what we'd use to truly remove overhead between experiments.

The runbook will present both, with quick-suppress as the default for "I'm done measuring for now" and full-disable for "we don't need this on the cluster anymore".

## Deliverable: `docs/kubernetes-tracing.md`

### Structure

```
# Kubernetes component tracing

## Components and where the toggles live
- table: component | config file on node | restart trigger
  - kube-apiserver | /etc/kubernetes/manifests/kube-apiserver.yaml (flag) +
                     /etc/kubernetes/tracing-config.yaml (config) | kubelet auto-restarts static pod
  - kubelet         | /var/lib/kubelet/config.yaml (`tracing:` block) +
                     kube-system/kubelet-config CM (cluster-wide source of truth) | systemctl restart kubelet

## Quick suppress (no component restart)
1. `ssh novakj@192.168.0.29 'cd /path/to/tracing && docker compose stop jaeger'`
2. Components continue running with tracing wired up but spans go nowhere.
3. To resume: `docker compose start jaeger`.
Caveat: components still build spans and attempt OTLP export — small CPU/network overhead. For a real "off", use full disable.

## Full enable (live cluster)

Prereqs: Jaeger is up on docker-29 (`docker compose up -d` in `docker-29/tracing/`).

### kube-apiserver (master only)
On 192.168.0.31:
```bash
# 1. Drop the TracingConfiguration file (same content as files/tracing-config.yaml)
sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'EOF'
apiVersion: apiserver.config.k8s.io/v1beta1
kind: TracingConfiguration
endpoint: 192.168.0.29:4317
samplingRatePerMillion: 1000000
EOF

# 2. Back up the apiserver manifest
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing

# 3. Add the flag (insert after --tls-private-key-file line)
sudo sed -i '/--tls-private-key-file=/a\    - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml

# 4. Wait for kubelet to restart the static pod (~5-10s)
sudo crictl ps | grep apiserver         # new container id == restarted
kubectl get --raw /livez                 # should return "ok"
```

### kubelet (every node: master + workers)

Per-node edit:
```bash
# Append the tracing block under kubelet config (idempotent: check first)
grep -q '^tracing:' /var/lib/kubelet/config.yaml || sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'EOF'
tracing:
  endpoint: 192.168.0.29:4317
  samplingRatePerMillion: 1000000
EOF

sudo systemctl restart kubelet
```

Stagger across nodes (one at a time) so we don't NotReady the whole cluster at once.

Then make it survive node rebuilds/joins by updating the cluster ConfigMap:
```bash
kubectl -n kube-system edit cm kubelet-config
# add the same `tracing:` block under the `kubelet:` key
```

## Full disable (live cluster)

### kube-apiserver
```bash
# Restore the pre-tracing manifest (preferred — exact revert)
sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
# OR remove just the flag in place:
sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml

# Optional: remove the now-unused config
sudo rm /etc/kubernetes/tracing-config.yaml
```
kubelet restarts the static pod automatically.

### kubelet (every node)
```bash
sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
```
Then strip the `tracing:` block from `kube-system/kubelet-config` CM so node rejoins don't reintroduce it.

## Verification (after enable)
1. Open `http://192.168.0.29:16686` — Services dropdown shows `apiserver` (appears after the first traced request) and `kubelet`.
2. Generate end-to-end traffic:
   ```bash
   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   ```
3. In Jaeger: filter service `apiserver`, look for spans tagged with the probe pod name; filter service `kubelet`, look for `syncPod` spans.
4. To verify *disable*: after running disable steps, repeat (2). Within a minute, no new spans for `apiserver`/`kubelet` should appear in Jaeger.

## Notes
- 100% sampling is fine for short experiments; for steady-state leave tracing off (quick suppress is enough between sessions).
- The flag/config are GA in K8s 1.32 — no `--feature-gates` needed.
- This runbook only covers apiserver + kubelet. etcd and kube-scheduler tracing are documented as future steps in the parent plan.
```

## Files to change

| File | Change |
|---|---|
| `docs/kubernetes-tracing.md` | **new** — the runbook described above |

No other files modified. The Terraform/cloud-init side stays as already drafted in the parent plan.

## Verification of this plan's output

After the runbook is written:

1. Skim `docs/kubernetes-tracing.md` end-to-end and check that someone unfamiliar with the setup can follow it without opening the parent plan.
2. Cross-check the sed/tee commands against the actual files on the live master via SSH — confirm line patterns match (e.g., the `--tls-private-key-file=` line is present so the sed insertion lands in the right spot).
3. Dry-run the quick-suppress flow first (it's the lowest-risk path) and confirm Jaeger UI stops receiving new spans within ~30 s.