Jan Novak
d43ffd488e
Wire kube-apiserver and kubelet tracing to a Jaeger collector on docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and stand up Mimir + Loki on docker-30 as their backing stores. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
141 lines
6.8 KiB
Markdown
141 lines
6.8 KiB
Markdown
# Tracing enable/disable runbook for K8s components
|
||
|
||
## Context
|
||
|
||
The pod-creation tracing plan ([plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md](../../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md)) covers the *one-time setup* of OTLP tracing across kube-apiserver and kubelet, plus the Jaeger backend on docker-29. What's missing is a short operational doc for **toggling tracing on/off as quickly as possible** on the running cluster — useful for short measurement experiments where we want overhead only when we're actively recording.
|
||
|
||
The Terraform/cloud-init changes already in flight (`master.tf`, `files/manifests/kube-apiserver.yaml`, `files/tracing-config.yaml`) cover *rebuilds*. This runbook covers the *live* cluster: kube-master-31 + kube-node-32 + kube-node-33, K8s 1.32.
|
||
|
||
Output of this plan: a single new file `docs/kubernetes-tracing.md` containing three runbooks (quick suppress, full enable, full disable) and a verification section. No code or manifest changes — pure documentation.
|
||
|
||
## What "as quickly as possible" means
|
||
|
||
There are two speeds of toggling:
|
||
|
||
1. **Quick suppress** — leave the tracing flags/config in place on the components; just stop Jaeger so span exports go nowhere. No K8s component restart, takes seconds, but components still pay the (small) cost of building spans and attempting to export.
|
||
2. **Full enable / full disable** — edit configs and restart components. apiserver static-pod restart is automatic via kubelet manifest watch (~5–10s); kubelet itself needs `systemctl restart kubelet` on each node (brief NotReady blip per node, no pod evictions). This is what we'd use to truly remove overhead between experiments.
|
||
|
||
The runbook will present both, with quick-suppress as the default for "I'm done measuring for now" and full-disable for "we don't need this on the cluster anymore".
|
||
|
||
## Deliverable: `docs/kubernetes-tracing.md`
|
||
|
||
### Structure
|
||
|
||
```
|
||
# Kubernetes component tracing
|
||
|
||
## Components and where the toggles live
|
||
- table: component | config file on node | restart trigger
|
||
- kube-apiserver | /etc/kubernetes/manifests/kube-apiserver.yaml (flag) +
|
||
/etc/kubernetes/tracing-config.yaml (config) | kubelet auto-restarts static pod
|
||
- kubelet | /var/lib/kubelet/config.yaml (`tracing:` block) +
|
||
kube-system/kubelet-config CM (cluster-wide source of truth) | systemctl restart kubelet
|
||
|
||
## Quick suppress (no component restart)
|
||
1. `ssh novakj@192.168.0.29 'cd /path/to/tracing && docker compose stop jaeger'`
|
||
2. Components continue running with tracing wired up but spans go nowhere.
|
||
3. To resume: `docker compose start jaeger`.
|
||
Caveat: components still build spans and attempt OTLP export — small CPU/network overhead. For a real "off", use full disable.
|
||
|
||
## Full enable (live cluster)
|
||
|
||
Prereqs: Jaeger is up on docker-29 (`docker compose up -d` in `docker-29/tracing/`).
|
||
|
||
### kube-apiserver (master only)
|
||
On 192.168.0.31:
|
||
```bash
|
||
# 1. Drop the TracingConfiguration file (same content as files/tracing-config.yaml)
|
||
sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'EOF'
|
||
apiVersion: apiserver.config.k8s.io/v1beta1
|
||
kind: TracingConfiguration
|
||
endpoint: 192.168.0.29:4317
|
||
samplingRatePerMillion: 1000000
|
||
EOF
|
||
|
||
# 2. Back up the apiserver manifest
|
||
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing
|
||
|
||
# 3. Add the flag (insert after --tls-private-key-file line)
|
||
sudo sed -i '/--tls-private-key-file=/a\ - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml
|
||
|
||
# 4. Wait for kubelet to restart the static pod (~5-10s)
|
||
sudo crictl ps | grep apiserver # new container id == restarted
|
||
kubectl get --raw /livez # should return "ok"
|
||
```
|
||
|
||
### kubelet (every node: master + workers)
|
||
|
||
Per-node edit:
|
||
```bash
|
||
# Append the tracing block under kubelet config (idempotent: check first)
|
||
grep -q '^tracing:' /var/lib/kubelet/config.yaml || sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'EOF'
|
||
tracing:
|
||
endpoint: 192.168.0.29:4317
|
||
samplingRatePerMillion: 1000000
|
||
EOF
|
||
|
||
sudo systemctl restart kubelet
|
||
```
|
||
|
||
Stagger across nodes (one at a time) so we don't NotReady the whole cluster at once.
|
||
|
||
Then make it survive node rebuilds/joins by updating the cluster ConfigMap:
|
||
```bash
|
||
kubectl -n kube-system edit cm kubelet-config
|
||
# add the same `tracing:` block under the `kubelet:` key
|
||
```
|
||
|
||
## Full disable (live cluster)
|
||
|
||
### kube-apiserver
|
||
```bash
|
||
# Restore the pre-tracing manifest (preferred — exact revert)
|
||
sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
|
||
# OR remove just the flag in place:
|
||
sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml
|
||
|
||
# Optional: remove the now-unused config
|
||
sudo rm /etc/kubernetes/tracing-config.yaml
|
||
```
|
||
kubelet restarts the static pod automatically.
|
||
|
||
### kubelet (every node)
|
||
```bash
|
||
sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
|
||
sudo systemctl restart kubelet
|
||
```
|
||
Then strip the `tracing:` block from `kube-system/kubelet-config` CM so node rejoins don't reintroduce it.
|
||
|
||
## Verification (after enable)
|
||
1. Open `http://192.168.0.29:16686` — Services dropdown shows `apiserver` (appears after the first traced request) and `kubelet`.
|
||
2. Generate end-to-end traffic:
|
||
```bash
|
||
kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
|
||
kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
|
||
kubectl delete pod trace-probe
|
||
```
|
||
3. In Jaeger: filter service `apiserver`, look for spans tagged with the probe pod name; filter service `kubelet`, look for `syncPod` spans.
|
||
4. To verify *disable*: after running disable steps, repeat (2). Within a minute, no new spans for `apiserver`/`kubelet` should appear in Jaeger.
|
||
|
||
## Notes
|
||
- 100% sampling is fine for short experiments; for steady-state leave tracing off (quick suppress is enough between sessions).
|
||
- The flag/config are GA in K8s 1.32 — no `--feature-gates` needed.
|
||
- This runbook only covers apiserver + kubelet. etcd and kube-scheduler tracing are documented as future steps in the parent plan.
|
||
```
|
||
|
||
## Files to change
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `docs/kubernetes-tracing.md` | **new** — the runbook described above |
|
||
|
||
No other files modified. The Terraform/cloud-init side stays as already drafted in the parent plan.
|
||
|
||
## Verification of this plan's output
|
||
|
||
After the runbook is written:
|
||
|
||
1. Skim `docs/kubernetes-tracing.md` end-to-end and check that someone unfamiliar with the setup can follow it without opening the parent plan.
|
||
2. Cross-check the sed/tee commands against the actual files on the live master via SSH — confirm line patterns match (e.g., the `--tls-private-key-file=` line is present so the sed insertion lands in the right spot).
|
||
3. Dry-run the quick-suppress flow first (it's the lowest-risk path) and confirm Jaeger UI stops receiving new spans within ~30 s.
|