observability: add k8s API/kubelet tracing, Alloy, Mimir and Loki

Wire kube-apiserver and kubelet tracing to a Jaeger collector on docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and stand up Mimir + Loki on docker-30 as their backing stores. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:05:21 +02:00
parent 95355ef7a5
commit d43ffd488e
29 changed files with 1586 additions and 1 deletions
--- a/docs/plans/2026-05-21-2035-tracing-toggle-runbook.md
+++ b/docs/plans/2026-05-21-2035-tracing-toggle-runbook.md
@@ -0,0 +1,140 @@
+# Tracing enable/disable runbook for K8s components
+
+## Context
+
+The pod-creation tracing plan ([plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md](../../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md)) covers the *one-time setup* of OTLP tracing across kube-apiserver and kubelet, plus the Jaeger backend on docker-29. What's missing is a short operational doc for **toggling tracing on/off as quickly as possible** on the running cluster — useful for short measurement experiments where we want overhead only when we're actively recording.
+
+The Terraform/cloud-init changes already in flight (`master.tf`, `files/manifests/kube-apiserver.yaml`, `files/tracing-config.yaml`) cover *rebuilds*. This runbook covers the *live* cluster: kube-master-31 + kube-node-32 + kube-node-33, K8s 1.32.
+
+Output of this plan: a single new file `docs/kubernetes-tracing.md` containing three runbooks (quick suppress, full enable, full disable) and a verification section. No code or manifest changes — pure documentation.
+
+## What "as quickly as possible" means
+
+There are two speeds of toggling:
+
+1. **Quick suppress** — leave the tracing flags/config in place on the components; just stop Jaeger so span exports go nowhere. No K8s component restart, takes seconds, but components still pay the (small) cost of building spans and attempting to export.
+2. **Full enable / full disable** — edit configs and restart components. apiserver static-pod restart is automatic via kubelet manifest watch (~5–10s); kubelet itself needs `systemctl restart kubelet` on each node (brief NotReady blip per node, no pod evictions). This is what we'd use to truly remove overhead between experiments.
+
+The runbook will present both, with quick-suppress as the default for "I'm done measuring for now" and full-disable for "we don't need this on the cluster anymore".
+
+## Deliverable: `docs/kubernetes-tracing.md`
+
+### Structure
+
+```
+# Kubernetes component tracing
+
+## Components and where the toggles live
+- table: component | config file on node | restart trigger
+  - kube-apiserver | /etc/kubernetes/manifests/kube-apiserver.yaml (flag) +
+                     /etc/kubernetes/tracing-config.yaml (config) | kubelet auto-restarts static pod
+  - kubelet         | /var/lib/kubelet/config.yaml (`tracing:` block) +
+                     kube-system/kubelet-config CM (cluster-wide source of truth) | systemctl restart kubelet
+
+## Quick suppress (no component restart)
+1. `ssh novakj@192.168.0.29 'cd /path/to/tracing && docker compose stop jaeger'`
+2. Components continue running with tracing wired up but spans go nowhere.
+3. To resume: `docker compose start jaeger`.
+Caveat: components still build spans and attempt OTLP export — small CPU/network overhead. For a real "off", use full disable.
+
+## Full enable (live cluster)
+
+Prereqs: Jaeger is up on docker-29 (`docker compose up -d` in `docker-29/tracing/`).
+
+### kube-apiserver (master only)
+On 192.168.0.31:
+```bash
+# 1. Drop the TracingConfiguration file (same content as files/tracing-config.yaml)
+sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'EOF'
+apiVersion: apiserver.config.k8s.io/v1beta1
+kind: TracingConfiguration
+endpoint: 192.168.0.29:4317
+samplingRatePerMillion: 1000000
+EOF
+
+# 2. Back up the apiserver manifest
+sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing
+
+# 3. Add the flag (insert after --tls-private-key-file line)
+sudo sed -i '/--tls-private-key-file=/a\    - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml
+
+# 4. Wait for kubelet to restart the static pod (~5-10s)
+sudo crictl ps | grep apiserver         # new container id == restarted
+kubectl get --raw /livez                 # should return "ok"
+```
+
+### kubelet (every node: master + workers)
+
+Per-node edit:
+```bash
+# Append the tracing block under kubelet config (idempotent: check first)
+grep -q '^tracing:' /var/lib/kubelet/config.yaml || sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'EOF'
+tracing:
+  endpoint: 192.168.0.29:4317
+  samplingRatePerMillion: 1000000
+EOF
+
+sudo systemctl restart kubelet
+```
+
+Stagger across nodes (one at a time) so we don't NotReady the whole cluster at once.
+
+Then make it survive node rebuilds/joins by updating the cluster ConfigMap:
+```bash
+kubectl -n kube-system edit cm kubelet-config
+# add the same `tracing:` block under the `kubelet:` key
+```
+
+## Full disable (live cluster)
+
+### kube-apiserver
+```bash
+# Restore the pre-tracing manifest (preferred — exact revert)
+sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
+# OR remove just the flag in place:
+sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml
+
+# Optional: remove the now-unused config
+sudo rm /etc/kubernetes/tracing-config.yaml
+```
+kubelet restarts the static pod automatically.
+
+### kubelet (every node)
+```bash
+sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
+sudo systemctl restart kubelet
+```
+Then strip the `tracing:` block from `kube-system/kubelet-config` CM so node rejoins don't reintroduce it.
+
+## Verification (after enable)
+1. Open `http://192.168.0.29:16686` — Services dropdown shows `apiserver` (appears after the first traced request) and `kubelet`.
+2. Generate end-to-end traffic:
+   ```bash
+   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
+   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
+   kubectl delete pod trace-probe
+   ```
+3. In Jaeger: filter service `apiserver`, look for spans tagged with the probe pod name; filter service `kubelet`, look for `syncPod` spans.
+4. To verify *disable*: after running disable steps, repeat (2). Within a minute, no new spans for `apiserver`/`kubelet` should appear in Jaeger.
+
+## Notes
+- 100% sampling is fine for short experiments; for steady-state leave tracing off (quick suppress is enough between sessions).
+- The flag/config are GA in K8s 1.32 — no `--feature-gates` needed.
+- This runbook only covers apiserver + kubelet. etcd and kube-scheduler tracing are documented as future steps in the parent plan.
+```
+
+## Files to change
+
+| File | Change |
+|---|---|
+| `docs/kubernetes-tracing.md` | **new** — the runbook described above |
+
+No other files modified. The Terraform/cloud-init side stays as already drafted in the parent plan.
+
+## Verification of this plan's output
+
+After the runbook is written:
+
+1. Skim `docs/kubernetes-tracing.md` end-to-end and check that someone unfamiliar with the setup can follow it without opening the parent plan.
+2. Cross-check the sed/tee commands against the actual files on the live master via SSH — confirm line patterns match (e.g., the `--tls-private-key-file=` line is present so the sed insertion lands in the right spot).
+3. Dry-run the quick-suppress flow first (it's the lowest-risk path) and confirm Jaeger UI stops receiving new spans within ~30 s.