home-kubernetes/docs/plans/2026-05-21-2310-tracing-host-move-to-docker-29.md

Move Jaeger/OTLP host from 192.168.0.30 (docker-30) → 192.168.0.29 (docker-29)

Context

The Jaeger backend that receives OTLP spans from kube-apiserver and kubelet has been relocated from the docker-30 host (192.168.0.30) to docker-29 (192.168.0.29). The new compose stack is already present at vms-home/docker-29/tracing/ and binds 0.0.0.0:4317/0.0.0.0:4318, so the backend side requires no changes.

The OTLP exporter endpoints on the Kubernetes side, plus the runbooks/plans that document them, still point at 192.168.0.30:4317. They need to be repointed to 192.168.0.29:4317. Without this change, apiserver and kubelet spans go nowhere once docker-30's Jaeger is shut down.

The other docker-30 references in the repo (Vault, Gitea, Zot registry mirror, Kanidm, nginx) are unrelated to tracing and must be left alone.

Scope

In-repo file edits only. Live cluster updates are documented as a follow-up checklist; no live execution is part of this plan.

File-level changes

1. Live OTEL config (load-bearing)

kubernetes-kvm-terraform/files/tracing-config.yaml:3

• endpoint: 192.168.0.30:4317 → endpoint: 192.168.0.29:4317

kubernetes-kvm-terraform/master.tf — two occurrences inside cloud-init heredocs:

• Line 122 (apiserver TracingConfiguration written to /etc/kubernetes/tracing-config.yaml): 192.168.0.30:4317 → 192.168.0.29:4317
• Line 169 (KubeletConfiguration.tracing.endpoint in the kubeadm config): 192.168.0.30:4317 → 192.168.0.29:4317
• Do NOT touch line 8 (zot_registry_ip = "192.168.0.30") — that's the Zot container-registry mirror, not tracing.

2. Runbook docs

docs/kubernetes-tracing.md — replace tracing-related references throughout (≈10 lines):

• All occurrences of 192.168.0.30:4317 → 192.168.0.29:4317 (lines 54, 83, 99)
• http://192.168.0.30:16686 → http://192.168.0.29:16686 (line 150)
• SSH targets novakj@192.168.0.30 → novakj@192.168.0.29 (lines 25, 31, 43)
• Remote directory ~/docker-30/tracing → ~/docker-29/tracing (lines 25, 31, 43)
• Prose "Jaeger on docker-30" → "Jaeger on docker-29" (lines 3, 40)

3. Plan files (historical record — also requested)

plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md — ~12 references:

• 192.168.0.30:4317 (lines 38, 76, 101, 111) and http://192.168.0.30:16686 (line 141) → .29
• docker-30/tracing/ directory path (lines 13, 16, 117, 128, 155) → docker-29/tracing/
• Prose mentions of "docker-30" in tracing context (lines 5, 11) → "docker-29"

docs/plans/2026-05-21-2035-tracing-toggle-runbook.md — ~6 references:

• 192.168.0.30:4317 (lines 51, 73) and http://192.168.0.30:16686 (line 110) → .29
• SSH target novakj@192.168.0.30 (line 35) → novakj@192.168.0.29
• Prose mentions of "docker-30" in tracing context (lines 5, 42) → "docker-29"

Search safety

Use a per-file targeted replace with surrounding context (e.g. endpoint: 192.168.0.30:4317, novakj@192.168.0.30, docker-30/tracing), not a blanket repo-wide sed. The leave-alone list is large (Zot, Vault, Gitea, Kanidm, nginx, gitignore, claude settings, generated tfstate, commented-out blackbox/prometheus examples) — see Phase 1 exploration for the full inventory.

Verification

In-repo

# Confirm no tracing-related .30 references remain
grep -nE '192\.168\.0\.30:(4317|16686)' -r .
grep -nE 'docker-30/tracing' -r .
# Confirm new IP is in place
grep -nE '192\.168\.0\.29:4317' kubernetes-kvm-terraform/

Both grep calls in step 1 should return empty. The leave-alone Zot/Vault/Gitea/nginx hits at port 3000/8200/9443/etc. should still be present (sanity check that the surgical replace didn't over-match).

Re-run tofu plan in kubernetes-kvm-terraform/ to confirm the diff only shows the two endpoint changes and no unrelated drift.

Live cluster follow-up (out of scope for this plan's execution, documented for the operator)

The in-repo changes only affect newly-bootstrapped nodes. The existing master + workers still have the old endpoint baked into their live files. After merging the repo changes, run:

1. kube-apiserver (master, 192.168.0.31):

   ssh novakj@192.168.0.31 'sudo sed -i s/192.168.0.30:4317/192.168.0.29:4317/ /etc/kubernetes/tracing-config.yaml'
   # kubelet auto-restarts the static pod within ~10s
   

2. kubelet on each node (192.168.0.31, .32, .33), serially:

   ssh novakj@$NODE 'sudo sed -i s/192.168.0.30:4317/192.168.0.29:4317/ /var/lib/kubelet/config.yaml && sudo systemctl restart kubelet'
   kubectl wait node --for=condition=Ready --timeout=60s <node-name>
   

3. Cluster ConfigMap so future joins/reboots pick it up:

   kubectl -n kube-system edit cm kubelet-config
   # update the tracing.endpoint value under kubelet:
   

4. End-to-end smoke test (from docs/kubernetes-tracing.md "Verification" section):

   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   

   Then open http://192.168.0.29:16686 and confirm apiserver and kubelet services show recent spans for trace-probe.