home-kubernetes/docs/plans/2026-05-21-2310-tracing-host-move-to-docker-29.md

# Move Jaeger/OTLP host from 192.168.0.30 (docker-30) → 192.168.0.29 (docker-29)

## Context

The Jaeger backend that receives OTLP spans from kube-apiserver and kubelet has been relocated from the `docker-30` host (192.168.0.30) to `docker-29` (192.168.0.29). The new compose stack is already present at [vms-home/docker-29/tracing/](../../vms-home/docker-29/tracing/) and binds `0.0.0.0:4317`/`0.0.0.0:4318`, so the backend side requires no changes.

The OTLP exporter endpoints on the Kubernetes side, plus the runbooks/plans that document them, still point at `192.168.0.30:4317`. They need to be repointed to `192.168.0.29:4317`. Without this change, apiserver and kubelet spans go nowhere once docker-30's Jaeger is shut down.

The other docker-30 references in the repo (Vault, Gitea, Zot registry mirror, Kanidm, nginx) are unrelated to tracing and must be left alone.

## Scope

In-repo file edits only. Live cluster updates are documented as a follow-up checklist; no live execution is part of this plan.

## File-level changes

### 1. Live OTEL config (load-bearing)

**[kubernetes-kvm-terraform/files/tracing-config.yaml:3](../../kubernetes-kvm-terraform/files/tracing-config.yaml#L3)**
- `endpoint: 192.168.0.30:4317` → `endpoint: 192.168.0.29:4317`

**[kubernetes-kvm-terraform/master.tf](../../kubernetes-kvm-terraform/master.tf)** — two occurrences inside cloud-init heredocs:
- Line 122 (apiserver `TracingConfiguration` written to `/etc/kubernetes/tracing-config.yaml`): `192.168.0.30:4317` → `192.168.0.29:4317`
- Line 169 (`KubeletConfiguration.tracing.endpoint` in the kubeadm config): `192.168.0.30:4317` → `192.168.0.29:4317`
- **Do NOT touch line 8** (`zot_registry_ip = "192.168.0.30"`) — that's the Zot container-registry mirror, not tracing.

### 2. Runbook docs

**[docs/kubernetes-tracing.md](../kubernetes-tracing.md)** — replace tracing-related references throughout (≈10 lines):
- All occurrences of `192.168.0.30:4317` → `192.168.0.29:4317` (lines 54, 83, 99)
- `http://192.168.0.30:16686` → `http://192.168.0.29:16686` (line 150)
- SSH targets `novakj@192.168.0.30` → `novakj@192.168.0.29` (lines 25, 31, 43)
- Remote directory `~/docker-30/tracing` → `~/docker-29/tracing` (lines 25, 31, 43)
- Prose "Jaeger on docker-30" → "Jaeger on docker-29" (lines 3, 40)

### 3. Plan files (historical record — also requested)

**[plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md](../../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md)** — ~12 references:
- `192.168.0.30:4317` (lines 38, 76, 101, 111) and `http://192.168.0.30:16686` (line 141) → `.29`
- `docker-30/tracing/` directory path (lines 13, 16, 117, 128, 155) → `docker-29/tracing/`
- Prose mentions of "docker-30" in tracing context (lines 5, 11) → "docker-29"

**[docs/plans/2026-05-21-2035-tracing-toggle-runbook.md](2026-05-21-2035-tracing-toggle-runbook.md)** — ~6 references:
- `192.168.0.30:4317` (lines 51, 73) and `http://192.168.0.30:16686` (line 110) → `.29`
- SSH target `novakj@192.168.0.30` (line 35) → `novakj@192.168.0.29`
- Prose mentions of "docker-30" in tracing context (lines 5, 42) → "docker-29"

### Search safety

Use a per-file targeted replace with surrounding context (e.g. `endpoint: 192.168.0.30:4317`, `novakj@192.168.0.30`, `docker-30/tracing`), not a blanket repo-wide `sed`. The leave-alone list is large (Zot, Vault, Gitea, Kanidm, nginx, gitignore, claude settings, generated tfstate, commented-out blackbox/prometheus examples) — see Phase 1 exploration for the full inventory.

## Verification

### In-repo
```bash
# Confirm no tracing-related .30 references remain
grep -nE '192\.168\.0\.30:(4317|16686)' -r .
grep -nE 'docker-30/tracing' -r .
# Confirm new IP is in place
grep -nE '192\.168\.0\.29:4317' kubernetes-kvm-terraform/
```
Both `grep` calls in step 1 should return empty. The leave-alone Zot/Vault/Gitea/nginx hits at port 3000/8200/9443/etc. should still be present (sanity check that the surgical replace didn't over-match).

Re-run `tofu plan` in [kubernetes-kvm-terraform/](../../kubernetes-kvm-terraform/) to confirm the diff only shows the two endpoint changes and no unrelated drift.

### Live cluster follow-up (out of scope for this plan's execution, documented for the operator)

The in-repo changes only affect newly-bootstrapped nodes. The existing master + workers still have the old endpoint baked into their live files. After merging the repo changes, run:

1. **kube-apiserver** (master, 192.168.0.31):
   ```bash
   ssh novakj@192.168.0.31 'sudo sed -i s/192.168.0.30:4317/192.168.0.29:4317/ /etc/kubernetes/tracing-config.yaml'
   # kubelet auto-restarts the static pod within ~10s
   ```
2. **kubelet** on each node (192.168.0.31, .32, .33), serially:
   ```bash
   ssh novakj@$NODE 'sudo sed -i s/192.168.0.30:4317/192.168.0.29:4317/ /var/lib/kubelet/config.yaml && sudo systemctl restart kubelet'
   kubectl wait node --for=condition=Ready --timeout=60s <node-name>
   ```
3. **Cluster ConfigMap** so future joins/reboots pick it up:
   ```bash
   kubectl -n kube-system edit cm kubelet-config
   # update the tracing.endpoint value under kubelet:
   ```
4. **End-to-end smoke test** (from [docs/kubernetes-tracing.md](../kubernetes-tracing.md) "Verification" section):
   ```bash
   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   ```
   Then open `http://192.168.0.29:16686` and confirm `apiserver` and `kubelet` services show recent spans for `trace-probe`.