# Pod-creation latency tracing with OpenTelemetry

## Context

Goal: measure end-to-end latency of `kubectl apply` → pod running, broken down by component. The approach is to enable native OpenTelemetry tracing in Kubernetes 1.32 components (kube-apiserver, kubelet, etcd) and ship spans to a Jaeger backend running outside the cluster on `docker-29`. With all three reporting under the same trace id (object-resource correlation), Jaeger will let us see how much time is spent in admission/etcd, on the wire, and inside the kubelet's pod sync loop.

Constraints / known gaps:
- **kube-scheduler has no native OTLP tracing yet**; there will be a gap between "object stored" and "kubelet sees assignment". We can infer it from event timestamps (`kubectl get events`) or `.status.conditions` PodScheduled/Initialized/ContainersReady/Ready timestamps.
- Cluster is K8s 1.32, kubeadm-bootstrapped via Terraform/cloud-init. Per Phase-3 decision, the canonical source-of-truth is the Terraform cloud-init; changes there cover future rebuilds. To activate tracing on the *currently running* nodes we also need a one-time manual application step (documented at the bottom).

## Backend: Jaeger on docker-29

New directory `docker-29/tracing/` containing `docker-compose.yaml`. Uses Jaeger all-in-one v1 with Badger on-disk storage and the built-in OTLP receiver enabled.

```yaml
# docker-29/tracing/docker-compose.yaml
services:
  jaeger:
    image: jaegertracing/all-in-one:1.62
    container_name: jaeger
    restart: unless-stopped
    environment:
      SPAN_STORAGE_TYPE: badger
      BADGER_EPHEMERAL: "false"
      BADGER_DIRECTORY_VALUE: /badger/data
      BADGER_DIRECTORY_KEY: /badger/key
      BADGER_SPAN_STORE_TTL: 4h       # auto-expire traces older than 4h
      BADGER_MAINTENANCE_INTERVAL: 5m # GC cycle that reclaims disk
      COLLECTOR_OTLP_ENABLED: "true"
    volumes:
      - ./badger:/badger
    ports:
      - "4317:4317"    # OTLP gRPC (apiserver + kubelet exporters)
      - "4318:4318"    # OTLP HTTP (optional)
      - "16686:16686"  # Jaeger UI
```

Endpoint that the cluster will send to: `192.168.0.29:4317`. Jaeger UI: `http://192.168.0.29:16686`.

The OTLP receiver in Jaeger all-in-one accepts **plaintext** OTLP/gRPC by default — matches `endpoint: ...:4317` on the K8s side with no TLS. Acceptable for a home lab on a private LAN.

### Storage sizing and retention

Two complementary levers control how much disk Badger uses:

1. **TTL** — `BADGER_SPAN_STORE_TTL=4h` deletes spans older than 4 hours; Badger's value-log GC reclaims disk on `BADGER_MAINTENANCE_INTERVAL` (default 5m). This is the hard upper bound on retention.
2. **Sampling rate** — controls *how many* spans get written in the first place. Set in the apiserver/kubelet TracingConfiguration (`samplingRatePerMillion`).

Rough estimate for this cluster (1 master + 2 workers, mostly idle, Flux reconciling continuously):

| Sampling | Span ingest rate | Disk footprint at 4h TTL |
|---|---|---|
| 100% (`1000000`) | ~500–3000 spans/s | ~6–40 GB steady-state |
| 1% (`10000`) | ~5–30 spans/s | ~60–400 MB steady-state |
| 0.1% (`1000`) | ~0.5–3 spans/s | ~6–40 MB steady-state |

Note: 100% sampling on this cluster is sized for short experiments, not long-term operation. The 4h TTL prevents runaway growth if I forget to lower it, but Badger has no built-in *size* cap — only TTL + GC. If a hard size cap is needed, mount `./badger/` from a fixed-size LVM volume or zfs dataset with a quota.

**Recommended progression**:
- Start with `samplingRatePerMillion: 1000000` (100%) and `TTL=4h` so the first measurements are easy to find.
- After the first round of experiments, drop sampling to `10000` (1%) for steady-state and adjust `TTL` upward if a longer history is useful.

## kube-apiserver tracing

Two changes, both in the Terraform cloud-init / referenced manifest:

### 1. Write the TracingConfiguration file (master.tf cloud-init)

Add to `write_files` in [kubernetes-kvm-terraform/master.tf](kubernetes-kvm-terraform/master.tf):

```yaml
- path: /etc/kubernetes/tracing-config.yaml
  content: |
    apiVersion: apiserver.config.k8s.io/v1beta1
    kind: TracingConfiguration
    endpoint: 192.168.0.29:4317
    samplingRatePerMillion: 1000000  # 100% sampling for measurement work
```

The apiserver static pod already mounts `/etc/kubernetes` read-only via the `k8s-config` volume, so the file becomes visible inside the container automatically — no new volume mount needed.

### 2. Add the apiserver flag

Edit [kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml](kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml) — add one new flag in the `command:` list (alphabetical placement after `--tls-private-key-file` works; ordering isn't enforced):

```yaml
- --tracing-config-file=/etc/kubernetes/tracing-config.yaml
```

`APIServerTracing` is GA in 1.32 — no feature-gate flag required.

## kubelet tracing

Add a `tracing:` block to the existing `KubeletConfiguration` document inside the kubeadm-config heredoc in [kubernetes-kvm-terraform/master.tf](kubernetes-kvm-terraform/master.tf) (around line 158):

```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
tracing:
  endpoint: 192.168.0.29:4317
  samplingRatePerMillion: 1000000
```

`KubeletTracing` is beta and on by default in 1.32 — no feature gate needed.

Kubeadm stores this `KubeletConfiguration` in the `kubelet-config` ConfigMap in `kube-system`, so workers joining via `kubeadm join` pick the tracing block up automatically. The two existing workers (kube-node-32, kube-node-33) need a one-time manual nudge (see below) because they were already joined before the change.

## etcd tracing (optional, second-step)

etcd supports OTLP via flags `--experimental-enable-distributed-tracing`, `--experimental-distributed-tracing-address=192.168.0.29:4317`, and `--experimental-distributed-tracing-sampling-rate=1000000`. Enabling these requires editing the etcd static-pod manifest on the master — propose deferring until after the apiserver+kubelet path is verified, so we keep the first iteration narrow.

## Files to change

| File | Change |
|---|---|
| `docker-29/tracing/docker-compose.yaml` | **new** — Jaeger all-in-one + Badger |
| `kubernetes-kvm-terraform/master.tf` | add `tracing-config.yaml` to `write_files`; add `tracing:` block to `KubeletConfiguration` |
| `kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml` | add `--tracing-config-file=/etc/kubernetes/tracing-config.yaml` |
| `kubernetes-kvm-terraform/files/tracing-config.yaml` | **new** — reference copy of the TracingConfiguration (mirrors the inline content in master.tf, matches the existing `auth-config.yaml` convention) |

No changes to `nodes-on-homer.tf` / `nodes-on-beelink.tf` are needed — workers receive kubelet config from the cluster ConfigMap on join.

## Applying to the currently running cluster

The Terraform changes cover future rebuilds. To activate tracing **now** without rebuilding:

1. **Start Jaeger** on docker-29: `cd docker-29/tracing && docker compose up -d` (creates `./badger/` for storage).
2. **On the master (kube-master-31, 192.168.0.31)**:
   - `scp` or write `/etc/kubernetes/tracing-config.yaml` with the TracingConfiguration shown above.
   - Edit `/etc/kubernetes/manifests/kube-apiserver.yaml` to add `--tracing-config-file=/etc/kubernetes/tracing-config.yaml`. Kubelet will detect the manifest change and restart the apiserver static pod within seconds.
   - Watch: `crictl ps | grep apiserver`, then `kubectl get --raw /livez`.
3. **Update the kubelet-config ConfigMap** in `kube-system`:
   - `kubectl -n kube-system edit cm kubelet-config` and add the same `tracing:` block under `kubelet:`.
4. **On every node** (master + both workers): edit `/var/lib/kubelet/config.yaml` to append the `tracing:` block, then `systemctl restart kubelet`. Pods will *not* be evicted by a kubelet restart, but expect a brief NotReady blip per node — stagger the restarts.

## Verification

End-to-end test once everything is live:

1. UI reachable: open `http://192.168.0.29:16686` and check "Services" dropdown populates with `apiserver` (it will appear only after the first traced request).
2. Generate traffic that exercises the full path:
   ```
   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   ```
3. In Jaeger UI, search service `apiserver`, operation `KubernetesAPI` (or similar) — you should see a trace covering admission/etcd write. Search service `kubelet`, operation `syncPod` — kubelet-side pod sync spans should be visible.
4. Cross-check the gap (scheduler) by comparing pod `metadata.creationTimestamp`, the `PodScheduled` condition `lastTransitionTime`, and the kubelet span start. The diff between those values quantifies the scheduler portion that we cannot trace natively.
5. Confirm Badger persistence: `docker compose restart jaeger`, reload UI — earlier traces should still be present.

## Notes / risks

- 100% sampling rate is fine for measurement; once we're done we should drop to e.g. `samplingRatePerMillion: 10000` (1%) so storage doesn't explode.
- Badger volume `./badger/` on docker-29 will grow with span volume; size budget worth watching but not urgent.
- `tracing.endpoint` is plaintext OTLP/gRPC and traverses the LAN — fine for a home lab but explicitly *not* encrypted.
- If apiserver fails to come back after the manifest edit, the most common cause is a typo in the new flag; revert is straightforward — keep a copy of the original `/etc/kubernetes/manifests/kube-apiserver.yaml` in `/root/` before editing.