Jan Novak
d43ffd488e
Wire kube-apiserver and kubelet tracing to a Jaeger collector on docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and stand up Mimir + Loki on docker-30 as their backing stores. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
158 lines
9.6 KiB
Markdown
158 lines
9.6 KiB
Markdown
# Pod-creation latency tracing with OpenTelemetry
|
||
|
||
## Context
|
||
|
||
Goal: measure end-to-end latency of `kubectl apply` → pod running, broken down by component. The approach is to enable native OpenTelemetry tracing in Kubernetes 1.32 components (kube-apiserver, kubelet, etcd) and ship spans to a Jaeger backend running outside the cluster on `docker-29`. With all three reporting under the same trace id (object-resource correlation), Jaeger will let us see how much time is spent in admission/etcd, on the wire, and inside the kubelet's pod sync loop.
|
||
|
||
Constraints / known gaps:
|
||
- **kube-scheduler has no native OTLP tracing yet**; there will be a gap between "object stored" and "kubelet sees assignment". We can infer it from event timestamps (`kubectl get events`) or `.status.conditions` PodScheduled/Initialized/ContainersReady/Ready timestamps.
|
||
- Cluster is K8s 1.32, kubeadm-bootstrapped via Terraform/cloud-init. Per Phase-3 decision, the canonical source-of-truth is the Terraform cloud-init; changes there cover future rebuilds. To activate tracing on the *currently running* nodes we also need a one-time manual application step (documented at the bottom).
|
||
|
||
## Backend: Jaeger on docker-29
|
||
|
||
New directory `docker-29/tracing/` containing `docker-compose.yaml`. Uses Jaeger all-in-one v1 with Badger on-disk storage and the built-in OTLP receiver enabled.
|
||
|
||
```yaml
|
||
# docker-29/tracing/docker-compose.yaml
|
||
services:
|
||
jaeger:
|
||
image: jaegertracing/all-in-one:1.62
|
||
container_name: jaeger
|
||
restart: unless-stopped
|
||
environment:
|
||
SPAN_STORAGE_TYPE: badger
|
||
BADGER_EPHEMERAL: "false"
|
||
BADGER_DIRECTORY_VALUE: /badger/data
|
||
BADGER_DIRECTORY_KEY: /badger/key
|
||
BADGER_SPAN_STORE_TTL: 4h # auto-expire traces older than 4h
|
||
BADGER_MAINTENANCE_INTERVAL: 5m # GC cycle that reclaims disk
|
||
COLLECTOR_OTLP_ENABLED: "true"
|
||
volumes:
|
||
- ./badger:/badger
|
||
ports:
|
||
- "4317:4317" # OTLP gRPC (apiserver + kubelet exporters)
|
||
- "4318:4318" # OTLP HTTP (optional)
|
||
- "16686:16686" # Jaeger UI
|
||
```
|
||
|
||
Endpoint that the cluster will send to: `192.168.0.29:4317`. Jaeger UI: `http://192.168.0.29:16686`.
|
||
|
||
The OTLP receiver in Jaeger all-in-one accepts **plaintext** OTLP/gRPC by default — matches `endpoint: ...:4317` on the K8s side with no TLS. Acceptable for a home lab on a private LAN.
|
||
|
||
### Storage sizing and retention
|
||
|
||
Two complementary levers control how much disk Badger uses:
|
||
|
||
1. **TTL** — `BADGER_SPAN_STORE_TTL=4h` deletes spans older than 4 hours; Badger's value-log GC reclaims disk on `BADGER_MAINTENANCE_INTERVAL` (default 5m). This is the hard upper bound on retention.
|
||
2. **Sampling rate** — controls *how many* spans get written in the first place. Set in the apiserver/kubelet TracingConfiguration (`samplingRatePerMillion`).
|
||
|
||
Rough estimate for this cluster (1 master + 2 workers, mostly idle, Flux reconciling continuously):
|
||
|
||
| Sampling | Span ingest rate | Disk footprint at 4h TTL |
|
||
|---|---|---|
|
||
| 100% (`1000000`) | ~500–3000 spans/s | ~6–40 GB steady-state |
|
||
| 1% (`10000`) | ~5–30 spans/s | ~60–400 MB steady-state |
|
||
| 0.1% (`1000`) | ~0.5–3 spans/s | ~6–40 MB steady-state |
|
||
|
||
Note: 100% sampling on this cluster is sized for short experiments, not long-term operation. The 4h TTL prevents runaway growth if I forget to lower it, but Badger has no built-in *size* cap — only TTL + GC. If a hard size cap is needed, mount `./badger/` from a fixed-size LVM volume or zfs dataset with a quota.
|
||
|
||
**Recommended progression**:
|
||
- Start with `samplingRatePerMillion: 1000000` (100%) and `TTL=4h` so the first measurements are easy to find.
|
||
- After the first round of experiments, drop sampling to `10000` (1%) for steady-state and adjust `TTL` upward if a longer history is useful.
|
||
|
||
## kube-apiserver tracing
|
||
|
||
Two changes, both in the Terraform cloud-init / referenced manifest:
|
||
|
||
### 1. Write the TracingConfiguration file (master.tf cloud-init)
|
||
|
||
Add to `write_files` in [kubernetes-kvm-terraform/master.tf](kubernetes-kvm-terraform/master.tf):
|
||
|
||
```yaml
|
||
- path: /etc/kubernetes/tracing-config.yaml
|
||
content: |
|
||
apiVersion: apiserver.config.k8s.io/v1beta1
|
||
kind: TracingConfiguration
|
||
endpoint: 192.168.0.29:4317
|
||
samplingRatePerMillion: 1000000 # 100% sampling for measurement work
|
||
```
|
||
|
||
The apiserver static pod already mounts `/etc/kubernetes` read-only via the `k8s-config` volume, so the file becomes visible inside the container automatically — no new volume mount needed.
|
||
|
||
### 2. Add the apiserver flag
|
||
|
||
Edit [kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml](kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml) — add one new flag in the `command:` list (alphabetical placement after `--tls-private-key-file` works; ordering isn't enforced):
|
||
|
||
```yaml
|
||
- --tracing-config-file=/etc/kubernetes/tracing-config.yaml
|
||
```
|
||
|
||
`APIServerTracing` is GA in 1.32 — no feature-gate flag required.
|
||
|
||
## kubelet tracing
|
||
|
||
Add a `tracing:` block to the existing `KubeletConfiguration` document inside the kubeadm-config heredoc in [kubernetes-kvm-terraform/master.tf](kubernetes-kvm-terraform/master.tf) (around line 158):
|
||
|
||
```yaml
|
||
apiVersion: kubelet.config.k8s.io/v1beta1
|
||
kind: KubeletConfiguration
|
||
cgroupDriver: systemd
|
||
tracing:
|
||
endpoint: 192.168.0.29:4317
|
||
samplingRatePerMillion: 1000000
|
||
```
|
||
|
||
`KubeletTracing` is beta and on by default in 1.32 — no feature gate needed.
|
||
|
||
Kubeadm stores this `KubeletConfiguration` in the `kubelet-config` ConfigMap in `kube-system`, so workers joining via `kubeadm join` pick the tracing block up automatically. The two existing workers (kube-node-32, kube-node-33) need a one-time manual nudge (see below) because they were already joined before the change.
|
||
|
||
## etcd tracing (optional, second-step)
|
||
|
||
etcd supports OTLP via flags `--experimental-enable-distributed-tracing`, `--experimental-distributed-tracing-address=192.168.0.29:4317`, and `--experimental-distributed-tracing-sampling-rate=1000000`. Enabling these requires editing the etcd static-pod manifest on the master — propose deferring until after the apiserver+kubelet path is verified, so we keep the first iteration narrow.
|
||
|
||
## Files to change
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `docker-29/tracing/docker-compose.yaml` | **new** — Jaeger all-in-one + Badger |
|
||
| `kubernetes-kvm-terraform/master.tf` | add `tracing-config.yaml` to `write_files`; add `tracing:` block to `KubeletConfiguration` |
|
||
| `kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml` | add `--tracing-config-file=/etc/kubernetes/tracing-config.yaml` |
|
||
| `kubernetes-kvm-terraform/files/tracing-config.yaml` | **new** — reference copy of the TracingConfiguration (mirrors the inline content in master.tf, matches the existing `auth-config.yaml` convention) |
|
||
|
||
No changes to `nodes-on-homer.tf` / `nodes-on-beelink.tf` are needed — workers receive kubelet config from the cluster ConfigMap on join.
|
||
|
||
## Applying to the currently running cluster
|
||
|
||
The Terraform changes cover future rebuilds. To activate tracing **now** without rebuilding:
|
||
|
||
1. **Start Jaeger** on docker-29: `cd docker-29/tracing && docker compose up -d` (creates `./badger/` for storage).
|
||
2. **On the master (kube-master-31, 192.168.0.31)**:
|
||
- `scp` or write `/etc/kubernetes/tracing-config.yaml` with the TracingConfiguration shown above.
|
||
- Edit `/etc/kubernetes/manifests/kube-apiserver.yaml` to add `--tracing-config-file=/etc/kubernetes/tracing-config.yaml`. Kubelet will detect the manifest change and restart the apiserver static pod within seconds.
|
||
- Watch: `crictl ps | grep apiserver`, then `kubectl get --raw /livez`.
|
||
3. **Update the kubelet-config ConfigMap** in `kube-system`:
|
||
- `kubectl -n kube-system edit cm kubelet-config` and add the same `tracing:` block under `kubelet:`.
|
||
4. **On every node** (master + both workers): edit `/var/lib/kubelet/config.yaml` to append the `tracing:` block, then `systemctl restart kubelet`. Pods will *not* be evicted by a kubelet restart, but expect a brief NotReady blip per node — stagger the restarts.
|
||
|
||
## Verification
|
||
|
||
End-to-end test once everything is live:
|
||
|
||
1. UI reachable: open `http://192.168.0.29:16686` and check "Services" dropdown populates with `apiserver` (it will appear only after the first traced request).
|
||
2. Generate traffic that exercises the full path:
|
||
```
|
||
kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
|
||
kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
|
||
kubectl delete pod trace-probe
|
||
```
|
||
3. In Jaeger UI, search service `apiserver`, operation `KubernetesAPI` (or similar) — you should see a trace covering admission/etcd write. Search service `kubelet`, operation `syncPod` — kubelet-side pod sync spans should be visible.
|
||
4. Cross-check the gap (scheduler) by comparing pod `metadata.creationTimestamp`, the `PodScheduled` condition `lastTransitionTime`, and the kubelet span start. The diff between those values quantifies the scheduler portion that we cannot trace natively.
|
||
5. Confirm Badger persistence: `docker compose restart jaeger`, reload UI — earlier traces should still be present.
|
||
|
||
## Notes / risks
|
||
|
||
- 100% sampling rate is fine for measurement; once we're done we should drop to e.g. `samplingRatePerMillion: 10000` (1%) so storage doesn't explode.
|
||
- Badger volume `./badger/` on docker-29 will grow with span volume; size budget worth watching but not urgent.
|
||
- `tracing.endpoint` is plaintext OTLP/gRPC and traverses the LAN — fine for a home lab but explicitly *not* encrypted.
|
||
- If apiserver fails to come back after the manifest edit, the most common cause is a typo in the new flag; revert is straightforward — keep a copy of the original `/etc/kubernetes/manifests/kube-apiserver.yaml` in `/root/` before editing.
|