kacerr/home-kubernetes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

observability: add k8s API/kubelet tracing, Alloy, Mimir and Loki

Browse Source 
Wire kube-apiserver and kubelet tracing to a Jaeger collector on
docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and
stand up Mimir + Loki on docker-30 as their backing stores.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Jan Novak
2026-07-09 00:05:21 +02:00
parent 95355ef7a5
commit d43ffd488e
29 changed files with 1586 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

157
plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md Normal file
View File

@@ -0,0 +1,157 @@
# Pod-creation latency tracing with OpenTelemetry
## Context
Goal: measure end-to-end latency of `kubectl apply` → pod running, broken down by component. The approach is to enable native OpenTelemetry tracing in Kubernetes 1.32 components (kube-apiserver, kubelet, etcd) and ship spans to a Jaeger backend running outside the cluster on `docker-29`. With all three reporting under the same trace id (object-resource correlation), Jaeger will let us see how much time is spent in admission/etcd, on the wire, and inside the kubelet's pod sync loop.
Constraints / known gaps:
- **kube-scheduler has no native OTLP tracing yet**; there will be a gap between "object stored" and "kubelet sees assignment". We can infer it from event timestamps (`kubectl get events`) or `.status.conditions` PodScheduled/Initialized/ContainersReady/Ready timestamps.
- Cluster is K8s 1.32, kubeadm-bootstrapped via Terraform/cloud-init. Per Phase-3 decision, the canonical source-of-truth is the Terraform cloud-init; changes there cover future rebuilds. To activate tracing on the *currently running* nodes we also need a one-time manual application step (documented at the bottom).
## Backend: Jaeger on docker-29
New directory `docker-29/tracing/` containing `docker-compose.yaml`. Uses Jaeger all-in-one v1 with Badger on-disk storage and the built-in OTLP receiver enabled.
```yaml
# docker-29/tracing/docker-compose.yaml
services:
jaeger:
image: jaegertracing/all-in-one:1.62
container_name: jaeger
restart: unless-stopped
environment:
SPAN_STORAGE_TYPE: badger
BADGER_EPHEMERAL: "false"
BADGER_DIRECTORY_VALUE: /badger/data
BADGER_DIRECTORY_KEY: /badger/key
BADGER_SPAN_STORE_TTL: 4h # auto-expire traces older than 4h
BADGER_MAINTENANCE_INTERVAL: 5m # GC cycle that reclaims disk
COLLECTOR_OTLP_ENABLED: "true"
volumes:
- ./badger:/badger
ports:
- "4317:4317" # OTLP gRPC (apiserver + kubelet exporters)
- "4318:4318" # OTLP HTTP (optional)
- "16686:16686" # Jaeger UI
```
Endpoint that the cluster will send to: `192.168.0.29:4317`. Jaeger UI: `http://192.168.0.29:16686`.
The OTLP receiver in Jaeger all-in-one accepts **plaintext** OTLP/gRPC by default — matches `endpoint: ...:4317` on the K8s side with no TLS. Acceptable for a home lab on a private LAN.
### Storage sizing and retention
Two complementary levers control how much disk Badger uses:
1. **TTL** — `BADGER_SPAN_STORE_TTL=4h` deletes spans older than 4 hours; Badger's value-log GC reclaims disk on `BADGER_MAINTENANCE_INTERVAL` (default 5m). This is the hard upper bound on retention.
2. **Sampling rate** — controls *how many* spans get written in the first place. Set in the apiserver/kubelet TracingConfiguration (`samplingRatePerMillion`).
Rough estimate for this cluster (1 master + 2 workers, mostly idle, Flux reconciling continuously):
| Sampling | Span ingest rate | Disk footprint at 4h TTL |
|---|---|---|
| 100% (`1000000`) | ~500–3000 spans/s | ~6–40 GB steady-state |
| 1% (`10000`) | ~5–30 spans/s | ~60–400 MB steady-state |
| 0.1% (`1000`) | ~0.5–3 spans/s | ~6–40 MB steady-state |
Note: 100% sampling on this cluster is sized for short experiments, not long-term operation. The 4h TTL prevents runaway growth if I forget to lower it, but Badger has no built-in *size* cap — only TTL + GC. If a hard size cap is needed, mount `./badger/` from a fixed-size LVM volume or zfs dataset with a quota.
**Recommended progression**:
- Start with `samplingRatePerMillion: 1000000` (100%) and `TTL=4h` so the first measurements are easy to find.
- After the first round of experiments, drop sampling to `10000` (1%) for steady-state and adjust `TTL` upward if a longer history is useful.
## kube-apiserver tracing
Two changes, both in the Terraform cloud-init / referenced manifest:
### 1. Write the TracingConfiguration file (master.tf cloud-init)
Add to `write_files` in [kubernetes-kvm-terraform/master.tf](kubernetes-kvm-terraform/master.tf):
```yaml
- path: /etc/kubernetes/tracing-config.yaml
content: |
apiVersion: apiserver.config.k8s.io/v1beta1
kind: TracingConfiguration
endpoint: 192.168.0.29:4317
samplingRatePerMillion: 1000000 # 100% sampling for measurement work
```
The apiserver static pod already mounts `/etc/kubernetes` read-only via the `k8s-config` volume, so the file becomes visible inside the container automatically — no new volume mount needed.
### 2. Add the apiserver flag
Edit [kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml](kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml) — add one new flag in the `command:` list (alphabetical placement after `--tls-private-key-file` works; ordering isn't enforced):
```yaml
- --tracing-config-file=/etc/kubernetes/tracing-config.yaml
```
`APIServerTracing` is GA in 1.32 — no feature-gate flag required.
## kubelet tracing
Add a `tracing:` block to the existing `KubeletConfiguration` document inside the kubeadm-config heredoc in [kubernetes-kvm-terraform/master.tf](kubernetes-kvm-terraform/master.tf) (around line 158):
```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
tracing:
endpoint: 192.168.0.29:4317
samplingRatePerMillion: 1000000
```
`KubeletTracing` is beta and on by default in 1.32 — no feature gate needed.
Kubeadm stores this `KubeletConfiguration` in the `kubelet-config` ConfigMap in `kube-system`, so workers joining via `kubeadm join` pick the tracing block up automatically. The two existing workers (kube-node-32, kube-node-33) need a one-time manual nudge (see below) because they were already joined before the change.
## etcd tracing (optional, second-step)
etcd supports OTLP via flags `--experimental-enable-distributed-tracing`, `--experimental-distributed-tracing-address=192.168.0.29:4317`, and `--experimental-distributed-tracing-sampling-rate=1000000`. Enabling these requires editing the etcd static-pod manifest on the master — propose deferring until after the apiserver+kubelet path is verified, so we keep the first iteration narrow.
## Files to change
| File | Change |
|---|---|
| `docker-29/tracing/docker-compose.yaml` | **new** — Jaeger all-in-one + Badger |
| `kubernetes-kvm-terraform/master.tf` | add `tracing-config.yaml` to `write_files`; add `tracing:` block to `KubeletConfiguration` |
| `kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml` | add `--tracing-config-file=/etc/kubernetes/tracing-config.yaml` |
| `kubernetes-kvm-terraform/files/tracing-config.yaml` | **new** — reference copy of the TracingConfiguration (mirrors the inline content in master.tf, matches the existing `auth-config.yaml` convention) |
No changes to `nodes-on-homer.tf` / `nodes-on-beelink.tf` are needed — workers receive kubelet config from the cluster ConfigMap on join.
## Applying to the currently running cluster
The Terraform changes cover future rebuilds. To activate tracing **now** without rebuilding:
1. **Start Jaeger** on docker-29: `cd docker-29/tracing && docker compose up -d` (creates `./badger/` for storage).
2. **On the master (kube-master-31, 192.168.0.31)**:
- `scp` or write `/etc/kubernetes/tracing-config.yaml` with the TracingConfiguration shown above.
- Edit `/etc/kubernetes/manifests/kube-apiserver.yaml` to add `--tracing-config-file=/etc/kubernetes/tracing-config.yaml`. Kubelet will detect the manifest change and restart the apiserver static pod within seconds.
- Watch: `crictl ps | grep apiserver`, then `kubectl get --raw /livez`.
3. **Update the kubelet-config ConfigMap** in `kube-system`:
- `kubectl -n kube-system edit cm kubelet-config` and add the same `tracing:` block under `kubelet:`.
4. **On every node** (master + both workers): edit `/var/lib/kubelet/config.yaml` to append the `tracing:` block, then `systemctl restart kubelet`. Pods will *not* be evicted by a kubelet restart, but expect a brief NotReady blip per node — stagger the restarts.
## Verification
End-to-end test once everything is live:
1. UI reachable: open `http://192.168.0.29:16686` and check "Services" dropdown populates with `apiserver` (it will appear only after the first traced request).
2. Generate traffic that exercises the full path:
```
kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
kubectl delete pod trace-probe
```
3. In Jaeger UI, search service `apiserver`, operation `KubernetesAPI` (or similar) — you should see a trace covering admission/etcd write. Search service `kubelet`, operation `syncPod` — kubelet-side pod sync spans should be visible.
4. Cross-check the gap (scheduler) by comparing pod `metadata.creationTimestamp`, the `PodScheduled` condition `lastTransitionTime`, and the kubelet span start. The diff between those values quantifies the scheduler portion that we cannot trace natively.
5. Confirm Badger persistence: `docker compose restart jaeger`, reload UI — earlier traces should still be present.
## Notes / risks
- 100% sampling rate is fine for measurement; once we're done we should drop to e.g. `samplingRatePerMillion: 10000` (1%) so storage doesn't explode.
- Badger volume `./badger/` on docker-29 will grow with span volume; size budget worth watching but not urgent.
- `tracing.endpoint` is plaintext OTLP/gRPC and traverses the LAN — fine for a home lab but explicitly *not* encrypted.
- If apiserver fails to come back after the manifest edit, the most common cause is a typo in the new flag; revert is straightforward — keep a copy of the original `/etc/kubernetes/manifests/kube-apiserver.yaml` in `/root/` before editing.