observability: add k8s API/kubelet tracing, Alloy, Mimir and Loki

Wire kube-apiserver and kubelet tracing to a Jaeger collector on docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and stand up Mimir + Loki on docker-30 as their backing stores. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:05:21 +02:00
parent 95355ef7a5
commit d43ffd488e
29 changed files with 1586 additions and 1 deletions
--- a/docs/plans/2026-07-08-0034-deploy-mimir.md
+++ b/docs/plans/2026-07-08-0034-deploy-mimir.md
@@ -0,0 +1,165 @@
+# Deploy Mimir on docker-30 (mirror the Loki setup)
+
+## Context
+
+Loki is already deployed on the docker host (`192.168.0.30`) under
+`vms-home/docker-30/loki/`, storing chunks in Garage S3 and queried through a
+shared Grafana. The user wants Mimir deployed for **metrics** in the same style:
+self-contained compose dir, Garage S3 backend, `.env`-supplied credentials,
+env-expanded config.
+
+Decisions made:
+- **Mimir only for now.** No producer is wired up in this plan; Mimir simply
+  exposes its remote_write endpoint (`http://192.168.0.30:9009/api/v1/push`) for
+  a Prometheus/Alloy to point at later.
+- **Reuse the existing Grafana** (the one in the Loki compose) by adding a
+  Prometheus-type datasource — no second Grafana.
+
+Mimir runs in **monolithic mode** (`-target=all`, single binary), which is the
+right shape for a single-node home lab and matches Loki's single-process model.
+
+## Reference: how Loki does it (the pattern to copy)
+
+`vms-home/docker-30/loki/loki-config.yaml` + `docker-compose.yaml`:
+- Garage S3 at `192.168.0.30:3900`, `region: garage`, `s3forcepathstyle: true`,
+  `insecure: true`, bucket `loki-chunks`.
+- Creds via `${GARAGE_ACCESS_KEY}` / `${GARAGE_SECRET_KEY}` from `.env`,
+  expanded with `-config.expand-env=true`.
+- `obs` bridge network; Grafana published on `3001`.
+
+## Files to create (new dir `vms-home/docker-30/mimir/`)
+
+### 1. `mimir-config.yaml` — monolithic + Garage S3
+
+```yaml
+multitenancy_enabled: false            # single-tenant; Loki mirrors this (auth_enabled:false)
+
+server:
+  http_listen_port: 9009
+  grpc_listen_port: 9095
+  log_level: info
+
+common:
+  storage:
+    backend: s3
+    s3:
+      endpoint: 192.168.0.30:3900      # no scheme; insecure toggles http
+      region: garage                   # MUST byte-match s3_region in garage.toml
+      access_key_id: ${GARAGE_ACCESS_KEY}
+      secret_access_key: ${GARAGE_SECRET_KEY}
+      insecure: true                   # http on LAN
+      bucket_lookup_type: path         # path-style — Garage requirement (== s3forcepathstyle)
+
+blocks_storage:
+  s3: { bucket_name: mimir-blocks }
+  tsdb:        { dir: /data/tsdb }
+  bucket_store:{ sync_dir: /data/tsdb-sync }
+
+ruler_storage:
+  s3: { bucket_name: mimir-ruler }
+
+alertmanager_storage:
+  s3: { bucket_name: mimir-alertmanager }
+
+compactor:
+  data_dir: /data/compactor
+
+ruler:
+  rule_path: /data/ruler
+
+alertmanager:
+  data_dir: /data/alertmanager
+
+limits:
+  compactor_blocks_retention_period: 744h   # 31d, matches Loki's retention
+  ingestion_rate: 50000                      # samples/s per tenant; bump on 429s
+  ingestion_burst_size: 100000
+```
+
+Note: the three storage components inherit endpoint/creds from `common.storage`;
+only `bucket_name` is overridden per component. Buckets can be collapsed to one
+if preferred, but three is cleaner and Garage bucket creation is cheap.
+
+### 2. `docker-compose.yaml` — copy Loki's, swap image/ports/paths
+
+```yaml
+services:
+  mimir:
+    image: grafana/mimir:2.16.1          # pin latest stable — verify tag before apply
+    container_name: mimir
+    restart: unless-stopped
+    command:
+      - -config.file=/etc/mimir/config.yaml
+      - -config.expand-env=true          # REQUIRED — expands ${GARAGE_*}
+      - -target=all                      # monolithic single-binary mode
+    ports:
+      - "9009:9009"                      # push (/api/v1/push) + query (/prometheus)
+    volumes:
+      - ./mimir-config.yaml:/etc/mimir/config.yaml:ro
+      - ./mimir-data:/data
+    env_file: [.env]
+    networks: [obs]
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://localhost:9009/ready || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 10
+      start_period: 30s
+    logging:
+      driver: json-file
+      options: { max-size: "10m", max-file: "3" }
+
+networks:
+  obs: { driver: bridge }
+```
+
+### 3. `.env` — reuse the same Garage credentials as Loki
+
+```
+GARAGE_ACCESS_KEY=<same key id Loki uses>
+GARAGE_SECRET_KEY=<same secret>
+```
+
+## Manual steps on docker-30 (execution)
+
+1. **Create Garage buckets** and grant the existing Loki key access:
+   ```bash
+   ssh novakj@192.168.0.30
+   for b in mimir-blocks mimir-ruler mimir-alertmanager; do
+     docker exec garage /garage bucket create "$b"
+   done
+   docker exec garage /garage key list           # get the key name Loki already uses
+   for b in mimir-blocks mimir-ruler mimir-alertmanager; do
+     docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
+   done
+   ```
+2. Copy the `mimir/` dir to the host (or `git pull` if this repo is checked out
+   there), fill `.env`, then `cd mimir && docker compose up -d`.
+3. **Add Mimir datasource to the existing Grafana** (`:3001`):
+   - Type: **Prometheus**
+   - URL: `http://192.168.0.30:9009/prometheus`
+   - No `X-Scope-OrgID` header needed (multitenancy disabled).
+
+## Verification
+
+- `curl http://192.168.0.30:9009/ready` → `ready` (also gates the healthcheck).
+- `docker compose ps` shows `mimir` healthy; `docker compose logs -f mimir`
+  clean (no S3 auth / bucket errors — confirms Garage wiring).
+- Smoke-test the write path without a producer:
+  ```bash
+  curl http://192.168.0.30:9009/api/v1/push   # expect 4xx (proto body), NOT conn-refused
+  ```
+- In Grafana → Explore → Mimir datasource, run `up` or
+  `count({__name__!=""})`. Empty until a producer remote_writes — expected.
+- Confirm blocks land in Garage after ~2h (first block flush):
+  `docker exec garage /garage bucket info mimir-blocks`.
+
+## Later (out of scope here): wiring a producer
+
+Point any Prometheus/Alloy at:
+```yaml
+remote_write:
+  - url: http://192.168.0.30:9009/api/v1/push
+```
+The existing `vms/utility-101-shadow/docker/monitoring/prometheus.yml` is the
+natural first candidate.