observability: add k8s API/kubelet tracing, Alloy, Mimir and Loki

Wire kube-apiserver and kubelet tracing to a Jaeger collector on
docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and
stand up Mimir + Loki on docker-30 as their backing stores.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

vms-home/docker-30/mimir/readme.md

@@ -0,0 +1,98 @@
+# Mimir
+
+Grafana Mimir, monolithic mode (`-target=all`), single-tenant, backed by Garage S3
+(same bucket-per-Garage-instance pattern as `../loki/`). Metrics ingest via
+Prometheus remote_write — Mimir does not scrape anything itself.
+
+- HTTP (query + push): `9009`
+- gRPC (internal): `9095`
+
+## 1. Garage setup (do this first — before `docker compose up`)
+
+Mimir needs three buckets. Create them and grant the existing Loki Garage key
+read+write access (reuse the key rather than minting a new one, unless you
+want tighter separation):
+
+```bash
+ssh novakj@192.168.0.30
+
+for b in mimir-blocks mimir-ruler mimir-alertmanager; do
+  docker exec garage /garage bucket create "$b"
+done
+
+docker exec garage /garage key list           # find the key name Loki already uses
+
+for b in mimir-blocks mimir-ruler mimir-alertmanager; do
+  docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
+done
+```
+
+## 2. `.env`
+
+Reuse the same Garage credentials Loki uses (see `../loki/.env`):
+
+```
+GARAGE_ACCESS_KEY=GK...
+GARAGE_SECRET_KEY=...
+```
+
+## 3. Start
+
+```bash
+cd /path/to/mimir
+docker compose up -d
+```
+
+## 4. Add Mimir as a Grafana datasource
+
+Reuses the Grafana already running from `../loki/docker-compose.yaml` (`:3001`)
+— no second Grafana instance.
+
+- Type: **Prometheus**
+- URL: `http://192.168.0.30:9009/prometheus`
+- No `X-Scope-OrgID` header needed (`multitenancy_enabled: false`)
+
+## 5. Verify
+
+```bash
+curl http://192.168.0.30:9009/ready                # -> ready
+docker compose ps                                   # mimir healthy
+docker compose logs -f mimir                         # no S3 auth/bucket errors
+
+# push endpoint reachable (expect a 4xx proto-decode error, not connection refused)
+curl http://192.168.0.30:9009/api/v1/push
+```
+
+In Grafana → Explore → Mimir datasource, run `up` or `count({__name__!=""})`.
+Empty result is expected until a producer remote_writes into Mimir.
+
+After ~2h (first block flush interval), confirm blocks landed in Garage:
+
+```bash
+docker exec garage /garage bucket info mimir-blocks
+```
+
+## 6. Wiring a producer (later, not done here)
+
+Point any Prometheus/Alloy remote_write at Mimir:
+
+```yaml
+remote_write:
+  - url: http://192.168.0.30:9009/api/v1/push
+```
+
+`vms/utility-101-shadow/docker/monitoring/prometheus.yml` is the natural first
+candidate to wire up.
+
+## Notes / gotchas
+
+- `bucket_lookup_type: path` in `mimir-config.yaml` is mandatory for Garage
+  (equivalent to Loki's `s3forcepathstyle: true`).
+- `-config.expand-env=true` is required on the command line, or `${GARAGE_*}`
+  in `mimir-config.yaml` won't expand.
+- First metrics won't appear in Garage until the first block flush (~2h) —
+  don't panic if `mimir-blocks` looks empty right after startup.
+- Retention is 31d, set via `limits.compactor_blocks_retention_period` (matches
+  Loki's `retention_period`).
+- `region: garage` in the config must byte-match `s3_region` in Garage's
+  `garage.toml` — same requirement as Loki.