home-kubernetes/vms-home/docker-30/mimir/readme.md

Mimir

Grafana Mimir, monolithic mode (-target=all), single-tenant, backed by Garage S3 (same bucket-per-Garage-instance pattern as ../loki/). Metrics ingest via Prometheus remote_write — Mimir does not scrape anything itself.

• HTTP (query + push): 9009
• gRPC (internal): 9095

1. Garage setup (do this first — before docker compose up)

Mimir needs three buckets. Create them and grant the existing Loki Garage key read+write access (reuse the key rather than minting a new one, unless you want tighter separation):

ssh novakj@192.168.0.30

for b in mimir-blocks mimir-ruler mimir-alertmanager; do
  docker exec garage /garage bucket create "$b"
done

docker exec garage /garage key list           # find the key name Loki already uses

for b in mimir-blocks mimir-ruler mimir-alertmanager; do
  docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
done

2. .env

Reuse the same Garage credentials Loki uses (see ../loki/.env):

GARAGE_ACCESS_KEY=GK...
GARAGE_SECRET_KEY=...

3. Start

cd /path/to/mimir
docker compose up -d

4. Add Mimir as a Grafana datasource

Reuses the Grafana already running from ../loki/docker-compose.yaml (:3001) — no second Grafana instance.

• Type: Prometheus
• URL: http://192.168.0.30:9009/prometheus
• No X-Scope-OrgID header needed (multitenancy_enabled: false)

5. Verify

curl http://192.168.0.30:9009/ready                # -> ready
docker compose ps                                   # mimir healthy
docker compose logs -f mimir                         # no S3 auth/bucket errors

# push endpoint reachable (expect a 4xx proto-decode error, not connection refused)
curl http://192.168.0.30:9009/api/v1/push

In Grafana → Explore → Mimir datasource, run up or count({__name__!=""}). Empty result is expected until a producer remote_writes into Mimir.

After ~2h (first block flush interval), confirm blocks landed in Garage:

docker exec garage /garage bucket info mimir-blocks

6. Wiring a producer (later, not done here)

Point any Prometheus/Alloy remote_write at Mimir:

remote_write:
  - url: http://192.168.0.30:9009/api/v1/push

vms/utility-101-shadow/docker/monitoring/prometheus.yml is the natural first candidate to wire up.

Notes / gotchas

• bucket_lookup_type: path in mimir-config.yaml is mandatory for Garage (equivalent to Loki's s3forcepathstyle: true).
• -config.expand-env=true is required on the command line, or ${GARAGE_*} in mimir-config.yaml won't expand.
• First metrics won't appear in Garage until the first block flush (~2h) — don't panic if mimir-blocks looks empty right after startup.
• Retention is 31d, set via limits.compactor_blocks_retention_period (matches Loki's retention_period).
• region: garage in the config must byte-match s3_region in Garage's garage.toml — same requirement as Loki.