home-kubernetes/docs/plans/2026-07-08-0034-deploy-mimir.md

Deploy Mimir on docker-30 (mirror the Loki setup)

Context

Loki is already deployed on the docker host (192.168.0.30) under vms-home/docker-30/loki/, storing chunks in Garage S3 and queried through a shared Grafana. The user wants Mimir deployed for metrics in the same style: self-contained compose dir, Garage S3 backend, .env-supplied credentials, env-expanded config.

Decisions made:

• Mimir only for now. No producer is wired up in this plan; Mimir simply exposes its remote_write endpoint (http://192.168.0.30:9009/api/v1/push) for a Prometheus/Alloy to point at later.
• Reuse the existing Grafana (the one in the Loki compose) by adding a Prometheus-type datasource — no second Grafana.

Mimir runs in monolithic mode (-target=all, single binary), which is the right shape for a single-node home lab and matches Loki's single-process model.

Reference: how Loki does it (the pattern to copy)

vms-home/docker-30/loki/loki-config.yaml + docker-compose.yaml:

• Garage S3 at 192.168.0.30:3900, region: garage, s3forcepathstyle: true, insecure: true, bucket loki-chunks.
• Creds via ${GARAGE_ACCESS_KEY} / ${GARAGE_SECRET_KEY} from .env, expanded with -config.expand-env=true.
• obs bridge network; Grafana published on 3001.

Files to create (new dir vms-home/docker-30/mimir/)

1. mimir-config.yaml — monolithic + Garage S3

multitenancy_enabled: false            # single-tenant; Loki mirrors this (auth_enabled:false)

server:
  http_listen_port: 9009
  grpc_listen_port: 9095
  log_level: info

common:
  storage:
    backend: s3
    s3:
      endpoint: 192.168.0.30:3900      # no scheme; insecure toggles http
      region: garage                   # MUST byte-match s3_region in garage.toml
      access_key_id: ${GARAGE_ACCESS_KEY}
      secret_access_key: ${GARAGE_SECRET_KEY}
      insecure: true                   # http on LAN
      bucket_lookup_type: path         # path-style — Garage requirement (== s3forcepathstyle)

blocks_storage:
  s3: { bucket_name: mimir-blocks }
  tsdb:        { dir: /data/tsdb }
  bucket_store:{ sync_dir: /data/tsdb-sync }

ruler_storage:
  s3: { bucket_name: mimir-ruler }

alertmanager_storage:
  s3: { bucket_name: mimir-alertmanager }

compactor:
  data_dir: /data/compactor

ruler:
  rule_path: /data/ruler

alertmanager:
  data_dir: /data/alertmanager

limits:
  compactor_blocks_retention_period: 744h   # 31d, matches Loki's retention
  ingestion_rate: 50000                      # samples/s per tenant; bump on 429s
  ingestion_burst_size: 100000

Note: the three storage components inherit endpoint/creds from common.storage; only bucket_name is overridden per component. Buckets can be collapsed to one if preferred, but three is cleaner and Garage bucket creation is cheap.

2. docker-compose.yaml — copy Loki's, swap image/ports/paths

services:
  mimir:
    image: grafana/mimir:2.16.1          # pin latest stable — verify tag before apply
    container_name: mimir
    restart: unless-stopped
    command:
      - -config.file=/etc/mimir/config.yaml
      - -config.expand-env=true          # REQUIRED — expands ${GARAGE_*}
      - -target=all                      # monolithic single-binary mode
    ports:
      - "9009:9009"                      # push (/api/v1/push) + query (/prometheus)
    volumes:
      - ./mimir-config.yaml:/etc/mimir/config.yaml:ro
      - ./mimir-data:/data
    env_file: [.env]
    networks: [obs]
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:9009/ready || exit 1"]
      interval: 15s
      timeout: 5s
      retries: 10
      start_period: 30s
    logging:
      driver: json-file
      options: { max-size: "10m", max-file: "3" }

networks:
  obs: { driver: bridge }

3. .env — reuse the same Garage credentials as Loki

GARAGE_ACCESS_KEY=<same key id Loki uses>
GARAGE_SECRET_KEY=<same secret>

Manual steps on docker-30 (execution)

1. Create Garage buckets and grant the existing Loki key access:

   ssh novakj@192.168.0.30
   for b in mimir-blocks mimir-ruler mimir-alertmanager; do
     docker exec garage /garage bucket create "$b"
   done
   docker exec garage /garage key list           # get the key name Loki already uses
   for b in mimir-blocks mimir-ruler mimir-alertmanager; do
     docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
   done
   

2. Copy the mimir/ dir to the host (or git pull if this repo is checked out there), fill .env, then cd mimir && docker compose up -d.
3. Add Mimir datasource to the existing Grafana (:3001):
   • Type: Prometheus
   • URL: http://192.168.0.30:9009/prometheus
   • No X-Scope-OrgID header needed (multitenancy disabled).

Verification

• curl http://192.168.0.30:9009/ready → ready (also gates the healthcheck).
• docker compose ps shows mimir healthy; docker compose logs -f mimir clean (no S3 auth / bucket errors — confirms Garage wiring).
• Smoke-test the write path without a producer:

  curl http://192.168.0.30:9009/api/v1/push   # expect 4xx (proto body), NOT conn-refused
  

• In Grafana → Explore → Mimir datasource, run up or count({__name__!=""}). Empty until a producer remote_writes — expected.
• Confirm blocks land in Garage after ~2h (first block flush): docker exec garage /garage bucket info mimir-blocks.

Later (out of scope here): wiring a producer

Point any Prometheus/Alloy at:

remote_write:
  - url: http://192.168.0.30:9009/api/v1/push

The existing vms/utility-101-shadow/docker/monitoring/prometheus.yml is the natural first candidate.