home-kubernetes/docs/plans/2026-07-08-0034-deploy-mimir.md

# Deploy Mimir on docker-30 (mirror the Loki setup)

## Context

Loki is already deployed on the docker host (`192.168.0.30`) under
`vms-home/docker-30/loki/`, storing chunks in Garage S3 and queried through a
shared Grafana. The user wants Mimir deployed for **metrics** in the same style:
self-contained compose dir, Garage S3 backend, `.env`-supplied credentials,
env-expanded config.

Decisions made:
- **Mimir only for now.** No producer is wired up in this plan; Mimir simply
  exposes its remote_write endpoint (`http://192.168.0.30:9009/api/v1/push`) for
  a Prometheus/Alloy to point at later.
- **Reuse the existing Grafana** (the one in the Loki compose) by adding a
  Prometheus-type datasource — no second Grafana.

Mimir runs in **monolithic mode** (`-target=all`, single binary), which is the
right shape for a single-node home lab and matches Loki's single-process model.

## Reference: how Loki does it (the pattern to copy)

`vms-home/docker-30/loki/loki-config.yaml` + `docker-compose.yaml`:
- Garage S3 at `192.168.0.30:3900`, `region: garage`, `s3forcepathstyle: true`,
  `insecure: true`, bucket `loki-chunks`.
- Creds via `${GARAGE_ACCESS_KEY}` / `${GARAGE_SECRET_KEY}` from `.env`,
  expanded with `-config.expand-env=true`.
- `obs` bridge network; Grafana published on `3001`.

## Files to create (new dir `vms-home/docker-30/mimir/`)

### 1. `mimir-config.yaml` — monolithic + Garage S3

```yaml
multitenancy_enabled: false            # single-tenant; Loki mirrors this (auth_enabled:false)

server:
  http_listen_port: 9009
  grpc_listen_port: 9095
  log_level: info

common:
  storage:
    backend: s3
    s3:
      endpoint: 192.168.0.30:3900      # no scheme; insecure toggles http
      region: garage                   # MUST byte-match s3_region in garage.toml
      access_key_id: ${GARAGE_ACCESS_KEY}
      secret_access_key: ${GARAGE_SECRET_KEY}
      insecure: true                   # http on LAN
      bucket_lookup_type: path         # path-style — Garage requirement (== s3forcepathstyle)

blocks_storage:
  s3: { bucket_name: mimir-blocks }
  tsdb:        { dir: /data/tsdb }
  bucket_store:{ sync_dir: /data/tsdb-sync }

ruler_storage:
  s3: { bucket_name: mimir-ruler }

alertmanager_storage:
  s3: { bucket_name: mimir-alertmanager }

compactor:
  data_dir: /data/compactor

ruler:
  rule_path: /data/ruler

alertmanager:
  data_dir: /data/alertmanager

limits:
  compactor_blocks_retention_period: 744h   # 31d, matches Loki's retention
  ingestion_rate: 50000                      # samples/s per tenant; bump on 429s
  ingestion_burst_size: 100000
```

Note: the three storage components inherit endpoint/creds from `common.storage`;
only `bucket_name` is overridden per component. Buckets can be collapsed to one
if preferred, but three is cleaner and Garage bucket creation is cheap.

### 2. `docker-compose.yaml` — copy Loki's, swap image/ports/paths

```yaml
services:
  mimir:
    image: grafana/mimir:2.16.1          # pin latest stable — verify tag before apply
    container_name: mimir
    restart: unless-stopped
    command:
      - -config.file=/etc/mimir/config.yaml
      - -config.expand-env=true          # REQUIRED — expands ${GARAGE_*}
      - -target=all                      # monolithic single-binary mode
    ports:
      - "9009:9009"                      # push (/api/v1/push) + query (/prometheus)
    volumes:
      - ./mimir-config.yaml:/etc/mimir/config.yaml:ro
      - ./mimir-data:/data
    env_file: [.env]
    networks: [obs]
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:9009/ready || exit 1"]
      interval: 15s
      timeout: 5s
      retries: 10
      start_period: 30s
    logging:
      driver: json-file
      options: { max-size: "10m", max-file: "3" }

networks:
  obs: { driver: bridge }
```

### 3. `.env` — reuse the same Garage credentials as Loki

```
GARAGE_ACCESS_KEY=<same key id Loki uses>
GARAGE_SECRET_KEY=<same secret>
```

## Manual steps on docker-30 (execution)

1. **Create Garage buckets** and grant the existing Loki key access:
   ```bash
   ssh novakj@192.168.0.30
   for b in mimir-blocks mimir-ruler mimir-alertmanager; do
     docker exec garage /garage bucket create "$b"
   done
   docker exec garage /garage key list           # get the key name Loki already uses
   for b in mimir-blocks mimir-ruler mimir-alertmanager; do
     docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
   done
   ```
2. Copy the `mimir/` dir to the host (or `git pull` if this repo is checked out
   there), fill `.env`, then `cd mimir && docker compose up -d`.
3. **Add Mimir datasource to the existing Grafana** (`:3001`):
   - Type: **Prometheus**
   - URL: `http://192.168.0.30:9009/prometheus`
   - No `X-Scope-OrgID` header needed (multitenancy disabled).

## Verification

- `curl http://192.168.0.30:9009/ready` → `ready` (also gates the healthcheck).
- `docker compose ps` shows `mimir` healthy; `docker compose logs -f mimir`
  clean (no S3 auth / bucket errors — confirms Garage wiring).
- Smoke-test the write path without a producer:
  ```bash
  curl http://192.168.0.30:9009/api/v1/push   # expect 4xx (proto body), NOT conn-refused
  ```
- In Grafana → Explore → Mimir datasource, run `up` or
  `count({__name__!=""})`. Empty until a producer remote_writes — expected.
- Confirm blocks land in Garage after ~2h (first block flush):
  `docker exec garage /garage bucket info mimir-blocks`.

## Later (out of scope here): wiring a producer

Point any Prometheus/Alloy at:
```yaml
remote_write:
  - url: http://192.168.0.30:9009/api/v1/push
```
The existing `vms/utility-101-shadow/docker/monitoring/prometheus.yml` is the
natural first candidate.