Jan Novak
d43ffd488e
Wire kube-apiserver and kubelet tracing to a Jaeger collector on docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and stand up Mimir + Loki on docker-30 as their backing stores. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
99 lines
2.9 KiB
Markdown
99 lines
2.9 KiB
Markdown
# Mimir
|
|
|
|
Grafana Mimir, monolithic mode (`-target=all`), single-tenant, backed by Garage S3
|
|
(same bucket-per-Garage-instance pattern as `../loki/`). Metrics ingest via
|
|
Prometheus remote_write — Mimir does not scrape anything itself.
|
|
|
|
- HTTP (query + push): `9009`
|
|
- gRPC (internal): `9095`
|
|
|
|
## 1. Garage setup (do this first — before `docker compose up`)
|
|
|
|
Mimir needs three buckets. Create them and grant the existing Loki Garage key
|
|
read+write access (reuse the key rather than minting a new one, unless you
|
|
want tighter separation):
|
|
|
|
```bash
|
|
ssh novakj@192.168.0.30
|
|
|
|
for b in mimir-blocks mimir-ruler mimir-alertmanager; do
|
|
docker exec garage /garage bucket create "$b"
|
|
done
|
|
|
|
docker exec garage /garage key list # find the key name Loki already uses
|
|
|
|
for b in mimir-blocks mimir-ruler mimir-alertmanager; do
|
|
docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
|
|
done
|
|
```
|
|
|
|
## 2. `.env`
|
|
|
|
Reuse the same Garage credentials Loki uses (see `../loki/.env`):
|
|
|
|
```
|
|
GARAGE_ACCESS_KEY=GK...
|
|
GARAGE_SECRET_KEY=...
|
|
```
|
|
|
|
## 3. Start
|
|
|
|
```bash
|
|
cd /path/to/mimir
|
|
docker compose up -d
|
|
```
|
|
|
|
## 4. Add Mimir as a Grafana datasource
|
|
|
|
Reuses the Grafana already running from `../loki/docker-compose.yaml` (`:3001`)
|
|
— no second Grafana instance.
|
|
|
|
- Type: **Prometheus**
|
|
- URL: `http://192.168.0.30:9009/prometheus`
|
|
- No `X-Scope-OrgID` header needed (`multitenancy_enabled: false`)
|
|
|
|
## 5. Verify
|
|
|
|
```bash
|
|
curl http://192.168.0.30:9009/ready # -> ready
|
|
docker compose ps # mimir healthy
|
|
docker compose logs -f mimir # no S3 auth/bucket errors
|
|
|
|
# push endpoint reachable (expect a 4xx proto-decode error, not connection refused)
|
|
curl http://192.168.0.30:9009/api/v1/push
|
|
```
|
|
|
|
In Grafana → Explore → Mimir datasource, run `up` or `count({__name__!=""})`.
|
|
Empty result is expected until a producer remote_writes into Mimir.
|
|
|
|
After ~2h (first block flush interval), confirm blocks landed in Garage:
|
|
|
|
```bash
|
|
docker exec garage /garage bucket info mimir-blocks
|
|
```
|
|
|
|
## 6. Wiring a producer (later, not done here)
|
|
|
|
Point any Prometheus/Alloy remote_write at Mimir:
|
|
|
|
```yaml
|
|
remote_write:
|
|
- url: http://192.168.0.30:9009/api/v1/push
|
|
```
|
|
|
|
`vms/utility-101-shadow/docker/monitoring/prometheus.yml` is the natural first
|
|
candidate to wire up.
|
|
|
|
## Notes / gotchas
|
|
|
|
- `bucket_lookup_type: path` in `mimir-config.yaml` is mandatory for Garage
|
|
(equivalent to Loki's `s3forcepathstyle: true`).
|
|
- `-config.expand-env=true` is required on the command line, or `${GARAGE_*}`
|
|
in `mimir-config.yaml` won't expand.
|
|
- First metrics won't appear in Garage until the first block flush (~2h) —
|
|
don't panic if `mimir-blocks` looks empty right after startup.
|
|
- Retention is 31d, set via `limits.compactor_blocks_retention_period` (matches
|
|
Loki's `retention_period`).
|
|
- `region: garage` in the config must byte-match `s3_region` in Garage's
|
|
`garage.toml` — same requirement as Loki.
|