home-kubernetes/docs/plans/2026-07-08-2346-alloy-metrics-to-mimir.md

# Alloy → Mimir: scrape Kubernetes metrics + Prometheus Operator monitors

> On execution, also copy this file into the repo per CLAUDE.md convention:
> `docs/plans/2026-07-08-HHMM-alloy-metrics-to-mimir.md` (get `HHMM` via `date "+%H%M"`).

## Context

Mimir is now up on docker-30 (`http://192.168.0.30:9009`, single-tenant, plain
HTTP over LAN) but nothing in the cluster pushes metrics to it. Grafana Alloy
already exists in the repo but is **logs-only** (ships pod/journal logs + k8s
events to Loki) and — importantly — the entire `gitops/home-kubernetes/alloy/`
directory is **not wired into Flux** yet.

kube-prometheus-stack is deployed and active: it provides the Prometheus
Operator CRDs (ServiceMonitor / PodMonitor), kube-state-metrics, node-exporter,
and ServiceMonitors for all standard targets (kubelet, cAdvisor, apiserver,
coredns, etc.). Its bundled Prometheus scrapes everything into a local 60d TSDB
but does **not** remote_write anywhere.

**Goal:** add a dedicated `alloy-metrics` instance that scrapes standard
Kubernetes metrics and reads ServiceMonitors/PodMonitors from the Prometheus
Operator, remote-writing to Mimir — additive alongside the existing Prometheus
(no changes to kube-prometheus-stack). Because kube-prometheus-stack already
ships ServiceMonitors for every standard target, a single
`prometheus.operator.servicemonitors` + `prometheus.operator.podmonitors` pair
satisfies **both** requirements at once. As part of wiring the `alloy/` dir into
Flux, the existing `alloy-logs` and `alloy-events` releases also come under
GitOps management (confirmed desired).

**Decisions confirmed:** additive/coexist with bundled Prometheus; activate all
three Alloy instances (logs, events, metrics).

## Design

- Metrics scraping runs as a **single Deployment** (`controller.type: deployment`,
  1 replica) — never a DaemonSet, or every cluster-wide target would be scraped
  once per node. This is why it must be a separate release from `alloy-logs`
  (which is a DaemonSet).
- Mirror the conventions already in `helmrelease_alloy-logs.yaml`: chart `alloy`
  `>=1.8.0 <2.0.0`, image `tag: v1.17.1`, `rbac.create: false`,
  `alloy.enableReporting: false`, `driftDetection: enabled`, install/upgrade
  `retries: 3`, `interval: 30m`, namespace `monitoring`.
- River config = three components:
  1. `prometheus.operator.servicemonitors "sm"` — discovers/scrapes **all**
     ServiceMonitors cluster-wide (covers kubelet, cAdvisor, kube-state-metrics,
     node-exporter, apiserver, coredns, controller-manager, scheduler — i.e. the
     "standard kubernetes metrics").
  2. `prometheus.operator.podmonitors "pm"` — discovers/scrapes all PodMonitors.
  3. `prometheus.remote_write "mimir"` → `http://192.168.0.30:9009/api/v1/push`,
     `external_labels = { cluster = "homelab" }` (matches the logs config).
  Both operator components `forward_to = [prometheus.remote_write.mimir.receiver]`.
- Reuse `cluster = "homelab"` external label for consistency with the Loki side.

## Files

### New — `gitops/home-kubernetes/alloy/helmrelease_alloy-metrics.yaml`
HelmRelease `alloy-metrics` (ns `monitoring`), Deployment, serviceAccount
`alloy-metrics`, inline `alloy.configMap.content` with the three components
above. Modest resources (e.g. requests cpu 100m / mem 256Mi, limit mem 768Mi —
operator discovery + scrape buffers use more than the log shipper).

### New — `gitops/home-kubernetes/alloy/rbac_metrics-collector.yaml`
The `prometheus.operator.*` components need read access beyond the existing
`alloy-log-reader` role. ClusterRole `alloy-metrics-reader` + ClusterRoleBinding
to SA `alloy-metrics` (monitoring), granting:
- core: `namespaces, nodes, nodes/metrics, nodes/proxy, services, endpoints, pods` — get/list/watch
- `discovery.k8s.io`: `endpointslices` — get/list/watch
- `networking.k8s.io`: `ingresses` — get/list/watch
- `monitoring.coreos.com`: `servicemonitors, podmonitors, probes, scrapeconfigs` — get/list/watch
- nonResourceURLs `/metrics`, `/metrics/cadvisor` — get (kubelet scraping)

(Kept as a separate file so the metrics RBAC is self-contained; the existing
`rbac_log-collector.yaml` stays untouched.)

### New — `gitops/home-kubernetes/alloy/kustomization.yaml`
`kind: Kustomization` listing all manifests in the dir: `helmrepository_grafana.yaml`,
`helmrelease_alloy-logs.yaml`, `helmrelease_allow-events.yaml`,
`helmrelease_alloy-metrics.yaml`, `rbac_log-collector.yaml`,
`rbac_metrics-collector.yaml`. Makes the Flux build explicit/deterministic.

### Modified — `gitops/home-kubernetes/flux-system/extra-kustomizations.yaml`
Append an `alloy` Flux Kustomization:
```yaml
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: alloy
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./gitops/home-kubernetes/alloy
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  dependsOn:
    - name: kube-prometheus   # needs the monitoring ns + operator CRDs/ServiceMonitors
```

## Verification

1. Commit + push; force reconcile: `flux reconcile kustomization flux-system --with-source`.
2. `kubectl get kustomizations -A` — `alloy` Ready=True.
3. `kubectl get helmreleases -n monitoring` — `alloy-metrics` (+ logs/events) Ready.
4. `kubectl -n monitoring get pods -l app.kubernetes.io/instance=alloy-metrics`
   Running; `kubectl -n monitoring logs deploy/alloy-metrics` shows no RBAC
   `forbidden` errors and targets discovered.
5. Alloy UI: `kubectl -n monitoring port-forward deploy/alloy-metrics 12345:12345`
   → open `http://localhost:12345`; confirm the two operator components list
   healthy targets and `prometheus.remote_write.mimir` shows successful sends.
6. Confirm Mimir is receiving: query it directly, e.g.
   `curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=up' | jq '.data.result | length'`
   returns > 0 (or check the Grafana Mimir datasource for `up`,
   `kube_pod_info`, `node_cpu_seconds_total`, `container_cpu_usage_seconds_total`).
7. Sanity: a `cluster="homelab"` label is present on the ingested series.