home-kubernetes/docs/plans/2026-07-08-2346-alloy-metrics-to-mimir.md

Alloy → Mimir: scrape Kubernetes metrics + Prometheus Operator monitors

On execution, also copy this file into the repo per CLAUDE.md convention: docs/plans/2026-07-08-HHMM-alloy-metrics-to-mimir.md (get HHMM via date "+%H%M").

Context

Mimir is now up on docker-30 (http://192.168.0.30:9009, single-tenant, plain HTTP over LAN) but nothing in the cluster pushes metrics to it. Grafana Alloy already exists in the repo but is logs-only (ships pod/journal logs + k8s events to Loki) and — importantly — the entire gitops/home-kubernetes/alloy/ directory is not wired into Flux yet.

kube-prometheus-stack is deployed and active: it provides the Prometheus Operator CRDs (ServiceMonitor / PodMonitor), kube-state-metrics, node-exporter, and ServiceMonitors for all standard targets (kubelet, cAdvisor, apiserver, coredns, etc.). Its bundled Prometheus scrapes everything into a local 60d TSDB but does not remote_write anywhere.

Goal: add a dedicated alloy-metrics instance that scrapes standard Kubernetes metrics and reads ServiceMonitors/PodMonitors from the Prometheus Operator, remote-writing to Mimir — additive alongside the existing Prometheus (no changes to kube-prometheus-stack). Because kube-prometheus-stack already ships ServiceMonitors for every standard target, a single prometheus.operator.servicemonitors + prometheus.operator.podmonitors pair satisfies both requirements at once. As part of wiring the alloy/ dir into Flux, the existing alloy-logs and alloy-events releases also come under GitOps management (confirmed desired).

Decisions confirmed: additive/coexist with bundled Prometheus; activate all three Alloy instances (logs, events, metrics).

Design

• Metrics scraping runs as a single Deployment (controller.type: deployment, 1 replica) — never a DaemonSet, or every cluster-wide target would be scraped once per node. This is why it must be a separate release from alloy-logs (which is a DaemonSet).
• Mirror the conventions already in helmrelease_alloy-logs.yaml: chart alloy >=1.8.0 <2.0.0, image tag: v1.17.1, rbac.create: false, alloy.enableReporting: false, driftDetection: enabled, install/upgrade retries: 3, interval: 30m, namespace monitoring.
• River config = three components:
  1. prometheus.operator.servicemonitors "sm" — discovers/scrapes all ServiceMonitors cluster-wide (covers kubelet, cAdvisor, kube-state-metrics, node-exporter, apiserver, coredns, controller-manager, scheduler — i.e. the "standard kubernetes metrics").
  2. prometheus.operator.podmonitors "pm" — discovers/scrapes all PodMonitors.
  3. prometheus.remote_write "mimir" → http://192.168.0.30:9009/api/v1/push, external_labels = { cluster = "homelab" } (matches the logs config). Both operator components forward_to = [prometheus.remote_write.mimir.receiver].
• Reuse cluster = "homelab" external label for consistency with the Loki side.

Files

New — gitops/home-kubernetes/alloy/helmrelease_alloy-metrics.yaml

HelmRelease alloy-metrics (ns monitoring), Deployment, serviceAccount alloy-metrics, inline alloy.configMap.content with the three components above. Modest resources (e.g. requests cpu 100m / mem 256Mi, limit mem 768Mi — operator discovery + scrape buffers use more than the log shipper).

New — gitops/home-kubernetes/alloy/rbac_metrics-collector.yaml

The prometheus.operator.* components need read access beyond the existing alloy-log-reader role. ClusterRole alloy-metrics-reader + ClusterRoleBinding to SA alloy-metrics (monitoring), granting:

• core: namespaces, nodes, nodes/metrics, nodes/proxy, services, endpoints, pods — get/list/watch
• discovery.k8s.io: endpointslices — get/list/watch
• networking.k8s.io: ingresses — get/list/watch
• monitoring.coreos.com: servicemonitors, podmonitors, probes, scrapeconfigs — get/list/watch
• nonResourceURLs /metrics, /metrics/cadvisor — get (kubelet scraping)

(Kept as a separate file so the metrics RBAC is self-contained; the existing rbac_log-collector.yaml stays untouched.)

New — gitops/home-kubernetes/alloy/kustomization.yaml

kind: Kustomization listing all manifests in the dir: helmrepository_grafana.yaml, helmrelease_alloy-logs.yaml, helmrelease_allow-events.yaml, helmrelease_alloy-metrics.yaml, rbac_log-collector.yaml, rbac_metrics-collector.yaml. Makes the Flux build explicit/deterministic.

Modified — gitops/home-kubernetes/flux-system/extra-kustomizations.yaml

Append an alloy Flux Kustomization:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: alloy
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./gitops/home-kubernetes/alloy
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  dependsOn:
    - name: kube-prometheus   # needs the monitoring ns + operator CRDs/ServiceMonitors

Verification

1. Commit + push; force reconcile: flux reconcile kustomization flux-system --with-source.
2. kubectl get kustomizations -A — alloy Ready=True.
3. kubectl get helmreleases -n monitoring — alloy-metrics (+ logs/events) Ready.
4. kubectl -n monitoring get pods -l app.kubernetes.io/instance=alloy-metrics Running; kubectl -n monitoring logs deploy/alloy-metrics shows no RBAC forbidden errors and targets discovered.
5. Alloy UI: kubectl -n monitoring port-forward deploy/alloy-metrics 12345:12345 → open http://localhost:12345; confirm the two operator components list healthy targets and prometheus.remote_write.mimir shows successful sends.
6. Confirm Mimir is receiving: query it directly, e.g. curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=up' | jq '.data.result | length' returns > 0 (or check the Grafana Mimir datasource for up, kube_pod_info, node_cpu_seconds_total, container_cpu_usage_seconds_total).
7. Sanity: a cluster="homelab" label is present on the ingested series.