home-kubernetes/docs/kubernetes/alloy.md

Alloy: deployment & operations

Three Grafana Alloy instances run in the monitoring namespace, all deployed via the Grafana alloy Helm chart (>=1.8.0 <2.0.0, image v1.17.1) and managed by Flux with driftDetection: enabled (manual kubectl edits get reverted — always change config via git). Logs and events ship to Loki; metrics ship to Mimir. Every instance stamps cluster="homelab" as an external label.

---

Instances

Instance	Controller	Collects	Sink
alloy-logs	DaemonSet (runAsUser: 0, one pod per node)	pod logs + node journald	Loki http://192.168.0.30:3100/loki/api/v1/push
alloy-events	Deployment (1 replica)	Kubernetes events	Loki (same endpoint)
alloy-metrics	Deployment (1 replica)	all ServiceMonitors + PodMonitors cluster-wide	Mimir http://192.168.0.30:9009/api/v1/push

alloy-logs and alloy-events are one-per-node / cluster-scoped respectively — alloy-metrics is also a single Deployment, never a DaemonSet, since its Prometheus-Operator discovery is already cluster-wide (a DaemonSet would scrape every target once per node).

---

Files & Flux wiring

All manifests live in gitops/home-kubernetes/alloy/:

• helmrepository_grafana.yaml — the shared grafana HelmRepository.
• helmrelease_alloy-logs.yaml — logs DaemonSet.
• helmrelease_allow-events.yaml — events Deployment (note the allow filename typo, kept as-is).
• helmrelease_alloy-metrics.yaml — metrics Deployment.
• rbac_log-collector.yaml — ClusterRole alloy-log-reader, bound to the alloy-logs/alloy-events ServiceAccounts.
• rbac_metrics-collector.yaml — ClusterRole alloy-metrics-reader, bound to the alloy-metrics ServiceAccount.

Each HelmRelease sets rbac.create: false — RBAC is supplied manually by the two files above rather than by the chart, since the chart's default ClusterRole doesn't cover everything each pipeline needs (journald/events reads, or ServiceMonitor/PodMonitor/kubelet-metrics reads).

Reconciliation: a Flux Kustomization named alloy in gitops/home-kubernetes/flux-system/extra-kustomizations.yaml, pointing at ./gitops/home-kubernetes/alloy, with dependsOn: kube-prometheus — it needs the monitoring namespace and the Prometheus Operator CRDs (ServiceMonitor/ PodMonitor) to exist first.

---

How metrics discovery works

alloy-metrics runs two Alloy components — prometheus.operator.servicemonitors and prometheus.operator.podmonitors — which discover and scrape every ServiceMonitor/PodMonitor in the cluster, then forward to prometheus.remote_write targeting Mimir. There's no bespoke scrape config for "standard Kubernetes metrics": kube-prometheus-stack already ships ServiceMonitors for kubelet, cAdvisor, kube-state-metrics, node-exporter, apiserver, and coredns, so those are picked up automatically. Any future ServiceMonitor/PodMonitor added anywhere in the cluster is picked up the same way with no Alloy config change.

This is additive — kube-prometheus-stack's bundled Prometheus keeps scraping the same targets into its own local 60d TSDB. Alloy scraping the same targets a second time and shipping to Mimir is expected double-collection, not a bug.

---

Operating

Status:

flux get helmreleases -n monitoring
kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy -o wide

Force reconcile:

flux reconcile kustomization alloy --with-source
flux reconcile helmrelease alloy-metrics -n monitoring

Logs:

kubectl -n monitoring logs deploy/alloy-metrics
kubectl -n monitoring logs ds/alloy-logs
kubectl -n monitoring logs deploy/alloy-events

Alloy UI (component graph, target health, remote_write queue status):

kubectl -n monitoring port-forward deploy/alloy-metrics 12345:12345
# open http://localhost:12345

Editing config: change the inline alloy.configMap.content block in the relevant helmrelease_*.yaml, commit, push, then force-reconcile (above). Do not kubectl edit the generated ConfigMap directly — driftDetection will revert it on the next reconcile.

---

Verification / smoke tests

Metrics reaching Mimir:

curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=up' | jq '.data.result | length'
# expect > 0

Spot-check a few standard series exist and carry the cluster label:

curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=kube_pod_info' | jq '.data.result[0].metric'
curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=node_cpu_seconds_total' | jq '.data.result | length'
curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=container_cpu_usage_seconds_total' | jq '.data.result | length'
# confirm "cluster": "homelab" is present in the returned label sets

Logs reaching Loki:

curl -s http://192.168.0.30:3100/ready
# then in Grafana / logcli, query {cluster="homelab"}

---

Troubleshooting

Symptom	Likely cause
forbidden errors in alloy-metrics logs	alloy-metrics-reader ClusterRole missing a resource/verb — check rbac_metrics-collector.yaml against what the component is trying to read
No targets shown in the Alloy UI	Prometheus Operator CRDs not installed, or kube-prometheus Flux Kustomization not Ready yet (metrics depends on it)
prometheus.remote_write shows failed sends / 4xx-5xx	Mimir down, or wrong endpoint — must be http://192.168.0.30:9009/api/v1/push exactly
Loki push failing	Loki down at 192.168.0.30:3100, check with curl http://192.168.0.30:3100/ready
Same series appearing to be scraped twice	Expected — kube-prometheus-stack's bundled Prometheus and alloy-metrics both scrape the same ServiceMonitors by design (additive, not deduplicated)
Manual kubectl edit on the Alloy ConfigMap reverts itself	driftDetection: enabled — edit the HelmRelease in git instead