observability: add k8s API/kubelet tracing, Alloy, Mimir and Loki
Wire kube-apiserver and kubelet tracing to a Jaeger collector on docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and stand up Mimir + Loki on docker-30 as their backing stores. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
116
docs/plans/2026-07-08-2346-alloy-metrics-to-mimir.md
Normal file
116
docs/plans/2026-07-08-2346-alloy-metrics-to-mimir.md
Normal file
@@ -0,0 +1,116 @@
|
||||
# Alloy → Mimir: scrape Kubernetes metrics + Prometheus Operator monitors
|
||||
|
||||
> On execution, also copy this file into the repo per CLAUDE.md convention:
|
||||
> `docs/plans/2026-07-08-HHMM-alloy-metrics-to-mimir.md` (get `HHMM` via `date "+%H%M"`).
|
||||
|
||||
## Context
|
||||
|
||||
Mimir is now up on docker-30 (`http://192.168.0.30:9009`, single-tenant, plain
|
||||
HTTP over LAN) but nothing in the cluster pushes metrics to it. Grafana Alloy
|
||||
already exists in the repo but is **logs-only** (ships pod/journal logs + k8s
|
||||
events to Loki) and — importantly — the entire `gitops/home-kubernetes/alloy/`
|
||||
directory is **not wired into Flux** yet.
|
||||
|
||||
kube-prometheus-stack is deployed and active: it provides the Prometheus
|
||||
Operator CRDs (ServiceMonitor / PodMonitor), kube-state-metrics, node-exporter,
|
||||
and ServiceMonitors for all standard targets (kubelet, cAdvisor, apiserver,
|
||||
coredns, etc.). Its bundled Prometheus scrapes everything into a local 60d TSDB
|
||||
but does **not** remote_write anywhere.
|
||||
|
||||
**Goal:** add a dedicated `alloy-metrics` instance that scrapes standard
|
||||
Kubernetes metrics and reads ServiceMonitors/PodMonitors from the Prometheus
|
||||
Operator, remote-writing to Mimir — additive alongside the existing Prometheus
|
||||
(no changes to kube-prometheus-stack). Because kube-prometheus-stack already
|
||||
ships ServiceMonitors for every standard target, a single
|
||||
`prometheus.operator.servicemonitors` + `prometheus.operator.podmonitors` pair
|
||||
satisfies **both** requirements at once. As part of wiring the `alloy/` dir into
|
||||
Flux, the existing `alloy-logs` and `alloy-events` releases also come under
|
||||
GitOps management (confirmed desired).
|
||||
|
||||
**Decisions confirmed:** additive/coexist with bundled Prometheus; activate all
|
||||
three Alloy instances (logs, events, metrics).
|
||||
|
||||
## Design
|
||||
|
||||
- Metrics scraping runs as a **single Deployment** (`controller.type: deployment`,
|
||||
1 replica) — never a DaemonSet, or every cluster-wide target would be scraped
|
||||
once per node. This is why it must be a separate release from `alloy-logs`
|
||||
(which is a DaemonSet).
|
||||
- Mirror the conventions already in `helmrelease_alloy-logs.yaml`: chart `alloy`
|
||||
`>=1.8.0 <2.0.0`, image `tag: v1.17.1`, `rbac.create: false`,
|
||||
`alloy.enableReporting: false`, `driftDetection: enabled`, install/upgrade
|
||||
`retries: 3`, `interval: 30m`, namespace `monitoring`.
|
||||
- River config = three components:
|
||||
1. `prometheus.operator.servicemonitors "sm"` — discovers/scrapes **all**
|
||||
ServiceMonitors cluster-wide (covers kubelet, cAdvisor, kube-state-metrics,
|
||||
node-exporter, apiserver, coredns, controller-manager, scheduler — i.e. the
|
||||
"standard kubernetes metrics").
|
||||
2. `prometheus.operator.podmonitors "pm"` — discovers/scrapes all PodMonitors.
|
||||
3. `prometheus.remote_write "mimir"` → `http://192.168.0.30:9009/api/v1/push`,
|
||||
`external_labels = { cluster = "homelab" }` (matches the logs config).
|
||||
Both operator components `forward_to = [prometheus.remote_write.mimir.receiver]`.
|
||||
- Reuse `cluster = "homelab"` external label for consistency with the Loki side.
|
||||
|
||||
## Files
|
||||
|
||||
### New — `gitops/home-kubernetes/alloy/helmrelease_alloy-metrics.yaml`
|
||||
HelmRelease `alloy-metrics` (ns `monitoring`), Deployment, serviceAccount
|
||||
`alloy-metrics`, inline `alloy.configMap.content` with the three components
|
||||
above. Modest resources (e.g. requests cpu 100m / mem 256Mi, limit mem 768Mi —
|
||||
operator discovery + scrape buffers use more than the log shipper).
|
||||
|
||||
### New — `gitops/home-kubernetes/alloy/rbac_metrics-collector.yaml`
|
||||
The `prometheus.operator.*` components need read access beyond the existing
|
||||
`alloy-log-reader` role. ClusterRole `alloy-metrics-reader` + ClusterRoleBinding
|
||||
to SA `alloy-metrics` (monitoring), granting:
|
||||
- core: `namespaces, nodes, nodes/metrics, nodes/proxy, services, endpoints, pods` — get/list/watch
|
||||
- `discovery.k8s.io`: `endpointslices` — get/list/watch
|
||||
- `networking.k8s.io`: `ingresses` — get/list/watch
|
||||
- `monitoring.coreos.com`: `servicemonitors, podmonitors, probes, scrapeconfigs` — get/list/watch
|
||||
- nonResourceURLs `/metrics`, `/metrics/cadvisor` — get (kubelet scraping)
|
||||
|
||||
(Kept as a separate file so the metrics RBAC is self-contained; the existing
|
||||
`rbac_log-collector.yaml` stays untouched.)
|
||||
|
||||
### New — `gitops/home-kubernetes/alloy/kustomization.yaml`
|
||||
`kind: Kustomization` listing all manifests in the dir: `helmrepository_grafana.yaml`,
|
||||
`helmrelease_alloy-logs.yaml`, `helmrelease_allow-events.yaml`,
|
||||
`helmrelease_alloy-metrics.yaml`, `rbac_log-collector.yaml`,
|
||||
`rbac_metrics-collector.yaml`. Makes the Flux build explicit/deterministic.
|
||||
|
||||
### Modified — `gitops/home-kubernetes/flux-system/extra-kustomizations.yaml`
|
||||
Append an `alloy` Flux Kustomization:
|
||||
```yaml
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: alloy
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m0s
|
||||
path: ./gitops/home-kubernetes/alloy
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
dependsOn:
|
||||
- name: kube-prometheus # needs the monitoring ns + operator CRDs/ServiceMonitors
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
1. Commit + push; force reconcile: `flux reconcile kustomization flux-system --with-source`.
|
||||
2. `kubectl get kustomizations -A` — `alloy` Ready=True.
|
||||
3. `kubectl get helmreleases -n monitoring` — `alloy-metrics` (+ logs/events) Ready.
|
||||
4. `kubectl -n monitoring get pods -l app.kubernetes.io/instance=alloy-metrics`
|
||||
Running; `kubectl -n monitoring logs deploy/alloy-metrics` shows no RBAC
|
||||
`forbidden` errors and targets discovered.
|
||||
5. Alloy UI: `kubectl -n monitoring port-forward deploy/alloy-metrics 12345:12345`
|
||||
→ open `http://localhost:12345`; confirm the two operator components list
|
||||
healthy targets and `prometheus.remote_write.mimir` shows successful sends.
|
||||
6. Confirm Mimir is receiving: query it directly, e.g.
|
||||
`curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=up' | jq '.data.result | length'`
|
||||
returns > 0 (or check the Grafana Mimir datasource for `up`,
|
||||
`kube_pod_info`, `node_cpu_seconds_total`, `container_cpu_usage_seconds_total`).
|
||||
7. Sanity: a `cluster="homelab"` label is present on the ingested series.
|
||||
Reference in New Issue
Block a user