kacerr/home-kubernetes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

observability: add k8s API/kubelet tracing, Alloy, Mimir and Loki

Browse Source 
Wire kube-apiserver and kubelet tracing to a Jaeger collector on
docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and
stand up Mimir + Loki on docker-30 as their backing stores.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Jan Novak
2026-07-09 00:05:21 +02:00
parent 95355ef7a5
commit d43ffd488e
29 changed files with 1586 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

140
docs/plans/2026-05-21-2035-tracing-toggle-runbook.md Normal file
View File

@@ -0,0 +1,140 @@
# Tracing enable/disable runbook for K8s components
## Context
The pod-creation tracing plan ([plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md](../../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md)) covers the *one-time setup* of OTLP tracing across kube-apiserver and kubelet, plus the Jaeger backend on docker-29. What's missing is a short operational doc for **toggling tracing on/off as quickly as possible** on the running cluster — useful for short measurement experiments where we want overhead only when we're actively recording.
The Terraform/cloud-init changes already in flight (`master.tf`, `files/manifests/kube-apiserver.yaml`, `files/tracing-config.yaml`) cover *rebuilds*. This runbook covers the *live* cluster: kube-master-31 + kube-node-32 + kube-node-33, K8s 1.32.
Output of this plan: a single new file `docs/kubernetes-tracing.md` containing three runbooks (quick suppress, full enable, full disable) and a verification section. No code or manifest changes — pure documentation.
## What "as quickly as possible" means
There are two speeds of toggling:
1. **Quick suppress** — leave the tracing flags/config in place on the components; just stop Jaeger so span exports go nowhere. No K8s component restart, takes seconds, but components still pay the (small) cost of building spans and attempting to export.
2. **Full enable / full disable** — edit configs and restart components. apiserver static-pod restart is automatic via kubelet manifest watch (~5–10s); kubelet itself needs `systemctl restart kubelet` on each node (brief NotReady blip per node, no pod evictions). This is what we'd use to truly remove overhead between experiments.
The runbook will present both, with quick-suppress as the default for "I'm done measuring for now" and full-disable for "we don't need this on the cluster anymore".
## Deliverable: `docs/kubernetes-tracing.md`
### Structure
```
# Kubernetes component tracing
## Components and where the toggles live
- table: component | config file on node | restart trigger
- kube-apiserver | /etc/kubernetes/manifests/kube-apiserver.yaml (flag) +
/etc/kubernetes/tracing-config.yaml (config) | kubelet auto-restarts static pod
- kubelet | /var/lib/kubelet/config.yaml (`tracing:` block) +
kube-system/kubelet-config CM (cluster-wide source of truth) | systemctl restart kubelet
## Quick suppress (no component restart)
1. `ssh novakj@192.168.0.29 'cd /path/to/tracing && docker compose stop jaeger'`
2. Components continue running with tracing wired up but spans go nowhere.
3. To resume: `docker compose start jaeger`.
Caveat: components still build spans and attempt OTLP export — small CPU/network overhead. For a real "off", use full disable.
## Full enable (live cluster)
Prereqs: Jaeger is up on docker-29 (`docker compose up -d` in `docker-29/tracing/`).
### kube-apiserver (master only)
On 192.168.0.31:
```bash
# 1. Drop the TracingConfiguration file (same content as files/tracing-config.yaml)
sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'EOF'
apiVersion: apiserver.config.k8s.io/v1beta1
kind: TracingConfiguration
endpoint: 192.168.0.29:4317
samplingRatePerMillion: 1000000
EOF
# 2. Back up the apiserver manifest
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing
# 3. Add the flag (insert after --tls-private-key-file line)
sudo sed -i '/--tls-private-key-file=/a\ - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml
# 4. Wait for kubelet to restart the static pod (~5-10s)
sudo crictl ps | grep apiserver # new container id == restarted
kubectl get --raw /livez # should return "ok"
```
### kubelet (every node: master + workers)
Per-node edit:
```bash
# Append the tracing block under kubelet config (idempotent: check first)
grep -q '^tracing:' /var/lib/kubelet/config.yaml || sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'EOF'
tracing:
endpoint: 192.168.0.29:4317
samplingRatePerMillion: 1000000
EOF
sudo systemctl restart kubelet
```
Stagger across nodes (one at a time) so we don't NotReady the whole cluster at once.
Then make it survive node rebuilds/joins by updating the cluster ConfigMap:
```bash
kubectl -n kube-system edit cm kubelet-config
# add the same `tracing:` block under the `kubelet:` key
```
## Full disable (live cluster)
### kube-apiserver
```bash
# Restore the pre-tracing manifest (preferred — exact revert)
sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
# OR remove just the flag in place:
sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml
# Optional: remove the now-unused config
sudo rm /etc/kubernetes/tracing-config.yaml
```
kubelet restarts the static pod automatically.
### kubelet (every node)
```bash
sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
```
Then strip the `tracing:` block from `kube-system/kubelet-config` CM so node rejoins don't reintroduce it.
## Verification (after enable)
1. Open `http://192.168.0.29:16686` — Services dropdown shows `apiserver` (appears after the first traced request) and `kubelet`.
2. Generate end-to-end traffic:
```bash
kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
kubectl delete pod trace-probe
```
3. In Jaeger: filter service `apiserver`, look for spans tagged with the probe pod name; filter service `kubelet`, look for `syncPod` spans.
4. To verify *disable*: after running disable steps, repeat (2). Within a minute, no new spans for `apiserver`/`kubelet` should appear in Jaeger.
## Notes
- 100% sampling is fine for short experiments; for steady-state leave tracing off (quick suppress is enough between sessions).
- The flag/config are GA in K8s 1.32 — no `--feature-gates` needed.
- This runbook only covers apiserver + kubelet. etcd and kube-scheduler tracing are documented as future steps in the parent plan.
```
## Files to change
| File | Change |
|---|---|
| `docs/kubernetes-tracing.md` | **new** — the runbook described above |
No other files modified. The Terraform/cloud-init side stays as already drafted in the parent plan.
## Verification of this plan's output
After the runbook is written:
1. Skim `docs/kubernetes-tracing.md` end-to-end and check that someone unfamiliar with the setup can follow it without opening the parent plan.
2. Cross-check the sed/tee commands against the actual files on the live master via SSH — confirm line patterns match (e.g., the `--tls-private-key-file=` line is present so the sed insertion lands in the right spot).
3. Dry-run the quick-suppress flow first (it's the lowest-risk path) and confirm Jaeger UI stops receiving new spans within ~30 s.

91
docs/plans/2026-05-21-2310-tracing-host-move-to-docker-29.md Normal file
View File

@@ -0,0 +1,91 @@
# Move Jaeger/OTLP host from 192.168.0.30 (docker-30) → 192.168.0.29 (docker-29)
## Context
The Jaeger backend that receives OTLP spans from kube-apiserver and kubelet has been relocated from the `docker-30` host (192.168.0.30) to `docker-29` (192.168.0.29). The new compose stack is already present at [vms-home/docker-29/tracing/](../../vms-home/docker-29/tracing/) and binds `0.0.0.0:4317`/`0.0.0.0:4318`, so the backend side requires no changes.
The OTLP exporter endpoints on the Kubernetes side, plus the runbooks/plans that document them, still point at `192.168.0.30:4317`. They need to be repointed to `192.168.0.29:4317`. Without this change, apiserver and kubelet spans go nowhere once docker-30's Jaeger is shut down.
The other docker-30 references in the repo (Vault, Gitea, Zot registry mirror, Kanidm, nginx) are unrelated to tracing and must be left alone.
## Scope
In-repo file edits only. Live cluster updates are documented as a follow-up checklist; no live execution is part of this plan.
## File-level changes
### 1. Live OTEL config (load-bearing)
**[kubernetes-kvm-terraform/files/tracing-config.yaml:3](../../kubernetes-kvm-terraform/files/tracing-config.yaml#L3)**
- `endpoint: 192.168.0.30:4317` → `endpoint: 192.168.0.29:4317`
**[kubernetes-kvm-terraform/master.tf](../../kubernetes-kvm-terraform/master.tf)** — two occurrences inside cloud-init heredocs:
- Line 122 (apiserver `TracingConfiguration` written to `/etc/kubernetes/tracing-config.yaml`): `192.168.0.30:4317` → `192.168.0.29:4317`
- Line 169 (`KubeletConfiguration.tracing.endpoint` in the kubeadm config): `192.168.0.30:4317` → `192.168.0.29:4317`
- **Do NOT touch line 8** (`zot_registry_ip = "192.168.0.30"`) — that's the Zot container-registry mirror, not tracing.
### 2. Runbook docs
**[docs/kubernetes-tracing.md](../kubernetes-tracing.md)** — replace tracing-related references throughout (≈10 lines):
- All occurrences of `192.168.0.30:4317` → `192.168.0.29:4317` (lines 54, 83, 99)
- `http://192.168.0.30:16686` → `http://192.168.0.29:16686` (line 150)
- SSH targets `novakj@192.168.0.30` → `novakj@192.168.0.29` (lines 25, 31, 43)
- Remote directory `~/docker-30/tracing` → `~/docker-29/tracing` (lines 25, 31, 43)
- Prose "Jaeger on docker-30" → "Jaeger on docker-29" (lines 3, 40)
### 3. Plan files (historical record — also requested)
**[plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md](../../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md)** — ~12 references:
- `192.168.0.30:4317` (lines 38, 76, 101, 111) and `http://192.168.0.30:16686` (line 141) → `.29`
- `docker-30/tracing/` directory path (lines 13, 16, 117, 128, 155) → `docker-29/tracing/`
- Prose mentions of "docker-30" in tracing context (lines 5, 11) → "docker-29"
**[docs/plans/2026-05-21-2035-tracing-toggle-runbook.md](2026-05-21-2035-tracing-toggle-runbook.md)** — ~6 references:
- `192.168.0.30:4317` (lines 51, 73) and `http://192.168.0.30:16686` (line 110) → `.29`
- SSH target `novakj@192.168.0.30` (line 35) → `novakj@192.168.0.29`
- Prose mentions of "docker-30" in tracing context (lines 5, 42) → "docker-29"
### Search safety
Use a per-file targeted replace with surrounding context (e.g. `endpoint: 192.168.0.30:4317`, `novakj@192.168.0.30`, `docker-30/tracing`), not a blanket repo-wide `sed`. The leave-alone list is large (Zot, Vault, Gitea, Kanidm, nginx, gitignore, claude settings, generated tfstate, commented-out blackbox/prometheus examples) — see Phase 1 exploration for the full inventory.
## Verification
### In-repo
```bash
# Confirm no tracing-related .30 references remain
grep -nE '192\.168\.0\.30:(4317|16686)' -r .
grep -nE 'docker-30/tracing' -r .
# Confirm new IP is in place
grep -nE '192\.168\.0\.29:4317' kubernetes-kvm-terraform/
```
Both `grep` calls in step 1 should return empty. The leave-alone Zot/Vault/Gitea/nginx hits at port 3000/8200/9443/etc. should still be present (sanity check that the surgical replace didn't over-match).
Re-run `tofu plan` in [kubernetes-kvm-terraform/](../../kubernetes-kvm-terraform/) to confirm the diff only shows the two endpoint changes and no unrelated drift.
### Live cluster follow-up (out of scope for this plan's execution, documented for the operator)
The in-repo changes only affect newly-bootstrapped nodes. The existing master + workers still have the old endpoint baked into their live files. After merging the repo changes, run:
1. **kube-apiserver** (master, 192.168.0.31):
```bash
ssh novakj@192.168.0.31 'sudo sed -i s/192.168.0.30:4317/192.168.0.29:4317/ /etc/kubernetes/tracing-config.yaml'
# kubelet auto-restarts the static pod within ~10s
```
2. **kubelet** on each node (192.168.0.31, .32, .33), serially:
```bash
ssh novakj@$NODE 'sudo sed -i s/192.168.0.30:4317/192.168.0.29:4317/ /var/lib/kubelet/config.yaml && sudo systemctl restart kubelet'
kubectl wait node --for=condition=Ready --timeout=60s <node-name>
```
3. **Cluster ConfigMap** so future joins/reboots pick it up:
```bash
kubectl -n kube-system edit cm kubelet-config
# update the tracing.endpoint value under kubelet:
```
4. **End-to-end smoke test** (from [docs/kubernetes-tracing.md](../kubernetes-tracing.md) "Verification" section):
```bash
kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
kubectl delete pod trace-probe
```
Then open `http://192.168.0.29:16686` and confirm `apiserver` and `kubelet` services show recent spans for `trace-probe`.

165
docs/plans/2026-07-08-0034-deploy-mimir.md Normal file
View File

@@ -0,0 +1,165 @@
# Deploy Mimir on docker-30 (mirror the Loki setup)
## Context
Loki is already deployed on the docker host (`192.168.0.30`) under
`vms-home/docker-30/loki/`, storing chunks in Garage S3 and queried through a
shared Grafana. The user wants Mimir deployed for **metrics** in the same style:
self-contained compose dir, Garage S3 backend, `.env`-supplied credentials,
env-expanded config.
Decisions made:
- **Mimir only for now.** No producer is wired up in this plan; Mimir simply
exposes its remote_write endpoint (`http://192.168.0.30:9009/api/v1/push`) for
a Prometheus/Alloy to point at later.
- **Reuse the existing Grafana** (the one in the Loki compose) by adding a
Prometheus-type datasource — no second Grafana.
Mimir runs in **monolithic mode** (`-target=all`, single binary), which is the
right shape for a single-node home lab and matches Loki's single-process model.
## Reference: how Loki does it (the pattern to copy)
`vms-home/docker-30/loki/loki-config.yaml` + `docker-compose.yaml`:
- Garage S3 at `192.168.0.30:3900`, `region: garage`, `s3forcepathstyle: true`,
`insecure: true`, bucket `loki-chunks`.
- Creds via `${GARAGE_ACCESS_KEY}` / `${GARAGE_SECRET_KEY}` from `.env`,
expanded with `-config.expand-env=true`.
- `obs` bridge network; Grafana published on `3001`.
## Files to create (new dir `vms-home/docker-30/mimir/`)
### 1. `mimir-config.yaml` — monolithic + Garage S3
```yaml
multitenancy_enabled: false # single-tenant; Loki mirrors this (auth_enabled:false)
server:
http_listen_port: 9009
grpc_listen_port: 9095
log_level: info
common:
storage:
backend: s3
s3:
endpoint: 192.168.0.30:3900 # no scheme; insecure toggles http
region: garage # MUST byte-match s3_region in garage.toml
access_key_id: ${GARAGE_ACCESS_KEY}
secret_access_key: ${GARAGE_SECRET_KEY}
insecure: true # http on LAN
bucket_lookup_type: path # path-style — Garage requirement (== s3forcepathstyle)
blocks_storage:
s3: { bucket_name: mimir-blocks }
tsdb: { dir: /data/tsdb }
bucket_store:{ sync_dir: /data/tsdb-sync }
ruler_storage:
s3: { bucket_name: mimir-ruler }
alertmanager_storage:
s3: { bucket_name: mimir-alertmanager }
compactor:
data_dir: /data/compactor
ruler:
rule_path: /data/ruler
alertmanager:
data_dir: /data/alertmanager
limits:
compactor_blocks_retention_period: 744h # 31d, matches Loki's retention
ingestion_rate: 50000 # samples/s per tenant; bump on 429s
ingestion_burst_size: 100000
```
Note: the three storage components inherit endpoint/creds from `common.storage`;
only `bucket_name` is overridden per component. Buckets can be collapsed to one
if preferred, but three is cleaner and Garage bucket creation is cheap.
### 2. `docker-compose.yaml` — copy Loki's, swap image/ports/paths
```yaml
services:
mimir:
image: grafana/mimir:2.16.1 # pin latest stable — verify tag before apply
container_name: mimir
restart: unless-stopped
command:
- -config.file=/etc/mimir/config.yaml
- -config.expand-env=true # REQUIRED — expands ${GARAGE_*}
- -target=all # monolithic single-binary mode
ports:
- "9009:9009" # push (/api/v1/push) + query (/prometheus)
volumes:
- ./mimir-config.yaml:/etc/mimir/config.yaml:ro
- ./mimir-data:/data
env_file: [.env]
networks: [obs]
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://localhost:9009/ready || exit 1"]
interval: 15s
timeout: 5s
retries: 10
start_period: 30s
logging:
driver: json-file
options: { max-size: "10m", max-file: "3" }
networks:
obs: { driver: bridge }
```
### 3. `.env` — reuse the same Garage credentials as Loki
```
GARAGE_ACCESS_KEY=<same key id Loki uses>
GARAGE_SECRET_KEY=<same secret>
```
## Manual steps on docker-30 (execution)
1. **Create Garage buckets** and grant the existing Loki key access:
```bash
ssh novakj@192.168.0.30
for b in mimir-blocks mimir-ruler mimir-alertmanager; do
docker exec garage /garage bucket create "$b"
done
docker exec garage /garage key list # get the key name Loki already uses
for b in mimir-blocks mimir-ruler mimir-alertmanager; do
docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
done
```
2. Copy the `mimir/` dir to the host (or `git pull` if this repo is checked out
there), fill `.env`, then `cd mimir && docker compose up -d`.
3. **Add Mimir datasource to the existing Grafana** (`:3001`):
- Type: **Prometheus**
- URL: `http://192.168.0.30:9009/prometheus`
- No `X-Scope-OrgID` header needed (multitenancy disabled).
## Verification
- `curl http://192.168.0.30:9009/ready` → `ready` (also gates the healthcheck).
- `docker compose ps` shows `mimir` healthy; `docker compose logs -f mimir`
clean (no S3 auth / bucket errors — confirms Garage wiring).
- Smoke-test the write path without a producer:
```bash
curl http://192.168.0.30:9009/api/v1/push # expect 4xx (proto body), NOT conn-refused
```
- In Grafana → Explore → Mimir datasource, run `up` or
`count({__name__!=""})`. Empty until a producer remote_writes — expected.
- Confirm blocks land in Garage after ~2h (first block flush):
`docker exec garage /garage bucket info mimir-blocks`.
## Later (out of scope here): wiring a producer
Point any Prometheus/Alloy at:
```yaml
remote_write:
- url: http://192.168.0.30:9009/api/v1/push
```
The existing `vms/utility-101-shadow/docker/monitoring/prometheus.yml` is the
natural first candidate.

116
docs/plans/2026-07-08-2346-alloy-metrics-to-mimir.md Normal file
View File

@@ -0,0 +1,116 @@
# Alloy → Mimir: scrape Kubernetes metrics + Prometheus Operator monitors
> On execution, also copy this file into the repo per CLAUDE.md convention:
> `docs/plans/2026-07-08-HHMM-alloy-metrics-to-mimir.md` (get `HHMM` via `date "+%H%M"`).
## Context
Mimir is now up on docker-30 (`http://192.168.0.30:9009`, single-tenant, plain
HTTP over LAN) but nothing in the cluster pushes metrics to it. Grafana Alloy
already exists in the repo but is **logs-only** (ships pod/journal logs + k8s
events to Loki) and — importantly — the entire `gitops/home-kubernetes/alloy/`
directory is **not wired into Flux** yet.
kube-prometheus-stack is deployed and active: it provides the Prometheus
Operator CRDs (ServiceMonitor / PodMonitor), kube-state-metrics, node-exporter,
and ServiceMonitors for all standard targets (kubelet, cAdvisor, apiserver,
coredns, etc.). Its bundled Prometheus scrapes everything into a local 60d TSDB
but does **not** remote_write anywhere.
**Goal:** add a dedicated `alloy-metrics` instance that scrapes standard
Kubernetes metrics and reads ServiceMonitors/PodMonitors from the Prometheus
Operator, remote-writing to Mimir — additive alongside the existing Prometheus
(no changes to kube-prometheus-stack). Because kube-prometheus-stack already
ships ServiceMonitors for every standard target, a single
`prometheus.operator.servicemonitors` + `prometheus.operator.podmonitors` pair
satisfies **both** requirements at once. As part of wiring the `alloy/` dir into
Flux, the existing `alloy-logs` and `alloy-events` releases also come under
GitOps management (confirmed desired).
**Decisions confirmed:** additive/coexist with bundled Prometheus; activate all
three Alloy instances (logs, events, metrics).
## Design
- Metrics scraping runs as a **single Deployment** (`controller.type: deployment`,
1 replica) — never a DaemonSet, or every cluster-wide target would be scraped
once per node. This is why it must be a separate release from `alloy-logs`
(which is a DaemonSet).
- Mirror the conventions already in `helmrelease_alloy-logs.yaml`: chart `alloy`
`>=1.8.0 <2.0.0`, image `tag: v1.17.1`, `rbac.create: false`,
`alloy.enableReporting: false`, `driftDetection: enabled`, install/upgrade
`retries: 3`, `interval: 30m`, namespace `monitoring`.
- River config = three components:
1. `prometheus.operator.servicemonitors "sm"` — discovers/scrapes **all**
ServiceMonitors cluster-wide (covers kubelet, cAdvisor, kube-state-metrics,
node-exporter, apiserver, coredns, controller-manager, scheduler — i.e. the
"standard kubernetes metrics").
2. `prometheus.operator.podmonitors "pm"` — discovers/scrapes all PodMonitors.
3. `prometheus.remote_write "mimir"` → `http://192.168.0.30:9009/api/v1/push`,
`external_labels = { cluster = "homelab" }` (matches the logs config).
Both operator components `forward_to = [prometheus.remote_write.mimir.receiver]`.
- Reuse `cluster = "homelab"` external label for consistency with the Loki side.
## Files
### New — `gitops/home-kubernetes/alloy/helmrelease_alloy-metrics.yaml`
HelmRelease `alloy-metrics` (ns `monitoring`), Deployment, serviceAccount
`alloy-metrics`, inline `alloy.configMap.content` with the three components
above. Modest resources (e.g. requests cpu 100m / mem 256Mi, limit mem 768Mi —
operator discovery + scrape buffers use more than the log shipper).
### New — `gitops/home-kubernetes/alloy/rbac_metrics-collector.yaml`
The `prometheus.operator.*` components need read access beyond the existing
`alloy-log-reader` role. ClusterRole `alloy-metrics-reader` + ClusterRoleBinding
to SA `alloy-metrics` (monitoring), granting:
- core: `namespaces, nodes, nodes/metrics, nodes/proxy, services, endpoints, pods` — get/list/watch
- `discovery.k8s.io`: `endpointslices` — get/list/watch
- `networking.k8s.io`: `ingresses` — get/list/watch
- `monitoring.coreos.com`: `servicemonitors, podmonitors, probes, scrapeconfigs` — get/list/watch
- nonResourceURLs `/metrics`, `/metrics/cadvisor` — get (kubelet scraping)
(Kept as a separate file so the metrics RBAC is self-contained; the existing
`rbac_log-collector.yaml` stays untouched.)
### New — `gitops/home-kubernetes/alloy/kustomization.yaml`
`kind: Kustomization` listing all manifests in the dir: `helmrepository_grafana.yaml`,
`helmrelease_alloy-logs.yaml`, `helmrelease_allow-events.yaml`,
`helmrelease_alloy-metrics.yaml`, `rbac_log-collector.yaml`,
`rbac_metrics-collector.yaml`. Makes the Flux build explicit/deterministic.
### Modified — `gitops/home-kubernetes/flux-system/extra-kustomizations.yaml`
Append an `alloy` Flux Kustomization:
```yaml
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: alloy
namespace: flux-system
spec:
interval: 10m0s
path: ./gitops/home-kubernetes/alloy
prune: true
sourceRef:
kind: GitRepository
name: flux-system
dependsOn:
- name: kube-prometheus # needs the monitoring ns + operator CRDs/ServiceMonitors
```
## Verification
1. Commit + push; force reconcile: `flux reconcile kustomization flux-system --with-source`.
2. `kubectl get kustomizations -A` — `alloy` Ready=True.
3. `kubectl get helmreleases -n monitoring` — `alloy-metrics` (+ logs/events) Ready.
4. `kubectl -n monitoring get pods -l app.kubernetes.io/instance=alloy-metrics`
Running; `kubectl -n monitoring logs deploy/alloy-metrics` shows no RBAC
`forbidden` errors and targets discovered.
5. Alloy UI: `kubectl -n monitoring port-forward deploy/alloy-metrics 12345:12345`
→ open `http://localhost:12345`; confirm the two operator components list
healthy targets and `prometheus.remote_write.mimir` shows successful sends.
6. Confirm Mimir is receiving: query it directly, e.g.
`curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=up' | jq '.data.result | length'`
returns > 0 (or check the Grafana Mimir datasource for `up`,
`kube_pod_info`, `node_cpu_seconds_total`, `container_cpu_usage_seconds_total`).
7. Sanity: a `cluster="homelab"` label is present on the ingested series.