chore: add CLAUDE.md, kubectl notes, and settings/gitignore tweaks

Document the repo for Claude Code, note a couple of handy kubectl one-liners, allow a few more Bash/WebFetch/WebSearch permissions used during this session, and ignore the relocated zot sync-credentials.json. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
servers/shadow: open GitLab SSH port, add docker-dev/czechultimate nginx routes
2026-07-09 00:05:56 +02:00 · 2026-07-09 00:05:49 +02:00 · 2026-07-09 00:05:42 +02:00 · 2026-07-09 00:05:39 +02:00 · 2026-07-09 00:05:30 +02:00 · 2026-07-09 00:05:21 +02:00
92 changed files with 2819 additions and 48 deletions
--- a/.claude/settings.json
+++ b/.claude/settings.json
@@ -4,7 +4,11 @@
      "Bash(for f:*)",
      "Bash(do echo:*)",
      "Read(//Users/jan.novak/srv/personal/home-kubernetes/**)",
-      "Bash(done)"
+      "Bash(done)",
      "Bash(ssh docker-30 *)",
      "Bash(git add *)",
      "WebSearch",
      "WebFetch(domain:cert-manager.io)"
    ]
  }
 }
--- a/.gitignore
+++ b/.gitignore
@@ -11,4 +11,5 @@ tmp/
 vms/utility-101-shadow/docker/monitoring/smtp_password
 docker-30/zot/sync-credentials.json
 vms-home/docker-30/zot/sync-credentials.json
 kubernetes-kvm-terraform/gke_gcloud_auth_plugin_cache
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -0,0 +1,105 @@
 # CLAUDE.md
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 ## Repository Purpose
 Home Kubernetes lab infrastructure-as-code. Manages VM provisioning (Terraform/KVM), Kubernetes cluster configuration, and application deployments via Flux GitOps — all pointing at a self-hosted Gitea instance.
 ## Common Commands
 ### Terraform (kubernetes-kvm-terraform/)
 ```bash
 cd kubernetes-kvm-terraform
 tofu init
 tofu plan
 tofu apply
 tofu destroy
 ```
 ### Flux GitOps
 ```bash
 # Check reconciliation status
 kubectl get kustomizations -A
 kubectl get helmreleases -A
 # Force reconcile
 flux reconcile kustomization flux-system --with-source
 flux reconcile helmrelease <name> -n <namespace>
 # Watch logs
 flux logs --follow
 # Check source sync
 flux get sources git
 ```
 ### kubectl — common ops
 ```bash
 export KUBECONFIG=kubernetes-kvm-terraform/kubeconfig
 kubectl get nodes
 kubectl get pods -A
 kubectl get secrets -A
 ```
 ### Docker Compose (docker-30/ services)
 ```bash
 # These run on 192.168.0.30 (docker-30), accessed via SSH
 ssh novakj@192.168.0.30
 cd /path/to/service && docker compose up -d
 docker compose logs -f
 ```
 ## Architecture
 ### Infrastructure Layer
 - **Hypervisors**: homer (192.168.0.7) and beelink (192.168.0.6) run KVM/libvirt
 - **Terraform provider**: primary `qemu+ssh://novakj@192.168.0.7/system`, secondary alias `kvm-beelink` for 192.168.0.6
 - **Kubernetes master**: kube-master-31 @ 192.168.0.31:6443, bootstrapped via kubeadm
 - **OS**: Ubuntu 24.04 Noble cloud images, network bridge `br0`, subnet 192.168.0.0/24
 ### GitOps Layer (`gitops/home-kubernetes/`)
 Flux syncs from Gitea (`https://gitea.home.hrajfrisbee.cz`, main branch, every 10 minutes). The reconciliation order is enforced via `dependsOn` in `flux-system/extra-kustomizations.yaml`:
 ```
 00-crds → 00-rbac → cilium → cert-manager → external-secrets → everything else
 ```
 Each application lives in its own subdirectory under `gitops/home-kubernetes/` and is referenced as a Flux `Kustomization` resource. Prune is enabled — removing a manifest from git removes it from the cluster.
 ### Secrets Flow
 Vault (docker-30) → External-Secrets controller (in-cluster) → Kubernetes `Secret` objects. Applications reference `ExternalSecret` CRs that pull from Vault paths. Do not put real secrets in git.
 ### Networking
 - **CNI**: Cilium 1.19.x with Gateway API and Hubble UI enabled
 - **L2 LB**: Cilium `CiliumL2AnnouncementPolicy` + `CiliumLoadBalancerIPPool` for bare-metal load-balancer IPs (defined in `gitops/home-kubernetes/cilium/`)
 - **Ingress**: ingress-nginx for HTTP(S) workloads; Gateway API for newer apps
 - **TLS**: cert-manager issues wildcard cert (`*.home.hrajfrisbee.cz`) referenced by apps
 ### Storage
 - **democratic-CSI**: iSCSI volumes backed by FreeNAS at 192.168.0.40
 - **Longhorn**: configured but disabled in Flux (directory present, not in `extra-kustomizations.yaml`)
 ### Supporting Services on docker-30 (192.168.0.30)
 All managed with Docker Compose:
 - **Gitea** — Git server + act_runner (GitHub Actions-compatible CI)
 - **Vault** — secrets backend for External Secrets
 - **Zot** — private OCI/container registry
 - **Kanidm** — identity management / OIDC provider
 - **nginx** — reverse proxy for docker-30 services
 ## Key Conventions
 - **New application**: create a directory under `gitops/home-kubernetes/<app-name>/`, add a `Kustomization` entry in `flux-system/extra-kustomizations.yaml` with appropriate `dependsOn`.
 - **Helm apps**: use a `HelmRepository` + `HelmRelease` pair; pin chart versions explicitly.
 - **Secrets**: add an `ExternalSecret` CR pointing to the Vault path; never commit actual secret values.
 - **Terraform state**: `kubernetes-kvm-terraform/terraform.tfstate*` is gitignored/sensitive; treat it carefully.
 - **kubeconfig**: `kubernetes-kvm-terraform/kubeconfig` is gitignored; obtain it from the master node after provisioning.
 ## Plans
 When Claude Code's plan mode is used, save the plan file inside the repo at
 `docs/plans/YYYY-MM-DD-HHMM-<slug>.md` instead of the default `~/.claude/plans/`
 location. Get the timestamp with `date "+%Y-%m-%d-%H%M"` (matches the changelog
 convention). The `<slug>` should be a short kebab-case summary of the plan's topic.
--- a/docker-30/readme.md
+++ b/docker-30/readme.md
@@ -1,17 +0,0 @@
 # docker-30
 ## taiscale
 ```bash
 # Add signing key
 curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/$(lsb_release -cs).noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
 # Add repo
 echo "deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/tailscale.list
 # Install
 sudo apt update && sudo apt install tailscale
 # Start
 sudo tailscale up
 ```
--- a/docs/kubernetes-extras.md
+++ b/docs/kubernetes-extras.md
@@ -0,0 +1,14 @@
 ## kubectl
 ```bash
 # condensed -o wide
 kubectl get pods -A -o wide --watch | awk '
 /NOMINATED NODE/ { sub(/[[:space:]]+NOMINATED NODE[[:space:]]+READINESS GATES[[:space:]]*$/,""); print; fflush(); next }
 { sub(/[[:space:]]+[^[:space:]]+[[:space:]]+[^[:space:]]+[[:space:]]*$/,""); print; fflush() }
 '
 # condensed -o wide 
 kubectl get pods -A --watch -o custom-columns=\
 'NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase,RESTARTS:.status.containerStatuses[0].restartCount,IP:.status.podIP,NODE:.spec.nodeName'
 ```
--- a/docs/kubernetes-tracing.md
+++ b/docs/kubernetes-tracing.md
@@ -0,0 +1,180 @@
 # Kubernetes component tracing
 OpenTelemetry tracing for kube-apiserver and kubelet, shipping OTLP spans to Jaeger on docker-29. Useful for measuring pod-creation latency broken down by component.
 Related: [plans/k8s-pod-creation-tracing.md](../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md) — setup rationale, storage sizing, etcd tracing notes.
 ---
 ## Where the toggles live
 | Component | Config on node | Restart trigger |
 |---|---|---|
 | kube-apiserver | `/etc/kubernetes/manifests/kube-apiserver.yaml` (flag) + `/etc/kubernetes/tracing-config.yaml` | kubelet detects manifest change, auto-restarts static pod (~5–10 s) |
 | kubelet | `/var/lib/kubelet/config.yaml` (`tracing:` block) + `kube-system/kubelet-config` CM | `systemctl restart kubelet` |
 Nodes: kube-master-31 @ 192.168.0.31, kube-node-32 @ 192.168.0.32, kube-node-33 @ 192.168.0.33.
 ---
 ## Quick suppress (no component restart)
 Stop the Jaeger backend — components keep building spans but have nowhere to send them. Lowest-risk; use this between measurement sessions.
 ```bash
 ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose stop jaeger'
 ```
 Resume:
 ```bash
 ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose start jaeger'
 ```
 Caveat: components still pay the (small) cost of building spans. For true zero-overhead, use **Full disable** below.
 ---
 ## Full enable (live cluster)
 ### 1. Start Jaeger on docker-30
 ```bash
 ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose up -d'
 ```
 ### 2. kube-apiserver (master only — 192.168.0.31)
 ```bash
 ssh novakj@192.168.0.31 bash -s <<'EOF'
 # TracingConfiguration file
 sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'YAML'
 apiVersion: apiserver.config.k8s.io/v1beta1
 kind: TracingConfiguration
 endpoint: 192.168.0.29:4317
 samplingRatePerMillion: 1000000
 YAML
 # Back up the manifest before touching it
 sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing
 # Add the flag (after --tls-private-key-file line)
 sudo sed -i '/--tls-private-key-file=/a\    - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' \
    /etc/kubernetes/manifests/kube-apiserver.yaml
 # Wait for static pod to restart
 sleep 10
 sudo crictl ps | grep apiserver
 kubectl get --raw /livez
 EOF
 ```
 ### 3. kubelet (all three nodes)
 Run on each node in turn — wait for `Ready` before moving to the next to avoid simultaneous NotReady:
 ```bash
 for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
  echo "==> $NODE"
  ssh ubuntu@$NODE bash -s <<'EOF'
 grep -q '^tracing:' /var/lib/kubelet/config.yaml && echo "already enabled, skipping" && exit 0
 sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'YAML'
 tracing:
  endpoint: 192.168.0.29:4317
  samplingRatePerMillion: 1000000
 YAML
 sudo systemctl restart kubelet
 EOF
  # wait for node Ready before proceeding
  kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
 done
 ```
 ### 4. Update the cluster ConfigMap (so future joins/reboots pick it up)
 ```bash
 kubectl -n kube-system edit cm kubelet-config
 # under the `kubelet:` key, add:
 #   tracing:
 #     endpoint: 192.168.0.29:4317
 #     samplingRatePerMillion: 1000000
 ```
 ---
 ## Full disable (live cluster)
 ### kube-apiserver (master only)
 ```bash
 ssh novakj@192.168.0.31 bash -s <<'EOF'
 # Restore the pre-tracing manifest (exact revert)
 sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
 # Optional: remove the config file
 sudo rm -f /etc/kubernetes/tracing-config.yaml
 EOF
 ```
 Static pod restarts automatically. If the backup wasn't taken, remove the flag in-place instead:
 ```bash
 sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml
 ```
 ### kubelet (all three nodes)
 ```bash
 for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
  echo "==> $NODE"
  ssh novakj@$NODE bash -s <<'EOF'
 sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
 sudo systemctl restart kubelet
 EOF
  kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
 done
 ```
 Remove the tracing block from the cluster ConfigMap too:
 ```bash
 kubectl -n kube-system edit cm kubelet-config
 # remove the `tracing:` block under `kubelet:`
 ```
 ---
 ## Verification
 ### After enable
 1. Open Jaeger UI: `http://192.168.0.29:16686` — Services dropdown should show `apiserver` and `kubelet` (appear after first traced request).
 2. Generate a full pod-lifecycle trace:
   ```bash
   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   ```
 3. In Jaeger UI, search:
   - Service `apiserver` — look for spans covering the create/watch flow for `trace-probe`.
   - Service `kubelet` — look for `syncPod` spans tied to the same pod.
 4. The gap between apiserver spans and kubelet spans is scheduler time (kube-scheduler has no OTLP tracing in 1.32). Cross-check with `kubectl get events` or pod condition timestamps.
 ### After disable
 Repeat step 2 above, then confirm no new spans appear in Jaeger for `apiserver` or `kubelet` within ~1 minute.
 ---
 ## Sampling rate cheatsheet
 Change `samplingRatePerMillion` in `/etc/kubernetes/tracing-config.yaml` (apiserver) and `/var/lib/kubelet/config.yaml` (kubelet) then restart each component.
 | Value | Effective rate | Use case |
 |---|---|---|
 | `1000000` | 100% | Active measurement |
 | `10000` | 1% | Steady-state background |
 | `0` | off | Disable without removing config |
--- a/docs/kubernetes/alloy.md
+++ b/docs/kubernetes/alloy.md
@@ -0,0 +1,142 @@
 # Alloy: deployment & operations
 Three Grafana Alloy instances run in the `monitoring` namespace, all deployed via
 the Grafana `alloy` Helm chart (`>=1.8.0 <2.0.0`, image `v1.17.1`) and managed by
 Flux with `driftDetection: enabled` (manual `kubectl edit`s get reverted — always
 change config via git). Logs and events ship to Loki; metrics ship to Mimir. Every
 instance stamps `cluster="homelab"` as an external label.
 ---
 ## Instances
 | Instance | Controller | Collects | Sink |
 |---|---|---|---|
 | `alloy-logs` | DaemonSet (`runAsUser: 0`, one pod per node) | pod logs + node journald | Loki `http://192.168.0.30:3100/loki/api/v1/push` |
 | `alloy-events` | Deployment (1 replica) | Kubernetes events | Loki (same endpoint) |
 | `alloy-metrics` | Deployment (1 replica) | all ServiceMonitors + PodMonitors cluster-wide | Mimir `http://192.168.0.30:9009/api/v1/push` |
 `alloy-logs` and `alloy-events` are one-per-node / cluster-scoped respectively —
 `alloy-metrics` is also a single Deployment, never a DaemonSet, since its
 Prometheus-Operator discovery is already cluster-wide (a DaemonSet would scrape
 every target once per node).
 ---
 ## Files & Flux wiring
 All manifests live in `gitops/home-kubernetes/alloy/`:
 - `helmrepository_grafana.yaml` — the shared `grafana` HelmRepository.
 - `helmrelease_alloy-logs.yaml` — logs DaemonSet.
 - `helmrelease_allow-events.yaml` — events Deployment (note the `allow` filename typo, kept as-is).
 - `helmrelease_alloy-metrics.yaml` — metrics Deployment.
 - `rbac_log-collector.yaml` — ClusterRole `alloy-log-reader`, bound to the `alloy-logs`/`alloy-events` ServiceAccounts.
 - `rbac_metrics-collector.yaml` — ClusterRole `alloy-metrics-reader`, bound to the `alloy-metrics` ServiceAccount.
 Each HelmRelease sets `rbac.create: false` — RBAC is supplied manually by the two
 files above rather than by the chart, since the chart's default ClusterRole
 doesn't cover everything each pipeline needs (journald/events reads, or
 ServiceMonitor/PodMonitor/kubelet-metrics reads).
 Reconciliation: a Flux `Kustomization` named `alloy` in
 `gitops/home-kubernetes/flux-system/extra-kustomizations.yaml`, pointing at
 `./gitops/home-kubernetes/alloy`, with `dependsOn: kube-prometheus` — it needs
 the `monitoring` namespace and the Prometheus Operator CRDs (ServiceMonitor/
 PodMonitor) to exist first.
 ---
 ## How metrics discovery works
 `alloy-metrics` runs two Alloy components — `prometheus.operator.servicemonitors`
 and `prometheus.operator.podmonitors` — which discover and scrape **every**
 ServiceMonitor/PodMonitor in the cluster, then forward to
 `prometheus.remote_write` targeting Mimir. There's no bespoke scrape config for
 "standard Kubernetes metrics": kube-prometheus-stack already ships
 ServiceMonitors for kubelet, cAdvisor, kube-state-metrics, node-exporter,
 apiserver, and coredns, so those are picked up automatically. Any future
 ServiceMonitor/PodMonitor added anywhere in the cluster is picked up the same way
 with no Alloy config change.
 This is **additive** — kube-prometheus-stack's bundled Prometheus keeps scraping
 the same targets into its own local 60d TSDB. Alloy scraping the same targets a
 second time and shipping to Mimir is expected double-collection, not a bug.
 ---
 ## Operating
 Status:
 ```bash
 flux get helmreleases -n monitoring
 kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy -o wide
 ```
 Force reconcile:
 ```bash
 flux reconcile kustomization alloy --with-source
 flux reconcile helmrelease alloy-metrics -n monitoring
 ```
 Logs:
 ```bash
 kubectl -n monitoring logs deploy/alloy-metrics
 kubectl -n monitoring logs ds/alloy-logs
 kubectl -n monitoring logs deploy/alloy-events
 ```
 Alloy UI (component graph, target health, remote_write queue status):
 ```bash
 kubectl -n monitoring port-forward deploy/alloy-metrics 12345:12345
 # open http://localhost:12345
 ```
 Editing config: change the inline `alloy.configMap.content` block in the
 relevant `helmrelease_*.yaml`, commit, push, then force-reconcile (above).
 Do not `kubectl edit` the generated ConfigMap directly — `driftDetection` will
 revert it on the next reconcile.
 ---
 ## Verification / smoke tests
 **Metrics reaching Mimir:**
 ```bash
 curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=up' | jq '.data.result | length'
 # expect > 0
 ```
 Spot-check a few standard series exist and carry the cluster label:
 ```bash
 curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=kube_pod_info' | jq '.data.result[0].metric'
 curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=node_cpu_seconds_total' | jq '.data.result | length'
 curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=container_cpu_usage_seconds_total' | jq '.data.result | length'
 # confirm "cluster": "homelab" is present in the returned label sets
 ```
 **Logs reaching Loki:**
 ```bash
 curl -s http://192.168.0.30:3100/ready
 # then in Grafana / logcli, query {cluster="homelab"}
 ```
 ---
 ## Troubleshooting
 | Symptom | Likely cause |
 |---|---|
 | `forbidden` errors in `alloy-metrics` logs | `alloy-metrics-reader` ClusterRole missing a resource/verb — check `rbac_metrics-collector.yaml` against what the component is trying to read |
 | No targets shown in the Alloy UI | Prometheus Operator CRDs not installed, or `kube-prometheus` Flux Kustomization not Ready yet (metrics depends on it) |
 | `prometheus.remote_write` shows failed sends / 4xx-5xx | Mimir down, or wrong endpoint — must be `http://192.168.0.30:9009/api/v1/push` exactly |
 | Loki push failing | Loki down at `192.168.0.30:3100`, check with `curl http://192.168.0.30:3100/ready` |
 | Same series appearing to be scraped twice | Expected — kube-prometheus-stack's bundled Prometheus and `alloy-metrics` both scrape the same ServiceMonitors by design (additive, not deduplicated) |
 | Manual `kubectl edit` on the Alloy ConfigMap reverts itself | `driftDetection: enabled` — edit the HelmRelease in git instead |
--- a/docs/plans/2026-05-21-2035-tracing-toggle-runbook.md
+++ b/docs/plans/2026-05-21-2035-tracing-toggle-runbook.md
@@ -0,0 +1,140 @@
 # Tracing enable/disable runbook for K8s components
 ## Context
 The pod-creation tracing plan ([plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md](../../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md)) covers the *one-time setup* of OTLP tracing across kube-apiserver and kubelet, plus the Jaeger backend on docker-29. What's missing is a short operational doc for **toggling tracing on/off as quickly as possible** on the running cluster — useful for short measurement experiments where we want overhead only when we're actively recording.
 The Terraform/cloud-init changes already in flight (`master.tf`, `files/manifests/kube-apiserver.yaml`, `files/tracing-config.yaml`) cover *rebuilds*. This runbook covers the *live* cluster: kube-master-31 + kube-node-32 + kube-node-33, K8s 1.32.
 Output of this plan: a single new file `docs/kubernetes-tracing.md` containing three runbooks (quick suppress, full enable, full disable) and a verification section. No code or manifest changes — pure documentation.
 ## What "as quickly as possible" means
 There are two speeds of toggling:
 1. **Quick suppress** — leave the tracing flags/config in place on the components; just stop Jaeger so span exports go nowhere. No K8s component restart, takes seconds, but components still pay the (small) cost of building spans and attempting to export.
 2. **Full enable / full disable** — edit configs and restart components. apiserver static-pod restart is automatic via kubelet manifest watch (~5–10s); kubelet itself needs `systemctl restart kubelet` on each node (brief NotReady blip per node, no pod evictions). This is what we'd use to truly remove overhead between experiments.
 The runbook will present both, with quick-suppress as the default for "I'm done measuring for now" and full-disable for "we don't need this on the cluster anymore".
 ## Deliverable: `docs/kubernetes-tracing.md`
 ### Structure
 ```
 # Kubernetes component tracing
 ## Components and where the toggles live
 - table: component | config file on node | restart trigger
  - kube-apiserver | /etc/kubernetes/manifests/kube-apiserver.yaml (flag) +
                     /etc/kubernetes/tracing-config.yaml (config) | kubelet auto-restarts static pod
  - kubelet         | /var/lib/kubelet/config.yaml (`tracing:` block) +
                     kube-system/kubelet-config CM (cluster-wide source of truth) | systemctl restart kubelet
 ## Quick suppress (no component restart)
 1. `ssh novakj@192.168.0.29 'cd /path/to/tracing && docker compose stop jaeger'`
 2. Components continue running with tracing wired up but spans go nowhere.
 3. To resume: `docker compose start jaeger`.
 Caveat: components still build spans and attempt OTLP export — small CPU/network overhead. For a real "off", use full disable.
 ## Full enable (live cluster)
 Prereqs: Jaeger is up on docker-29 (`docker compose up -d` in `docker-29/tracing/`).
 ### kube-apiserver (master only)
 On 192.168.0.31:
 ```bash
 # 1. Drop the TracingConfiguration file (same content as files/tracing-config.yaml)
 sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'EOF'
 apiVersion: apiserver.config.k8s.io/v1beta1
 kind: TracingConfiguration
 endpoint: 192.168.0.29:4317
 samplingRatePerMillion: 1000000
 EOF
 # 2. Back up the apiserver manifest
 sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing
 # 3. Add the flag (insert after --tls-private-key-file line)
 sudo sed -i '/--tls-private-key-file=/a\    - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml
 # 4. Wait for kubelet to restart the static pod (~5-10s)
 sudo crictl ps | grep apiserver         # new container id == restarted
 kubectl get --raw /livez                 # should return "ok"
 ```
 ### kubelet (every node: master + workers)
 Per-node edit:
 ```bash
 # Append the tracing block under kubelet config (idempotent: check first)
 grep -q '^tracing:' /var/lib/kubelet/config.yaml || sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'EOF'
 tracing:
  endpoint: 192.168.0.29:4317
  samplingRatePerMillion: 1000000
 EOF
 sudo systemctl restart kubelet
 ```
 Stagger across nodes (one at a time) so we don't NotReady the whole cluster at once.
 Then make it survive node rebuilds/joins by updating the cluster ConfigMap:
 ```bash
 kubectl -n kube-system edit cm kubelet-config
 # add the same `tracing:` block under the `kubelet:` key
 ```
 ## Full disable (live cluster)
 ### kube-apiserver
 ```bash
 # Restore the pre-tracing manifest (preferred — exact revert)
 sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
 # OR remove just the flag in place:
 sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml
 # Optional: remove the now-unused config
 sudo rm /etc/kubernetes/tracing-config.yaml
 ```
 kubelet restarts the static pod automatically.
 ### kubelet (every node)
 ```bash
 sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
 sudo systemctl restart kubelet
 ```
 Then strip the `tracing:` block from `kube-system/kubelet-config` CM so node rejoins don't reintroduce it.
 ## Verification (after enable)
 1. Open `http://192.168.0.29:16686` — Services dropdown shows `apiserver` (appears after the first traced request) and `kubelet`.
 2. Generate end-to-end traffic:
   ```bash
   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   ```
 3. In Jaeger: filter service `apiserver`, look for spans tagged with the probe pod name; filter service `kubelet`, look for `syncPod` spans.
 4. To verify *disable*: after running disable steps, repeat (2). Within a minute, no new spans for `apiserver`/`kubelet` should appear in Jaeger.
 ## Notes
 - 100% sampling is fine for short experiments; for steady-state leave tracing off (quick suppress is enough between sessions).
 - The flag/config are GA in K8s 1.32 — no `--feature-gates` needed.
 - This runbook only covers apiserver + kubelet. etcd and kube-scheduler tracing are documented as future steps in the parent plan.
 ```
 ## Files to change
 | File | Change |
 |---|---|
 | `docs/kubernetes-tracing.md` | **new** — the runbook described above |
 No other files modified. The Terraform/cloud-init side stays as already drafted in the parent plan.
 ## Verification of this plan's output
 After the runbook is written:
 1. Skim `docs/kubernetes-tracing.md` end-to-end and check that someone unfamiliar with the setup can follow it without opening the parent plan.
 2. Cross-check the sed/tee commands against the actual files on the live master via SSH — confirm line patterns match (e.g., the `--tls-private-key-file=` line is present so the sed insertion lands in the right spot).
 3. Dry-run the quick-suppress flow first (it's the lowest-risk path) and confirm Jaeger UI stops receiving new spans within ~30 s.
--- a/docs/plans/2026-05-21-2310-tracing-host-move-to-docker-29.md
+++ b/docs/plans/2026-05-21-2310-tracing-host-move-to-docker-29.md
@@ -0,0 +1,91 @@
 # Move Jaeger/OTLP host from 192.168.0.30 (docker-30) → 192.168.0.29 (docker-29)
 ## Context
 The Jaeger backend that receives OTLP spans from kube-apiserver and kubelet has been relocated from the `docker-30` host (192.168.0.30) to `docker-29` (192.168.0.29). The new compose stack is already present at [vms-home/docker-29/tracing/](../../vms-home/docker-29/tracing/) and binds `0.0.0.0:4317`/`0.0.0.0:4318`, so the backend side requires no changes.
 The OTLP exporter endpoints on the Kubernetes side, plus the runbooks/plans that document them, still point at `192.168.0.30:4317`. They need to be repointed to `192.168.0.29:4317`. Without this change, apiserver and kubelet spans go nowhere once docker-30's Jaeger is shut down.
 The other docker-30 references in the repo (Vault, Gitea, Zot registry mirror, Kanidm, nginx) are unrelated to tracing and must be left alone.
 ## Scope
 In-repo file edits only. Live cluster updates are documented as a follow-up checklist; no live execution is part of this plan.
 ## File-level changes
 ### 1. Live OTEL config (load-bearing)
 **[kubernetes-kvm-terraform/files/tracing-config.yaml:3](../../kubernetes-kvm-terraform/files/tracing-config.yaml#L3)**
 - `endpoint: 192.168.0.30:4317` → `endpoint: 192.168.0.29:4317`
 **[kubernetes-kvm-terraform/master.tf](../../kubernetes-kvm-terraform/master.tf)** — two occurrences inside cloud-init heredocs:
 - Line 122 (apiserver `TracingConfiguration` written to `/etc/kubernetes/tracing-config.yaml`): `192.168.0.30:4317` → `192.168.0.29:4317`
 - Line 169 (`KubeletConfiguration.tracing.endpoint` in the kubeadm config): `192.168.0.30:4317` → `192.168.0.29:4317`
 - **Do NOT touch line 8** (`zot_registry_ip = "192.168.0.30"`) — that's the Zot container-registry mirror, not tracing.
 ### 2. Runbook docs
 **[docs/kubernetes-tracing.md](../kubernetes-tracing.md)** — replace tracing-related references throughout (≈10 lines):
 - All occurrences of `192.168.0.30:4317` → `192.168.0.29:4317` (lines 54, 83, 99)
 - `http://192.168.0.30:16686` → `http://192.168.0.29:16686` (line 150)
 - SSH targets `novakj@192.168.0.30` → `novakj@192.168.0.29` (lines 25, 31, 43)
 - Remote directory `~/docker-30/tracing` → `~/docker-29/tracing` (lines 25, 31, 43)
 - Prose "Jaeger on docker-30" → "Jaeger on docker-29" (lines 3, 40)
 ### 3. Plan files (historical record — also requested)
 **[plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md](../../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md)** — ~12 references:
 - `192.168.0.30:4317` (lines 38, 76, 101, 111) and `http://192.168.0.30:16686` (line 141) → `.29`
 - `docker-30/tracing/` directory path (lines 13, 16, 117, 128, 155) → `docker-29/tracing/`
 - Prose mentions of "docker-30" in tracing context (lines 5, 11) → "docker-29"
 **[docs/plans/2026-05-21-2035-tracing-toggle-runbook.md](2026-05-21-2035-tracing-toggle-runbook.md)** — ~6 references:
 - `192.168.0.30:4317` (lines 51, 73) and `http://192.168.0.30:16686` (line 110) → `.29`
 - SSH target `novakj@192.168.0.30` (line 35) → `novakj@192.168.0.29`
 - Prose mentions of "docker-30" in tracing context (lines 5, 42) → "docker-29"
 ### Search safety
 Use a per-file targeted replace with surrounding context (e.g. `endpoint: 192.168.0.30:4317`, `novakj@192.168.0.30`, `docker-30/tracing`), not a blanket repo-wide `sed`. The leave-alone list is large (Zot, Vault, Gitea, Kanidm, nginx, gitignore, claude settings, generated tfstate, commented-out blackbox/prometheus examples) — see Phase 1 exploration for the full inventory.
 ## Verification
 ### In-repo
 ```bash
 # Confirm no tracing-related .30 references remain
 grep -nE '192\.168\.0\.30:(4317|16686)' -r .
 grep -nE 'docker-30/tracing' -r .
 # Confirm new IP is in place
 grep -nE '192\.168\.0\.29:4317' kubernetes-kvm-terraform/
 ```
 Both `grep` calls in step 1 should return empty. The leave-alone Zot/Vault/Gitea/nginx hits at port 3000/8200/9443/etc. should still be present (sanity check that the surgical replace didn't over-match).
 Re-run `tofu plan` in [kubernetes-kvm-terraform/](../../kubernetes-kvm-terraform/) to confirm the diff only shows the two endpoint changes and no unrelated drift.
 ### Live cluster follow-up (out of scope for this plan's execution, documented for the operator)
 The in-repo changes only affect newly-bootstrapped nodes. The existing master + workers still have the old endpoint baked into their live files. After merging the repo changes, run:
 1. **kube-apiserver** (master, 192.168.0.31):
   ```bash
   ssh novakj@192.168.0.31 'sudo sed -i s/192.168.0.30:4317/192.168.0.29:4317/ /etc/kubernetes/tracing-config.yaml'
   # kubelet auto-restarts the static pod within ~10s
   ```
 2. **kubelet** on each node (192.168.0.31, .32, .33), serially:
   ```bash
   ssh novakj@$NODE 'sudo sed -i s/192.168.0.30:4317/192.168.0.29:4317/ /var/lib/kubelet/config.yaml && sudo systemctl restart kubelet'
   kubectl wait node --for=condition=Ready --timeout=60s <node-name>
   ```
 3. **Cluster ConfigMap** so future joins/reboots pick it up:
   ```bash
   kubectl -n kube-system edit cm kubelet-config
   # update the tracing.endpoint value under kubelet:
   ```
 4. **End-to-end smoke test** (from [docs/kubernetes-tracing.md](../kubernetes-tracing.md) "Verification" section):
   ```bash
   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   ```
   Then open `http://192.168.0.29:16686` and confirm `apiserver` and `kubelet` services show recent spans for `trace-probe`.
--- a/docs/plans/2026-07-08-0034-deploy-mimir.md
+++ b/docs/plans/2026-07-08-0034-deploy-mimir.md
@@ -0,0 +1,165 @@
 # Deploy Mimir on docker-30 (mirror the Loki setup)
 ## Context
 Loki is already deployed on the docker host (`192.168.0.30`) under
 `vms-home/docker-30/loki/`, storing chunks in Garage S3 and queried through a
 shared Grafana. The user wants Mimir deployed for **metrics** in the same style:
 self-contained compose dir, Garage S3 backend, `.env`-supplied credentials,
 env-expanded config.
 Decisions made:
 - **Mimir only for now.** No producer is wired up in this plan; Mimir simply
  exposes its remote_write endpoint (`http://192.168.0.30:9009/api/v1/push`) for
  a Prometheus/Alloy to point at later.
 - **Reuse the existing Grafana** (the one in the Loki compose) by adding a
  Prometheus-type datasource — no second Grafana.
 Mimir runs in **monolithic mode** (`-target=all`, single binary), which is the
 right shape for a single-node home lab and matches Loki's single-process model.
 ## Reference: how Loki does it (the pattern to copy)
 `vms-home/docker-30/loki/loki-config.yaml` + `docker-compose.yaml`:
 - Garage S3 at `192.168.0.30:3900`, `region: garage`, `s3forcepathstyle: true`,
  `insecure: true`, bucket `loki-chunks`.
 - Creds via `${GARAGE_ACCESS_KEY}` / `${GARAGE_SECRET_KEY}` from `.env`,
  expanded with `-config.expand-env=true`.
 - `obs` bridge network; Grafana published on `3001`.
 ## Files to create (new dir `vms-home/docker-30/mimir/`)
 ### 1. `mimir-config.yaml` — monolithic + Garage S3
 ```yaml
 multitenancy_enabled: false            # single-tenant; Loki mirrors this (auth_enabled:false)
 server:
  http_listen_port: 9009
  grpc_listen_port: 9095
  log_level: info
 common:
  storage:
    backend: s3
    s3:
      endpoint: 192.168.0.30:3900      # no scheme; insecure toggles http
      region: garage                   # MUST byte-match s3_region in garage.toml
      access_key_id: ${GARAGE_ACCESS_KEY}
      secret_access_key: ${GARAGE_SECRET_KEY}
      insecure: true                   # http on LAN
      bucket_lookup_type: path         # path-style — Garage requirement (== s3forcepathstyle)
 blocks_storage:
  s3: { bucket_name: mimir-blocks }
  tsdb:        { dir: /data/tsdb }
  bucket_store:{ sync_dir: /data/tsdb-sync }
 ruler_storage:
  s3: { bucket_name: mimir-ruler }
 alertmanager_storage:
  s3: { bucket_name: mimir-alertmanager }
 compactor:
  data_dir: /data/compactor
 ruler:
  rule_path: /data/ruler
 alertmanager:
  data_dir: /data/alertmanager
 limits:
  compactor_blocks_retention_period: 744h   # 31d, matches Loki's retention
  ingestion_rate: 50000                      # samples/s per tenant; bump on 429s
  ingestion_burst_size: 100000
 ```
 Note: the three storage components inherit endpoint/creds from `common.storage`;
 only `bucket_name` is overridden per component. Buckets can be collapsed to one
 if preferred, but three is cleaner and Garage bucket creation is cheap.
 ### 2. `docker-compose.yaml` — copy Loki's, swap image/ports/paths
 ```yaml
 services:
  mimir:
    image: grafana/mimir:2.16.1          # pin latest stable — verify tag before apply
    container_name: mimir
    restart: unless-stopped
    command:
      - -config.file=/etc/mimir/config.yaml
      - -config.expand-env=true          # REQUIRED — expands ${GARAGE_*}
      - -target=all                      # monolithic single-binary mode
    ports:
      - "9009:9009"                      # push (/api/v1/push) + query (/prometheus)
    volumes:
      - ./mimir-config.yaml:/etc/mimir/config.yaml:ro
      - ./mimir-data:/data
    env_file: [.env]
    networks: [obs]
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:9009/ready || exit 1"]
      interval: 15s
      timeout: 5s
      retries: 10
      start_period: 30s
    logging:
      driver: json-file
      options: { max-size: "10m", max-file: "3" }
 networks:
  obs: { driver: bridge }
 ```
 ### 3. `.env` — reuse the same Garage credentials as Loki
 ```
 GARAGE_ACCESS_KEY=<same key id Loki uses>
 GARAGE_SECRET_KEY=<same secret>
 ```
 ## Manual steps on docker-30 (execution)
 1. **Create Garage buckets** and grant the existing Loki key access:
   ```bash
   ssh novakj@192.168.0.30
   for b in mimir-blocks mimir-ruler mimir-alertmanager; do
     docker exec garage /garage bucket create "$b"
   done
   docker exec garage /garage key list           # get the key name Loki already uses
   for b in mimir-blocks mimir-ruler mimir-alertmanager; do
     docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
   done
   ```
 2. Copy the `mimir/` dir to the host (or `git pull` if this repo is checked out
   there), fill `.env`, then `cd mimir && docker compose up -d`.
 3. **Add Mimir datasource to the existing Grafana** (`:3001`):
   - Type: **Prometheus**
   - URL: `http://192.168.0.30:9009/prometheus`
   - No `X-Scope-OrgID` header needed (multitenancy disabled).
 ## Verification
 - `curl http://192.168.0.30:9009/ready` → `ready` (also gates the healthcheck).
 - `docker compose ps` shows `mimir` healthy; `docker compose logs -f mimir`
  clean (no S3 auth / bucket errors — confirms Garage wiring).
 - Smoke-test the write path without a producer:
  ```bash
  curl http://192.168.0.30:9009/api/v1/push   # expect 4xx (proto body), NOT conn-refused
  ```
 - In Grafana → Explore → Mimir datasource, run `up` or
  `count({__name__!=""})`. Empty until a producer remote_writes — expected.
 - Confirm blocks land in Garage after ~2h (first block flush):
  `docker exec garage /garage bucket info mimir-blocks`.
 ## Later (out of scope here): wiring a producer
 Point any Prometheus/Alloy at:
 ```yaml
 remote_write:
  - url: http://192.168.0.30:9009/api/v1/push
 ```
 The existing `vms/utility-101-shadow/docker/monitoring/prometheus.yml` is the
 natural first candidate.
--- a/docs/plans/2026-07-08-2346-alloy-metrics-to-mimir.md
+++ b/docs/plans/2026-07-08-2346-alloy-metrics-to-mimir.md
@@ -0,0 +1,116 @@
 # Alloy → Mimir: scrape Kubernetes metrics + Prometheus Operator monitors
 > On execution, also copy this file into the repo per CLAUDE.md convention:
 > `docs/plans/2026-07-08-HHMM-alloy-metrics-to-mimir.md` (get `HHMM` via `date "+%H%M"`).
 ## Context
 Mimir is now up on docker-30 (`http://192.168.0.30:9009`, single-tenant, plain
 HTTP over LAN) but nothing in the cluster pushes metrics to it. Grafana Alloy
 already exists in the repo but is **logs-only** (ships pod/journal logs + k8s
 events to Loki) and — importantly — the entire `gitops/home-kubernetes/alloy/`
 directory is **not wired into Flux** yet.
 kube-prometheus-stack is deployed and active: it provides the Prometheus
 Operator CRDs (ServiceMonitor / PodMonitor), kube-state-metrics, node-exporter,
 and ServiceMonitors for all standard targets (kubelet, cAdvisor, apiserver,
 coredns, etc.). Its bundled Prometheus scrapes everything into a local 60d TSDB
 but does **not** remote_write anywhere.
 **Goal:** add a dedicated `alloy-metrics` instance that scrapes standard
 Kubernetes metrics and reads ServiceMonitors/PodMonitors from the Prometheus
 Operator, remote-writing to Mimir — additive alongside the existing Prometheus
 (no changes to kube-prometheus-stack). Because kube-prometheus-stack already
 ships ServiceMonitors for every standard target, a single
 `prometheus.operator.servicemonitors` + `prometheus.operator.podmonitors` pair
 satisfies **both** requirements at once. As part of wiring the `alloy/` dir into
 Flux, the existing `alloy-logs` and `alloy-events` releases also come under
 GitOps management (confirmed desired).
 **Decisions confirmed:** additive/coexist with bundled Prometheus; activate all
 three Alloy instances (logs, events, metrics).
 ## Design
 - Metrics scraping runs as a **single Deployment** (`controller.type: deployment`,
  1 replica) — never a DaemonSet, or every cluster-wide target would be scraped
  once per node. This is why it must be a separate release from `alloy-logs`
  (which is a DaemonSet).
 - Mirror the conventions already in `helmrelease_alloy-logs.yaml`: chart `alloy`
  `>=1.8.0 <2.0.0`, image `tag: v1.17.1`, `rbac.create: false`,
  `alloy.enableReporting: false`, `driftDetection: enabled`, install/upgrade
  `retries: 3`, `interval: 30m`, namespace `monitoring`.
 - River config = three components:
  1. `prometheus.operator.servicemonitors "sm"` — discovers/scrapes **all**
     ServiceMonitors cluster-wide (covers kubelet, cAdvisor, kube-state-metrics,
     node-exporter, apiserver, coredns, controller-manager, scheduler — i.e. the
     "standard kubernetes metrics").
  2. `prometheus.operator.podmonitors "pm"` — discovers/scrapes all PodMonitors.
  3. `prometheus.remote_write "mimir"` → `http://192.168.0.30:9009/api/v1/push`,
     `external_labels = { cluster = "homelab" }` (matches the logs config).
  Both operator components `forward_to = [prometheus.remote_write.mimir.receiver]`.
 - Reuse `cluster = "homelab"` external label for consistency with the Loki side.
 ## Files
 ### New — `gitops/home-kubernetes/alloy/helmrelease_alloy-metrics.yaml`
 HelmRelease `alloy-metrics` (ns `monitoring`), Deployment, serviceAccount
 `alloy-metrics`, inline `alloy.configMap.content` with the three components
 above. Modest resources (e.g. requests cpu 100m / mem 256Mi, limit mem 768Mi —
 operator discovery + scrape buffers use more than the log shipper).
 ### New — `gitops/home-kubernetes/alloy/rbac_metrics-collector.yaml`
 The `prometheus.operator.*` components need read access beyond the existing
 `alloy-log-reader` role. ClusterRole `alloy-metrics-reader` + ClusterRoleBinding
 to SA `alloy-metrics` (monitoring), granting:
 - core: `namespaces, nodes, nodes/metrics, nodes/proxy, services, endpoints, pods` — get/list/watch
 - `discovery.k8s.io`: `endpointslices` — get/list/watch
 - `networking.k8s.io`: `ingresses` — get/list/watch
 - `monitoring.coreos.com`: `servicemonitors, podmonitors, probes, scrapeconfigs` — get/list/watch
 - nonResourceURLs `/metrics`, `/metrics/cadvisor` — get (kubelet scraping)
 (Kept as a separate file so the metrics RBAC is self-contained; the existing
 `rbac_log-collector.yaml` stays untouched.)
 ### New — `gitops/home-kubernetes/alloy/kustomization.yaml`
 `kind: Kustomization` listing all manifests in the dir: `helmrepository_grafana.yaml`,
 `helmrelease_alloy-logs.yaml`, `helmrelease_allow-events.yaml`,
 `helmrelease_alloy-metrics.yaml`, `rbac_log-collector.yaml`,
 `rbac_metrics-collector.yaml`. Makes the Flux build explicit/deterministic.
 ### Modified — `gitops/home-kubernetes/flux-system/extra-kustomizations.yaml`
 Append an `alloy` Flux Kustomization:
 ```yaml
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
  name: alloy
  namespace: flux-system
 spec:
  interval: 10m0s
  path: ./gitops/home-kubernetes/alloy
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  dependsOn:
    - name: kube-prometheus   # needs the monitoring ns + operator CRDs/ServiceMonitors
 ```
 ## Verification
 1. Commit + push; force reconcile: `flux reconcile kustomization flux-system --with-source`.
 2. `kubectl get kustomizations -A` — `alloy` Ready=True.
 3. `kubectl get helmreleases -n monitoring` — `alloy-metrics` (+ logs/events) Ready.
 4. `kubectl -n monitoring get pods -l app.kubernetes.io/instance=alloy-metrics`
   Running; `kubectl -n monitoring logs deploy/alloy-metrics` shows no RBAC
   `forbidden` errors and targets discovered.
 5. Alloy UI: `kubectl -n monitoring port-forward deploy/alloy-metrics 12345:12345`
   → open `http://localhost:12345`; confirm the two operator components list
   healthy targets and `prometheus.remote_write.mimir` shows successful sends.
 6. Confirm Mimir is receiving: query it directly, e.g.
   `curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=up' | jq '.data.result | length'`
   returns > 0 (or check the Grafana Mimir datasource for `up`,
   `kube_pod_info`, `node_cpu_seconds_total`, `container_cpu_usage_seconds_total`).
 7. Sanity: a `cluster="homelab"` label is present on the ingested series.
--- a/gitops/home-kubernetes/alloy/helmrelease_allow-events.yaml
+++ b/gitops/home-kubernetes/alloy/helmrelease_allow-events.yaml
@@ -0,0 +1,42 @@
 apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease
 metadata:
  name: alloy-events
  namespace: monitoring
 spec:
  interval: 30m
  chart:
    spec:
      chart: alloy
      version: ">=1.8.0 <2.0.0"
      sourceRef: { kind: HelmRepository, name: grafana, namespace: monitoring }
      interval: 12h
  install: { remediation: { retries: 3 } }
  upgrade: { remediation: { retries: 3 } }
  driftDetection: { mode: enabled }
  values:
    controller:
      type: deployment
      replicas: 1                          # cluster-scoped → exactly one
    image:
      tag: v1.17.1
    rbac:           { create: false }
    serviceAccount: { create: true, name: alloy-events }
    alloy:
      enableReporting: false
      resources:
        requests: { cpu: 50m, memory: 96Mi }
        limits:   { memory: 256Mi }
      configMap:
        content: |-
          loki.source.kubernetes_events "events" {
            forward_to = [loki.process.events.receiver]
          }
          loki.process "events" {
            stage.static_labels { values = { source = "kubernetes-events" } }
            forward_to = [loki.write.loki.receiver]
          }
          loki.write "loki" {
            endpoint { url = "http://192.168.0.30:3100/loki/api/v1/push" }
            external_labels = { cluster = "homelab" }
          }
--- a/gitops/home-kubernetes/alloy/helmrelease_alloy-logs.yaml
+++ b/gitops/home-kubernetes/alloy/helmrelease_alloy-logs.yaml
@@ -0,0 +1,106 @@
 apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease
 metadata:
  name: alloy-logs
  namespace: monitoring
 spec:
  interval: 30m
  chart:
    spec:
      chart: alloy
      version: ">=1.8.0 <2.0.0"          # newest 1.x each reconcile; pin exact if you want determinism
      sourceRef: { kind: HelmRepository, name: grafana, namespace: monitoring }
      interval: 12h                        # poll cadence for new chart versions
  install: { remediation: { retries: 3 } }
  upgrade: { remediation: { retries: 3 } }
  driftDetection: { mode: enabled }        # revert manual kubectl edits
  values:
    controller:
      type: daemonset
    image:
      tag: v1.17.1
    rbac:           { create: false }
    serviceAccount: { create: true, name: alloy-logs }
    alloy:
      enableReporting: false
      securityContext: { runAsUser: 0, runAsGroup: 0 }   # journald read
      mounts:
        varlog: true
      resources:
        requests: { cpu: 100m, memory: 128Mi }
        limits:   { memory: 512Mi }
      extraEnv:
        - name: NODE_NAME
          valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
      configMap:
        content: |-
          discovery.kubernetes "pods" {
            role = "pod"
            selectors {
              role  = "pod"
              field = "spec.nodeName=" + sys.env("NODE_NAME")
            }
          }
          discovery.relabel "pods" {
            targets = discovery.kubernetes.pods.targets
            rule {
              source_labels = ["__meta_kubernetes_namespace"]
              target_label  = "namespace"
            }
            rule {
              source_labels = ["__meta_kubernetes_pod_label_app"]
              target_label  = "app"
            }
            rule {
              source_labels = ["__meta_kubernetes_pod_container_name"]
              target_label  = "container"
            }
            rule {
              source_labels = ["__meta_kubernetes_pod_node_name"]
              target_label  = "node"
            }
            rule {
              source_labels = ["__meta_kubernetes_pod_name"]
              target_label  = "pod"
            }
          }
          loki.source.kubernetes "pods" {
            targets    = discovery.relabel.pods.output
            forward_to = [loki.process.pods.receiver]
          }
          loki.process "pods" {
            stage.static_labels {
              values = { source = "kubernetes" }
            }
            forward_to = [loki.write.loki.receiver]
          }
          discovery.relabel "journal" {
            targets = []
            rule {
              source_labels = ["__journal__systemd_unit"]
              target_label  = "unit"
            }
            rule {
              source_labels = ["__journal__hostname"]
              target_label  = "node"
            }
            rule {
              source_labels = ["__journal_priority_keyword"]
              target_label  = "level"
            }
          }
          loki.source.journal "node" {
            path          = "/var/log/journal"
            max_age       = "12h"
            labels        = { source = "node-journal" }
            relabel_rules = discovery.relabel.journal.rules
            forward_to    = [loki.write.loki.receiver]
          }
          loki.write "loki" {
            endpoint {
              url = "http://192.168.0.30:3100/loki/api/v1/push"
            }
            external_labels = { cluster = "homelab" }
          }
--- a/gitops/home-kubernetes/alloy/helmrelease_alloy-metrics.yaml
+++ b/gitops/home-kubernetes/alloy/helmrelease_alloy-metrics.yaml
@@ -0,0 +1,49 @@
 apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease
 metadata:
  name: alloy-metrics
  namespace: monitoring
 spec:
  interval: 30m
  chart:
    spec:
      chart: alloy
      version: ">=1.8.0 <2.0.0"
      sourceRef: { kind: HelmRepository, name: grafana, namespace: monitoring }
      interval: 12h
  install: { remediation: { retries: 3 } }
  upgrade: { remediation: { retries: 3 } }
  driftDetection: { mode: enabled }
  values:
    controller:
      type: deployment
      replicas: 1                          # cluster-scoped scrape target discovery → exactly one
    image:
      tag: v1.17.1
    rbac:           { create: false }
    serviceAccount: { create: true, name: alloy-metrics }
    alloy:
      enableReporting: false
      resources:
        requests: { cpu: 100m, memory: 256Mi }
        limits:   { memory: 768Mi }
      configMap:
        content: |-
          // Discovers every ServiceMonitor cluster-wide. kube-prometheus-stack already ships
          // ServiceMonitors for kubelet/cAdvisor, kube-state-metrics, node-exporter, apiserver,
          // coredns, controller-manager and scheduler — so this alone covers "standard k8s metrics".
          prometheus.operator.servicemonitors "sm" {
            forward_to = [prometheus.remote_write.mimir.receiver]
          }
          // Discovers every PodMonitor cluster-wide, same selector scope as above.
          prometheus.operator.podmonitors "pm" {
            forward_to = [prometheus.remote_write.mimir.receiver]
          }
          prometheus.remote_write "mimir" {
            endpoint {
              url = "http://192.168.0.30:9009/api/v1/push"
            }
            external_labels = { cluster = "homelab" }
          }
--- a/gitops/home-kubernetes/alloy/helmrepository_grafana.yaml
+++ b/gitops/home-kubernetes/alloy/helmrepository_grafana.yaml
@@ -0,0 +1,8 @@
 apiVersion: source.toolkit.fluxcd.io/v1
 kind: HelmRepository
 metadata:
  name: grafana
  namespace: monitoring
 spec:
  interval: 1h
  url: https://grafana.github.io/helm-charts
--- a/gitops/home-kubernetes/alloy/rbac_log-collector.yaml
+++ b/gitops/home-kubernetes/alloy/rbac_log-collector.yaml
@@ -0,0 +1,15 @@
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata: { name: alloy-log-reader }
 rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "namespaces", "nodes", "nodes/proxy", "events"]
    verbs: ["get", "list", "watch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata: { name: alloy-log-reader }
 roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: alloy-log-reader }
 subjects:
  - { kind: ServiceAccount, name: alloy-logs,   namespace: monitoring }
  - { kind: ServiceAccount, name: alloy-events, namespace: monitoring }
--- a/gitops/home-kubernetes/alloy/rbac_metrics-collector.yaml
+++ b/gitops/home-kubernetes/alloy/rbac_metrics-collector.yaml
@@ -0,0 +1,25 @@
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata: { name: alloy-metrics-reader }
 rules:
  - apiGroups: [""]
    resources: ["namespaces", "nodes", "nodes/metrics", "nodes/proxy", "services", "endpoints", "pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["monitoring.coreos.com"]
    resources: ["servicemonitors", "podmonitors", "probes", "scrapeconfigs"]
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
    verbs: ["get"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata: { name: alloy-metrics-reader }
 roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: alloy-metrics-reader }
 subjects:
  - { kind: ServiceAccount, name: alloy-metrics, namespace: monitoring }
--- a/gitops/home-kubernetes/alloy/readme.md
+++ b/gitops/home-kubernetes/alloy/readme.md
@@ -0,0 +1,10 @@
 ## debugging alloy
 ```bash
 POD=$(kubectl -n monitoring get pod -l app.kubernetes.io/name=alloy -o name | head -1)
 kubectl -n monitoring debug -it $POD --image=nicolaka/netshoot --target=alloy -- bash
 curl -v http://192.168.0.30:3100/ready
 ```
--- a/gitops/home-kubernetes/cert-manager/helmrelease.yaml
+++ b/gitops/home-kubernetes/cert-manager/helmrelease.yaml
@@ -8,7 +8,7 @@ spec:
  chart:
    spec:
      chart: cert-manager
-      version: "v1.17.2"
+      version: "v1.20.2"
      sourceRef:
        kind: HelmRepository
        name: cert-manager
--- a/gitops/home-kubernetes/flux-system/extra-kustomizations.yaml
+++ b/gitops/home-kubernetes/flux-system/extra-kustomizations.yaml
@@ -81,6 +81,21 @@ spec:
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
  name: alloy
  namespace: flux-system
 spec:
  interval: 10m0s
  path: ./gitops/home-kubernetes/alloy
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  dependsOn:
    - name: kube-prometheus   # needs the monitoring ns + operator CRDs/ServiceMonitors
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
  name: ingress-nginx
  namespace: flux-system
@@ -143,6 +158,19 @@ spec:
  sourceRef:
    kind: GitRepository
    name: flux-system
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
  name: fuj
  namespace: flux-system
 spec:
  interval: 10m0s
  path: ./gitops/home-kubernetes/fuj
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
 # ---
 # apiVersion: kustomize.toolkit.fluxcd.io/v1
 # kind: Kustomization
--- a/kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml
+++ b/kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml
@@ -44,6 +44,7 @@ spec:
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --tracing-config-file=/etc/kubernetes/tracing-config.yaml
    image: registry.k8s.io/kube-apiserver:v1.32.11
    imagePullPolicy: IfNotPresent
    livenessProbe:
--- a/kubernetes-kvm-terraform/files/tracing-config.yaml
+++ b/kubernetes-kvm-terraform/files/tracing-config.yaml
@@ -0,0 +1,4 @@
 apiVersion: apiserver.config.k8s.io/v1beta1
 kind: TracingConfiguration
 endpoint: 192.168.0.29:4317
 samplingRatePerMillion: 1000000
--- a/kubernetes-kvm-terraform/master.tf
+++ b/kubernetes-kvm-terraform/master.tf
@@ -115,6 +115,13 @@ locals {
            skip_verify = true
            override_path = true              
      - path: /etc/kubernetes/tracing-config.yaml
        content: |
          apiVersion: apiserver.config.k8s.io/v1beta1
          kind: TracingConfiguration
          endpoint: 192.168.0.29:4317
          samplingRatePerMillion: 1000000
      - path: /etc/kubernetes/auth-config.yaml
        content: |
          apiVersion: apiserver.config.k8s.io/v1beta1
@@ -158,6 +165,9 @@ locals {
          apiVersion: kubelet.config.k8s.io/v1beta1
          kind: KubeletConfiguration
          cgroupDriver: systemd
          tracing:
            endpoint: 192.168.0.29:4317
            samplingRatePerMillion: 1000000
      - path: /etc/profile.d/kubectl.sh
        content: |
--- a/k8s-pod-creation-tracing.md
+++ b/k8s-pod-creation-tracing.md
@@ -0,0 +1,157 @@
 # Pod-creation latency tracing with OpenTelemetry
 ## Context
 Goal: measure end-to-end latency of `kubectl apply` → pod running, broken down by component. The approach is to enable native OpenTelemetry tracing in Kubernetes 1.32 components (kube-apiserver, kubelet, etcd) and ship spans to a Jaeger backend running outside the cluster on `docker-29`. With all three reporting under the same trace id (object-resource correlation), Jaeger will let us see how much time is spent in admission/etcd, on the wire, and inside the kubelet's pod sync loop.
 Constraints / known gaps:
 - **kube-scheduler has no native OTLP tracing yet**; there will be a gap between "object stored" and "kubelet sees assignment". We can infer it from event timestamps (`kubectl get events`) or `.status.conditions` PodScheduled/Initialized/ContainersReady/Ready timestamps.
 - Cluster is K8s 1.32, kubeadm-bootstrapped via Terraform/cloud-init. Per Phase-3 decision, the canonical source-of-truth is the Terraform cloud-init; changes there cover future rebuilds. To activate tracing on the *currently running* nodes we also need a one-time manual application step (documented at the bottom).
 ## Backend: Jaeger on docker-29
 New directory `docker-29/tracing/` containing `docker-compose.yaml`. Uses Jaeger all-in-one v1 with Badger on-disk storage and the built-in OTLP receiver enabled.
 ```yaml
 # docker-29/tracing/docker-compose.yaml
 services:
  jaeger:
    image: jaegertracing/all-in-one:1.62
    container_name: jaeger
    restart: unless-stopped
    environment:
      SPAN_STORAGE_TYPE: badger
      BADGER_EPHEMERAL: "false"
      BADGER_DIRECTORY_VALUE: /badger/data
      BADGER_DIRECTORY_KEY: /badger/key
      BADGER_SPAN_STORE_TTL: 4h       # auto-expire traces older than 4h
      BADGER_MAINTENANCE_INTERVAL: 5m # GC cycle that reclaims disk
      COLLECTOR_OTLP_ENABLED: "true"
    volumes:
      - ./badger:/badger
    ports:
      - "4317:4317"    # OTLP gRPC (apiserver + kubelet exporters)
      - "4318:4318"    # OTLP HTTP (optional)
      - "16686:16686"  # Jaeger UI
 ```
 Endpoint that the cluster will send to: `192.168.0.29:4317`. Jaeger UI: `http://192.168.0.29:16686`.
 The OTLP receiver in Jaeger all-in-one accepts **plaintext** OTLP/gRPC by default — matches `endpoint: ...:4317` on the K8s side with no TLS. Acceptable for a home lab on a private LAN.
 ### Storage sizing and retention
 Two complementary levers control how much disk Badger uses:
 1. **TTL** — `BADGER_SPAN_STORE_TTL=4h` deletes spans older than 4 hours; Badger's value-log GC reclaims disk on `BADGER_MAINTENANCE_INTERVAL` (default 5m). This is the hard upper bound on retention.
 2. **Sampling rate** — controls *how many* spans get written in the first place. Set in the apiserver/kubelet TracingConfiguration (`samplingRatePerMillion`).
 Rough estimate for this cluster (1 master + 2 workers, mostly idle, Flux reconciling continuously):
 | Sampling | Span ingest rate | Disk footprint at 4h TTL |
 |---|---|---|
 | 100% (`1000000`) | ~500–3000 spans/s | ~6–40 GB steady-state |
 | 1% (`10000`) | ~5–30 spans/s | ~60–400 MB steady-state |
 | 0.1% (`1000`) | ~0.5–3 spans/s | ~6–40 MB steady-state |
 Note: 100% sampling on this cluster is sized for short experiments, not long-term operation. The 4h TTL prevents runaway growth if I forget to lower it, but Badger has no built-in *size* cap — only TTL + GC. If a hard size cap is needed, mount `./badger/` from a fixed-size LVM volume or zfs dataset with a quota.
 **Recommended progression**:
 - Start with `samplingRatePerMillion: 1000000` (100%) and `TTL=4h` so the first measurements are easy to find.
 - After the first round of experiments, drop sampling to `10000` (1%) for steady-state and adjust `TTL` upward if a longer history is useful.
 ## kube-apiserver tracing
 Two changes, both in the Terraform cloud-init / referenced manifest:
 ### 1. Write the TracingConfiguration file (master.tf cloud-init)
 Add to `write_files` in [kubernetes-kvm-terraform/master.tf](kubernetes-kvm-terraform/master.tf):
 ```yaml
 - path: /etc/kubernetes/tracing-config.yaml
  content: |
    apiVersion: apiserver.config.k8s.io/v1beta1
    kind: TracingConfiguration
    endpoint: 192.168.0.29:4317
    samplingRatePerMillion: 1000000  # 100% sampling for measurement work
 ```
 The apiserver static pod already mounts `/etc/kubernetes` read-only via the `k8s-config` volume, so the file becomes visible inside the container automatically — no new volume mount needed.
 ### 2. Add the apiserver flag
 Edit [kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml](kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml) — add one new flag in the `command:` list (alphabetical placement after `--tls-private-key-file` works; ordering isn't enforced):
 ```yaml
 - --tracing-config-file=/etc/kubernetes/tracing-config.yaml
 ```
 `APIServerTracing` is GA in 1.32 — no feature-gate flag required.
 ## kubelet tracing
 Add a `tracing:` block to the existing `KubeletConfiguration` document inside the kubeadm-config heredoc in [kubernetes-kvm-terraform/master.tf](kubernetes-kvm-terraform/master.tf) (around line 158):
 ```yaml
 apiVersion: kubelet.config.k8s.io/v1beta1
 kind: KubeletConfiguration
 cgroupDriver: systemd
 tracing:
  endpoint: 192.168.0.29:4317
  samplingRatePerMillion: 1000000
 ```
 `KubeletTracing` is beta and on by default in 1.32 — no feature gate needed.
 Kubeadm stores this `KubeletConfiguration` in the `kubelet-config` ConfigMap in `kube-system`, so workers joining via `kubeadm join` pick the tracing block up automatically. The two existing workers (kube-node-32, kube-node-33) need a one-time manual nudge (see below) because they were already joined before the change.
 ## etcd tracing (optional, second-step)
 etcd supports OTLP via flags `--experimental-enable-distributed-tracing`, `--experimental-distributed-tracing-address=192.168.0.29:4317`, and `--experimental-distributed-tracing-sampling-rate=1000000`. Enabling these requires editing the etcd static-pod manifest on the master — propose deferring until after the apiserver+kubelet path is verified, so we keep the first iteration narrow.
 ## Files to change
 | File | Change |
 |---|---|
 | `docker-29/tracing/docker-compose.yaml` | **new** — Jaeger all-in-one + Badger |
 | `kubernetes-kvm-terraform/master.tf` | add `tracing-config.yaml` to `write_files`; add `tracing:` block to `KubeletConfiguration` |
 | `kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml` | add `--tracing-config-file=/etc/kubernetes/tracing-config.yaml` |
 | `kubernetes-kvm-terraform/files/tracing-config.yaml` | **new** — reference copy of the TracingConfiguration (mirrors the inline content in master.tf, matches the existing `auth-config.yaml` convention) |
 No changes to `nodes-on-homer.tf` / `nodes-on-beelink.tf` are needed — workers receive kubelet config from the cluster ConfigMap on join.
 ## Applying to the currently running cluster
 The Terraform changes cover future rebuilds. To activate tracing **now** without rebuilding:
 1. **Start Jaeger** on docker-29: `cd docker-29/tracing && docker compose up -d` (creates `./badger/` for storage).
 2. **On the master (kube-master-31, 192.168.0.31)**:
   - `scp` or write `/etc/kubernetes/tracing-config.yaml` with the TracingConfiguration shown above.
   - Edit `/etc/kubernetes/manifests/kube-apiserver.yaml` to add `--tracing-config-file=/etc/kubernetes/tracing-config.yaml`. Kubelet will detect the manifest change and restart the apiserver static pod within seconds.
   - Watch: `crictl ps | grep apiserver`, then `kubectl get --raw /livez`.
 3. **Update the kubelet-config ConfigMap** in `kube-system`:
   - `kubectl -n kube-system edit cm kubelet-config` and add the same `tracing:` block under `kubelet:`.
 4. **On every node** (master + both workers): edit `/var/lib/kubelet/config.yaml` to append the `tracing:` block, then `systemctl restart kubelet`. Pods will *not* be evicted by a kubelet restart, but expect a brief NotReady blip per node — stagger the restarts.
 ## Verification
 End-to-end test once everything is live:
 1. UI reachable: open `http://192.168.0.29:16686` and check "Services" dropdown populates with `apiserver` (it will appear only after the first traced request).
 2. Generate traffic that exercises the full path:
   ```
   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
   kubectl delete pod trace-probe
   ```
 3. In Jaeger UI, search service `apiserver`, operation `KubernetesAPI` (or similar) — you should see a trace covering admission/etcd write. Search service `kubelet`, operation `syncPod` — kubelet-side pod sync spans should be visible.
 4. Cross-check the gap (scheduler) by comparing pod `metadata.creationTimestamp`, the `PodScheduled` condition `lastTransitionTime`, and the kubelet span start. The diff between those values quantifies the scheduler portion that we cannot trace natively.
 5. Confirm Badger persistence: `docker compose restart jaeger`, reload UI — earlier traces should still be present.
 ## Notes / risks
 - 100% sampling rate is fine for measurement; once we're done we should drop to e.g. `samplingRatePerMillion: 10000` (1%) so storage doesn't explode.
 - Badger volume `./badger/` on docker-29 will grow with span volume; size budget worth watching but not urgent.
 - `tracing.endpoint` is plaintext OTLP/gRPC and traverses the LAN — fine for a home lab but explicitly *not* encrypted.
 - If apiserver fails to come back after the manifest edit, the most common cause is a typo in the new flag; revert is straightforward — keep a copy of the original `/etc/kubernetes/manifests/kube-apiserver.yaml` in `/root/` before editing.
--- a/servers/barber/readme.md
+++ b/servers/barber/readme.md
@@ -0,0 +1,136 @@
 # Barber
 ## network
 ```bash
 # ip: 87.236.197.158, 87.236.197.157
 ```
 ## TODO
 - firewall
 - kvm
 ## KVM
 ```bash
 egrep -c '(vmx|svm)' /proc/cpuinfo      # > 0 expected (Intel VT-x / AMD-V)
 # qemu-kvm is a transitional metapackage on 26.04; qemu-system-x86 is the real binary.
 apt update
 apt install -y \
  qemu-system-x86 qemu-utils \
  libvirt-daemon-system libvirt-clients \
  virtinst cpu-checker ovmf
 apt install virt-manager
 # optional: libguestfs-tools (image surgery), virt-manager (GUI over X-fwd)
 # Verify acceleration + service
 kvm-ok                  # expect: "KVM acceleration can be used"
 lsmod | grep kvm        # kvm + kvm_intel|kvm_amd loaded
 sudo systemctl enable --now libvirtd
 sudo systemctl is-active libvirtd
 # non-root management
 sudo usermod -aG libvirt,kvm $USER
 newgrp libvirt          # apply to current shell; otherwise log out/in
 virsh list --all        # should run without sudo
 # default libvirt network configuration changed to: 192.168.124.*
 ```
 ## Firewall
 ```bash
 ufw default deny incoming
 ufw default allow outgoing
 ufw limit 22/tcp        # SSH — limit, not allow (see below)
 ufw enable              # answer 'y' — safe, the rule's already in
 ufw status verbose
 # Pin IP forwarding
 echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-ip-forward.conf
 sudo sysctl --system
 # Fix the FORWARD policy (mandatory for NAT guests)
 sudo sed -i 's/^DEFAULT_FORWARD_POLICY=.*/DEFAULT_FORWARD_POLICY="ACCEPT"/' /etc/default/ufw
 sudo ufw reload
 # DHCP/DNS to guests (usually automatic)
 sudo ufw allow in on virbr0 to any port 53 proto udp
 sudo ufw allow in on virbr0 to any port 67 proto udp
 # dns on utility-101
 ufw route allow proto udp from any to 192.168.124.101 port 53
 ufw route allow proto tcp from any to 192.168.124.101 port 53
 ```
 ```bash
 # ufw fights with libvirtd about some iptables rules
 # /etc/libvirt/hooks/network
 cat > /etc/libvirt/hooks/network <<'EOF'
 #!/bin/bash
 # Re-inject inbound DNS ACCEPT into libvirt's LIBVIRT_FWI chain.
 # libvirt flushes/rebuilds this chain on network start, so the rule
 # must be re-applied each time the network comes up.
 NET="$1"   # network name (arg 1)
 OP="$2"    # operation   (arg 2)
 TARGET_NET="default"      # <-- set to your `virsh net-list` name
 VM_IP="192.168.124.101"
 WAN_IF="enp5s0"
 BR_IF="virbr0"
 OSPROVIDER_NET="osprovider"
 OSPROVIDER_BR="virbr2"
 OSPROVIDER_SUBNET="10.10.50.0/24"
 [ "$OP" = "started" ] || exit 0
 if [ "$NET" = "$TARGET_NET" ]; then
    for proto in udp tcp; do
        iptables -C LIBVIRT_FWI -i "$WAN_IF" -o "$BR_IF" -d "$VM_IP" -p "$proto" --dport 53 -j ACCEPT 2>/dev/null \
          || iptables -I LIBVIRT_FWI 1 -i "$WAN_IF" -o "$BR_IF" -d "$VM_IP" -p "$proto" --dport 53 -j ACCEPT
    done
 fi
 if [ "$NET" = "$OSPROVIDER_NET" ]; then
    # libvirt's LIBVIRT_FWO chain REJECTs all forwarding from virbr2 because osprovider is
    # defined as an isolated network. Insert these rules at the top of FORWARD (before the
    # LIBVIRT_* chains run) so provider-network traffic can reach POSTROUTING/MASQUERADE.
    iptables -C FORWARD -i "$OSPROVIDER_BR" -s "$OSPROVIDER_SUBNET" -j ACCEPT 2>/dev/null \
      || iptables -I FORWARD 1 -i "$OSPROVIDER_BR" -s "$OSPROVIDER_SUBNET" -j ACCEPT
    iptables -C FORWARD -o "$OSPROVIDER_BR" -d "$OSPROVIDER_SUBNET" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2>/dev/null \
      || iptables -I FORWARD 1 -o "$OSPROVIDER_BR" -d "$OSPROVIDER_SUBNET" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 fi
 EOF
 chmod +x /etc/libvirt/hooks/network 
 ```
 ## VNC remote desktop
 ```bash
 apt install tigervnc-standalone-server xfce4 xfce4-goodies
 apt install dbus-x11
 # Fix the xstartup to launch Xfce explicitly inside a fresh dbus session:
 cat > ~/.vnc/xstartup <<'EOF'
 #!/bin/sh
 unset SESSION_MANAGER
 unset DBUS_SESSION_BUS_ADDRESS
 export XDG_SESSION_TYPE=x11
 export XDG_CURRENT_DESKTOP=XFCE
 [ -r "$HOME/.Xresources" ] && xrdb "$HOME/.Xresources"
 exec dbus-run-session -- startxfce4
 EOF
 chmod +x ~/.vnc/xstartup
 ```
--- a/servers/barber/vms/utility-101-barber.md
+++ b/servers/barber/vms/utility-101-barber.md
--- a/servers/psmf/dns-configuration.md
+++ b/servers/psmf/dns-configuration.md
@@ -0,0 +1,157 @@
 ## disable systemd-resolved
 ```bash
 sudo mkdir -p /etc/systemd/resolved.conf.d
 sudo tee /etc/systemd/resolved.conf.d/disable-stub.conf <<EOF
 [Resolve]
 DNSStubListener=no
 EOF
 # Fix /etc/resolv.conf symlink so the box itself still resolves
 sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
 sudo systemctl restart systemd-resolved
 ```
 ## dnsmasq in ts2 namespace
 ```bash
 sudo mkdir -p /etc/dnsmasq-ts2
 sudo tee /etc/dnsmasq-ts2/dnsmasq.conf <<'EOF'
 # Inside ts2 netns — tailscale0 is the only real interface.
 # Bind to all (namespace is isolated, so this is safe).
 no-resolv
 no-hosts
 # Upstreams (explicit, no /etc/resolv.conf dependency inside netns)
 server=1.1.1.1
 server=9.9.9.9
 # Zone
 local=/intranet/
 domain=intranet
 address=/psmf.intranet/100.90.25.77
 address=/psmf-new.intranet/100.90.25.77
 cache-size=1000
 log-facility=/var/log/dnsmasq-ts2.log
 user=dnsmasq
 pid-file=/run/dnsmasq-ts2.pid
 EOF
 ```
 ```bash
 # systemd-unit
 sudo tee /etc/systemd/system/dnsmasq-ts2.service <<'EOF'
 [Unit]
 Description=dnsmasq inside ts2 netns (intranet zone)
 After=network-online.target
 Wants=network-online.target
 # If you have a unit that sets up the ts2 namespace + tailscaled inside it,
 # add it here, e.g.:
 # Requires=tailscaled-ts2.service
 # After=tailscaled-ts2.service
 [Service]
 Type=simple
 NetworkNamespacePath=/var/run/netns/ts2
 ExecStartPre=/usr/sbin/dnsmasq --test -C /etc/dnsmasq-ts2/dnsmasq.conf
 ExecStart=/usr/sbin/dnsmasq -k -C /etc/dnsmasq-ts2/dnsmasq.conf
 Restart=on-failure
 RestartSec=5
 [Install]
 WantedBy=multi-user.target
 EOF
 ```
 ```bash
 # validate and start
 sudo systemctl daemon-reload
 sudo systemctl enable --now dnsmasq-ts2
 sudo systemctl status dnsmasq-ts2 --no-pager
 sudo ip netns exec ts2 ss -lnup | grep :53
 sudo ip netns exec ts2 dig @127.0.0.1 psmf.intranet +short
 sudo ip netns exec ts2 dig @127.0.0.1 cloudflare.com +short
 ```
 ## dnsmasq in root namespace
 ```bash
 sudo mkdir -p /etc/dnsmasq-root
 sudo tee /etc/dnsmasq-root/dnsmasq.conf <<'EOF'
 # Root netns instance. Stock dnsmasq.service must be disabled/masked
 # so it doesn't fight us for port 53.
 # systemd-resolved stub listener is disabled (see top of this file).
 # Only listen on these — don't grab :53 on every interface.
 bind-interfaces
 interface=br0
 interface=tailscale0
 # Upstreams (explicit, no /etc/resolv.conf lookup)
 no-resolv
 no-hosts
 server=1.1.1.1
 server=9.9.9.9
 # Zone
 local=/intranet/
 domain=intranet
 address=/psmf.intranet/100.90.25.77
 address=/psmf-new.intranet/100.90.25.77
 cache-size=1000
 log-facility=/var/log/dnsmasq-root.log
 user=dnsmasq
 pid-file=/run/dnsmasq-root.pid
 EOF
 ```
 ```bash
 # systemd-unit
 sudo tee /etc/systemd/system/dnsmasq-root.service <<'EOF'
 [Unit]
 Description=dnsmasq in root netns (intranet zone)
 After=network-online.target
 Wants=network-online.target
 # Make sure the stock apt dnsmasq is out of the way:
 # sudo systemctl disable --now dnsmasq
 # sudo systemctl mask dnsmasq
 [Service]
 Type=simple
 ExecStartPre=/usr/sbin/dnsmasq --test -C /etc/dnsmasq-root/dnsmasq.conf
 ExecStart=/usr/sbin/dnsmasq -k -C /etc/dnsmasq-root/dnsmasq.conf
 Restart=on-failure
 RestartSec=5
 [Install]
 WantedBy=multi-user.target
 EOF
 ```
 ```bash
 # validate and start
 sudo systemctl daemon-reload
 sudo systemctl enable --now dnsmasq-root
 sudo systemctl status dnsmasq-root --no-pager
 # verify it's bound only on br0 + tailscale0 (not on lo, not 0.0.0.0)
 sudo ss -lnup | grep :53
 # smoke test against the host's own br0 / tailscale0 IP
 BR0_IP=$(ip -4 -o addr show br0        | awk '{print $4}' | cut -d/ -f1)
 TS0_IP=$(ip -4 -o addr show tailscale0  | awk '{print $4}' | cut -d/ -f1)
 dig @"$BR0_IP" psmf.intranet  +short
 dig @"$TS0_IP" psmf.intranet  +short
 dig @"$BR0_IP" cloudflare.com +short
 ```
--- a/servers/psmf/files/nginx/default
+++ b/servers/psmf/files/nginx/default
@@ -0,0 +1,28 @@
 # Reverse proxy: horst-2 default site -> horst-1 Apache (10.0.0.5)
 # Deployed during the horst-1 -> horst-2 migration.
 # See servers/psmf/migration-plan.md.
 server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    # Long bodies / uploads from the legacy PHP intranet app.
    client_max_body_size 64m;
    location / {
        proxy_pass http://10.0.0.5;
        proxy_http_version 1.1;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host  $host;
        proxy_connect_timeout 10s;
        proxy_read_timeout    300s;
        proxy_send_timeout    300s;
    }
 }
--- a/servers/psmf/files/nginx/psmf-new.intranet
+++ b/servers/psmf/files/nginx/psmf-new.intranet
@@ -0,0 +1,27 @@
 # Reverse proxy: psmf-new.intranet -> dockerized PHP intranet app on horst-2 (127.0.0.1:8080)
 # See servers/psmf/migration-plan.md and servers/psmf/dns-configuration.md.
 server {
    listen 80;
    listen [::]:80;
    server_name psmf-new.intranet;
    # Long bodies / uploads from the PHP intranet app.
    client_max_body_size 64m;
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host  $host;
        proxy_connect_timeout 10s;
        proxy_read_timeout    300s;
        proxy_send_timeout    300s;
    }
 }
--- a/servers/psmf/files/nginx/psmf.intranet
+++ b/servers/psmf/files/nginx/psmf.intranet
@@ -0,0 +1,28 @@
 # Reverse proxy: psmf.intranet -> horst-1 Apache (10.0.0.5)
 # Same target as the default site, but bound to the psmf.intranet host header.
 # See servers/psmf/migration-plan.md and servers/psmf/dns-configuration.md.
 server {
    listen 80;
    listen [::]:80;
    server_name psmf.intranet;
    # Long bodies / uploads from the legacy PHP intranet app.
    client_max_body_size 64m;
    location / {
        proxy_pass http://10.0.0.5;
        proxy_http_version 1.1;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host  $host;
        proxy_connect_timeout 10s;
        proxy_read_timeout    300s;
        proxy_send_timeout    300s;
    }
 }
--- a/servers/psmf/files/srv/psmf/docker-compose.yml
+++ b/servers/psmf/files/srv/psmf/docker-compose.yml
@@ -0,0 +1,15 @@
 services:
  app:
    image: gitea.home.hrajfrisbee.cz/psmf/psmf-intranet:1.03
    ports:
      - "8080:8080"
    environment:
      DB_HOST: ${DB_HOST:-host.docker.internal}
      DB_NAME: ${DB_NAME:-psmf}
      DB_USER: ${DB_USER:-psmf}
      DB_PASS: ${DB_PASS:-psmf}
      # PSMF_BASE_URL: http://127.0.0.1:8080  # override for PDF export when behind a proxy
      PSMF_HEADER_TEXT: "Vítejte na intranetu PSMF (PHP {PHP_VERSION})"
      PSMF_HEADER_BG: "#a0b000"
      PSMF_SHOW_PHP_VERSION: "1"
    restart: unless-stopped
--- a/servers/psmf/files/tailscaled-ts2.service
+++ b/servers/psmf/files/tailscaled-ts2.service
@@ -0,0 +1,21 @@
 # location: /etc/systemd/system/tailscaled-ts2.service
 [Unit]
 Description=tailscaled in ts2 netns
 After=ts2-netns.service network-online.target
 Requires=ts2-netns.service
 BindsTo=ts2-netns.service
 [Service]
 ExecStart=/usr/sbin/ip netns exec ts2 /usr/sbin/tailscaled \
  --state=/var/lib/tailscale-ts2/tailscaled.state \
  --socket=/run/tailscale-ts2/tailscaled.sock \
  --port=41642 \
  --tun=tailscale0
 Restart=on-failure
 RestartSec=5s
 RuntimeDirectory=tailscale-ts2
 StateDirectory=tailscale-ts2
 [Install]
 WantedBy=multi-user.target
--- a/servers/psmf/files/ts2-netns-down
+++ b/servers/psmf/files/ts2-netns-down
@@ -0,0 +1,11 @@
 #!/usr/bin/env bash
 # best-effort cleanup, don't fail on missing pieces
 # location: /usr/local/sbin/ts2-netns-down
 iptables -t nat -D POSTROUTING -s 10.200.0.0/30 -o br0 -j MASQUERADE 2>/dev/null || true
 iptables -D FORWARD -i veth-ts2 -o br0 -j ACCEPT 2>/dev/null || true
 iptables -D FORWARD -o veth-ts2 -i br0 -m state --state RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || true
 ip link del veth-ts2 2>/dev/null || true
 ip netns del ts2 2>/dev/null || true
 rm -rf /etc/netns/ts2
--- a/servers/psmf/files/ts2-netns-up
+++ b/servers/psmf/files/ts2-netns-up
@@ -0,0 +1,55 @@
 #!/usr/bin/env bash
 # location: /usr/local/sbin/ts2-netns-up
 set -e
 # bail out clean if anything already exists from a previous run
 ip netns list | grep -qw ts2 || ip netns add ts2
 ip link show veth-ts2 &>/dev/null || \
  ip link add veth-ts2 type veth peer name veth-ts2-ns
 # move peer into ns only if it's still on the host
 ip link show veth-ts2-ns &>/dev/null && \
  ip link set veth-ts2-ns netns ts2
 ip addr add 10.200.0.1/30 dev veth-ts2 2>/dev/null || true
 ip link set veth-ts2 up
 ip -n ts2 addr add 10.200.0.2/30 dev veth-ts2-ns 2>/dev/null || true
 ip -n ts2 link set veth-ts2-ns up
 ip -n ts2 link set lo up
 ip -n ts2 route add default via 10.200.0.1 2>/dev/null || true
 mkdir -p /etc/netns/ts2
 echo "nameserver 1.1.1.1" > /etc/netns/ts2/resolv.conf
 sysctl -wq net.ipv4.ip_forward=1
 # -C checks if rule exists, -A appends only if missing
 iptables -t nat -C POSTROUTING -s 10.200.0.0/30 -o br0 -j MASQUERADE 2>/dev/null || \
  iptables -t nat -A POSTROUTING -s 10.200.0.0/30 -o br0 -j MASQUERADE
 iptables -C FORWARD -i veth-ts2 -o br0 -j ACCEPT 2>/dev/null || \
  iptables -A FORWARD -i veth-ts2 -o br0 -j ACCEPT
 iptables -C FORWARD -o veth-ts2 -i br0 -m state --state RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || \
  iptables -A FORWARD -o veth-ts2 -i br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
 # iptables rules + sysctl inside ts2 ns
 # Forwarding inside ts2
 ip netns exec ts2 sysctl -wq net.ipv4.ip_forward=1
 # DNAT incoming ssh to host
 ip netns exec ts2 iptables -t nat -C PREROUTING -i tailscale0 -p tcp --dport 22 -j DNAT --to-destination 10.200.0.1:22 2>/dev/null || \
  ip netns exec ts2 iptables -t nat -A PREROUTING -i tailscale0 -p tcp --dport 22 -j DNAT --to-destination 10.200.0.1:22
 # DNAT incoming http to host nginx
 ip netns exec ts2 iptables -t nat -C PREROUTING -i tailscale0 -p tcp --dport 80 -j DNAT --to-destination 10.200.0.1:80 2>/dev/null || \
  ip netns exec ts2 iptables -t nat -A PREROUTING -i tailscale0 -p tcp --dport 80 -j DNAT --to-destination 10.200.0.1:80
 # DNAT incoming 8080 to docker web app in root ns
 ip netns exec ts2 iptables -t nat -C PREROUTING -i tailscale0 -p tcp --dport 8080 -j DNAT --to-destination 10.200.0.1:8080 2>/dev/null || \
  ip netns exec ts2 iptables -t nat -A PREROUTING -i tailscale0 -p tcp --dport 8080 -j DNAT --to-destination 10.200.0.1:8080
 ip netns exec ts2 iptables -t nat -C POSTROUTING -o veth-ts2-ns -j MASQUERADE 2>/dev/null || \
  ip netns exec ts2 iptables -t nat -A POSTROUTING -o veth-ts2-ns -j MASQUERADE
--- a/servers/psmf/files/ts2-netns.service
+++ b/servers/psmf/files/ts2-netns.service
@@ -0,0 +1,17 @@
 # /etc/systemd/system/ts2-netns.service
 [Unit]
 Description=ts2 netns + veth + NAT
 After=network-online.target
 Wants=network-online.target
 # tie tailscaled lifecycle to this
 Before=tailscaled-ts2.service
 [Service]
 Type=oneshot
 RemainAfterExit=yes
 ExecStart=/usr/local/sbin/ts2-netns-up
 ExecStop=/usr/local/sbin/ts2-netns-down
 [Install]
 WantedBy=multi-user.target
--- a/servers/psmf/files/ts2-resolv.conf
+++ b/servers/psmf/files/ts2-resolv.conf
@@ -0,0 +1,2 @@
 # location: /etc/netns/ts2/resolv.conf
 nameserver 1.1.1.1
--- a/servers/psmf/horst-2/crontab
+++ b/servers/psmf/horst-2/crontab
@@ -0,0 +1,6 @@
 # backups
 15 11,15,19,22 * * * root /srv/bin/psmf-cli db backup --env-file ~/.psmf/horst-2.env --db psmf --dir /srv/backups/horst-2 -v >> /var/log/psmf-backup.log 2>&1
 17 11,15,19,22 * * * root /srv/bin/psmf-cli db backup --env-file ~/.psmf/horst.env --db psmf --dir /srv/backups/horst -v >> /var/log/psmf-backup.log 2>&1
 # prune backups
 20 11,15,19,22 * * * root /srv/bin/psmf-cli db prune-backups --db psmf --dir /srv/backups/horst-2 --keep-within 2w --keep-weekly 8 --keep-monthly 120 >> /var/log/psmf-backup.log 2>&1
 21 11,15,19,22 * * * root /srv/bin/psmf-cli db prune-backups --db psmf --dir /srv/backups/horst --keep-within 2w --keep-weekly 8 --keep-monthly 120 >> /var/log/psmf-backup.log 2>&1
--- a/servers/psmf/migration-plan.md
+++ b/servers/psmf/migration-plan.md
@@ -0,0 +1,93 @@
 # Migration from horst-1 to horst-2
 ## postgres database
 ```bash
 # [DONE] install postgres
 apt update
 apt install -y curl ca-certificates gnupg lsb-release
 # PGDG signing key (keyring-style, not the deprecated apt-key)
 sudo install -d /usr/share/postgresql-common/pgdg
 sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
  -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc
 # repo
 echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] \
 https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \
 | sudo tee /etc/apt/sources.list.d/pgdg.list
 sudo apt update
 sudo apt install -y postgresql-18 postgresql-contrib-18
 # backup database from horst-1
 # users backup
 pg_dumpall --globals-only -h 10.0.0.5 -U postgres > /tmp/pg-dumpall-globals
 # data backup
 pg_dump -h 10.0.0.5 -U postgres -C psmf > /tmp/psmf.pgsql
 # restore backup of database from horst-1
 # create users 
 cat /tmp/pg-dumpall-globals | psql
 # restore psmf database
 cat /tmp/psmf.pgsql | psql
 # install docker oin horst-2
 # Docker's GPG key (keyring, not apt-key)
 sudo install -m 0755 -d /etc/apt/keyrings
 sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /etc/apt/keyrings/docker.asc
 sudo chmod a+r /etc/apt/keyrings/docker.asc
 # repo
 echo \
 "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
 https://download.docker.com/linux/ubuntu \
 $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
 sudo apt-get update
 sudo apt-get install -y \
  docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin
 # [DONE] dockerize php intranet app
 # dockerized + upgraded to PHP 8.3.31
 # [DONE] start psmf app on horst-2 and point it to database on horst
 # [TODO] point php app on horst-1 to horst-2
 # validate application is still working
 # [OK] - wkhtml - PDF printing of html pages
 # prepare local backups
 # validate if pg sync to api still works from horst-1
 # set sync to api on horst-2
 # install nginx and configure default site as reverse proxy to horst-1
 sudo apt-get update
 sudo apt-get install -y nginx
 # drop the proxy config in place (file lives in this repo at servers/psmf/files/nginx/default)
 sudo install -m 0644 -o root -g root \
    servers/psmf/files/nginx/default /etc/nginx/sites-available/default
 # sites-enabled/default is already a symlink to ../sites-available/default on a stock Ubuntu install
 sudo nginx -t
 sudo systemctl reload nginx
 sudo systemctl enable nginx
 # intranet vhosts: psmf.intranet -> horst-1, psmf-new.intranet -> local docker app
 # (depends on the dnsmasq setup in servers/psmf/dns-configuration.md)
 sudo install -m 0644 -o root -g root \
    servers/psmf/files/nginx/psmf.intranet     /etc/nginx/sites-available/psmf.intranet
 sudo install -m 0644 -o root -g root \
    servers/psmf/files/nginx/psmf-new.intranet /etc/nginx/sites-available/psmf-new.intranet
 sudo ln -sf ../sites-available/psmf.intranet     /etc/nginx/sites-enabled/psmf.intranet
 sudo ln -sf ../sites-available/psmf-new.intranet /etc/nginx/sites-enabled/psmf-new.intranet
 sudo nginx -t
 sudo systemctl reload nginx
 ```
--- a/servers/psmf/readme.md
+++ b/servers/psmf/readme.md
@@ -0,0 +1,33 @@
 # Servers
 ### Horst
 - ip: 10.0.0.5
 - os: CentOS release 6.9 (Final)
 - services:
    - apache tcp/80
    - postgres tcp/5432
    - samba (not really in use)
    - cronjobs
        - /srv/bin/pg_psmf_dump.sh
        - /srv/bin/psmf-prepare-tables.sh 2023 2027
        - /srv/bin/psmf-api-sync.sh
 - specials:
    -   some binaries/libraries relevant to PDF printing from the app WKHTML        
 ```bash
 # crontab content
 00 11,15,19,22 * * * root /srv/bin/pg_psmf_dump.sh
 55 10,14,18,21 * * * root /srv/bin/psmf-prepare-tables.sh 2023 2027
 05 11,15,19,22 * * * root /srv/bin/psmf-api-sync.sh
 ```
 #### Problems
 - operating system is already that old, that there is not reasonable and safe upgrade path to something current
 ### Horst-2
 - ip: 10.0.0.6
--- a/servers/psmf/tailscale-second-instance.md
+++ b/servers/psmf/tailscale-second-instance.md
@@ -0,0 +1,39 @@
 ```bash
 # Create namespace
 sudo ip netns add ts2
 # veth pair: host side <-> ns side
 sudo ip link add veth-ts2 type veth peer name veth-ts2-ns
 sudo ip link set veth-ts2-ns netns ts2
 # Host side
 sudo ip addr add 10.200.0.1/30 dev veth-ts2
 sudo ip link set veth-ts2 up
 # Namespace side
 sudo ip -n ts2 addr add 10.200.0.2/30 dev veth-ts2-ns
 sudo ip -n ts2 link set veth-ts2-ns up
 sudo ip -n ts2 link set lo up
 sudo ip -n ts2 route add default via 10.200.0.1
 # DNS inside the netns
 sudo mkdir -p /etc/netns/ts2
 echo "nameserver 1.1.1.1" | sudo tee /etc/netns/ts2/resolv.conf
 # NAT from the netns out to the world (replace eth0 with your egress iface)
 sudo sysctl -w net.ipv4.ip_forward=1
 sudo iptables -t nat -A POSTROUTING -s 10.200.0.0/30 -o br0 -j MASQUERADE
 sudo iptables -A FORWARD -i veth-ts2 -o br0 -j ACCEPT
 sudo iptables -A FORWARD -o veth-ts2 -i br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
 # quick sanity check
 sudo ip netns exec ts2 curl https://controlplane.tailscale.com
 # expect: "OK"
 ```
 ```bash
 # ts2 alias
 echo 'alias ts2="sudo ip netns exec ts2 tailscale --socket=/run/tailscale-ts2/tailscaled.sock"' \
  | sudo tee /etc/profile.d/tailscale-ts2.sh
 ```
--- a/servers/saint/etc/ifcfg-eno0
+++ b/servers/saint/etc/ifcfg-eno0
@@ -0,0 +1,22 @@
 TYPE="Ethernet"
 BOOTPROTO="none"
 DEFROUTE="yes"
 IPV4_FAILURE_FATAL="no"
 IPV6INIT="yes"
 IPV6_AUTOCONF="yes"
 IPV6_DEFROUTE="yes"
 IPV6_FAILURE_FATAL="no"
 NAME="eno1"
 UUID="2f881122-85e5-4078-8cec-cf4a7e465321"
 DEVICE="eno1"
 ONBOOT="yes"
 IPADDR="87.236.197.83"
 PREFIX="23"
 GATEWAY="87.236.196.1"
 DNS1="87.236.198.210"
 #DNS1="87.236.198.211"
 DNS2="89.187.142.8"
 IPV6_PEERDNS="yes"
 IPV6_PEERROUTES="yes"
 IPV6_PRIVACY="no"
 ZONE=public
--- a/servers/shadow/iptables/rules.v4
+++ b/servers/shadow/iptables/rules.v4
@@ -49,6 +49,7 @@ COMMIT
 -A FORWARD -i eno1 -p udp -m udp --dport 5353 -j ACCEPT
 -A FORWARD -i eno1 -p udp -m udp --dport 51820 -j ACCEPT
 -A FORWARD -i eno1 -p udp -m udp --dport 1194 -j ACCEPT
 -A FORWARD -i eno1 -p tcp -m tcp --dport 33022 -j ACCEPT
 -A FORWARD -j DOCKER-USER
 -A FORWARD -j DOCKER-ISOLATION-STAGE-1
 -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
@@ -92,12 +93,14 @@ COMMIT
 -A LIBVIRT_INP -p tcp -m tcp --dport 53 -j ACCEPT
 -A LIBVIRT_INP -p udp -m udp --dport 5353 -j ACCEPT
 -A LIBVIRT_INP -p tcp -m tcp --dport 5353 -j ACCEPT
 -A LIBVIRT_INP -p tcp -m tcp --dport 33022 -j ACCEPT
 -A LIBVIRT_INP -p udp -m udp --dport 67 -j ACCEPT
 -A LIBVIRT_INP -p tcp -m tcp --dport 67 -j ACCEPT
 -A LIBVIRT_OUT -p udp -m udp --dport 53 -j ACCEPT
 -A LIBVIRT_OUT -p tcp -m tcp --dport 53 -j ACCEPT
 -A LIBVIRT_OUT -p udp -m udp --dport 5353 -j ACCEPT
 -A LIBVIRT_OUT -p tcp -m tcp --dport 5353 -j ACCEPT
 -A LIBVIRT_OUT -p tcp -m tcp --dport 33022 -j ACCEPT
 -A LIBVIRT_OUT -p udp -m udp --dport 68 -j ACCEPT
 -A LIBVIRT_OUT -p tcp -m tcp --dport 68 -j ACCEPT
 -A f2b-sshd -j RETURN
@@ -118,6 +121,7 @@ COMMIT
 -A PREROUTING -i eno1 -p udp -m udp --dport 51820 -j DNAT --to-destination 192.168.123.101:51820
 -A PREROUTING -i eno1 -p udp -m udp --dport 1194 -j DNAT --to-destination 192.168.123.101:1194
 -A PREROUTING -i eno1 -p tcp -m tcp --dport 21080 -j DNAT --to-destination 192.168.123.141:80
 -A PREROUTING -i eno1 -p tcp -m tcp --dport 33022 -j DNAT --to-destination 192.168.123.21:33022
 -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
 -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
 -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
--- a/servers/shadow/nginx/21_docker_hrajfrisbee.cz.conf
+++ b/servers/shadow/nginx/21_docker_hrajfrisbee.cz.conf
@@ -36,12 +36,6 @@ server {
 # ----------------
 # api-sync.psmf.hrajfrisbee.cz
 server {
  #  if ($host = api-sync.psmf.hrajfrisbee.cz) {
  #      return 301 https://$host$request_uri;
  #  } # managed by Certbot
  listen 80;
  client_max_body_size 15m;                                                                                                                                                              proxy_read_timeout 600;
  server_name
        api-sync.psmf.hrajfrisbee.cz
@@ -49,19 +43,12 @@ server {
  location / {
    proxy_pass http://192.168.123.21:8003;
  }
 }
 server {
    if ($host = api.psmf.hrajfrisbee.cz) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
  server_name
        api.psmf.hrajfrisbee.cz
        ;
    listen 80;
    return 404; # managed by Certbot
    listen 8443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/api-sync.psmf.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/api-sync.psmf.hrajfrisbee.cz/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
 }
@@ -75,6 +62,10 @@ server {
                gitlab.hrajfrisbee.cz
        ;
        # let certbot http-01 land
        location /.well-known/acme-challenge/ { root /var/www/html; }
          location / {
            proxy_pass http://192.168.123.21:80;
            proxy_set_header        X-Real-IP       $remote_addr;
@@ -92,16 +83,17 @@ server {
          }
-    #listen 443 ssl; # managed by Certbot
+    listen 8443 ssl; # managed by Certbot
-    #ssl_certificate /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
+    ssl_certificate /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
-    #ssl_certificate_key /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
+    ssl_certificate_key /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
-    #include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
+    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
-    #ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    } # managed by Certbot
 }
 server {
@@ -123,14 +115,79 @@ server {
          }
-    #listen 443 ssl; # managed by Certbot
+    listen 8443 ssl; # managed by Certbot
-    #ssl_certificate /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
+    ssl_certificate /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
-    #ssl_certificate_key /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
+    ssl_certificate_key /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
-    #include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
+    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
-    #ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    } # managed by Certbot
 }
 server {
    if ($host = api-sync.psmf.hrajfrisbee.cz) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
  listen 80;
  server_name
        api-sync.psmf.hrajfrisbee.cz
        ;
    return 404; # managed by Certbot
 }
 server {
    if ($host = registry.gitlab.hrajfrisbee.cz) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
        server_name
                registry.gitlab.hrajfrisbee.cz
        ;
    listen 80;
    return 404; # managed by Certbot
 }
 server {
    if ($host = gitlab.hrajfrisbee.cz) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
        server_name
                gitlab.hrajfrisbee.cz
        ;
    listen 80;
    return 404; # managed by Certbot
 }
 # ----------------
 # legacy sites
 # ----------------
 server {
        server_name
                vegtral.cz www.vegtral.cz
        ;
        location / {
                proxy_pass       http://192.168.123.21:8080;
                proxy_set_header Host            $host;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header X-Forwarded-Ssl on;
                client_max_body_size 3000M;
                client_body_buffer_size 200000k;
          }
 }
--- a/servers/shadow/nginx/22_docker-dev_hrajfrisbee.cz.conf
+++ b/servers/shadow/nginx/22_docker-dev_hrajfrisbee.cz.conf
@@ -0,0 +1,35 @@
 # ----------------
 # api-sync-dev.psmf.hrajfrisbee.cz
 server {
  client_max_body_size 15m;                                                                                                                                                              proxy_read_timeout 600;
  server_name
        api-sync-dev.psmf.hrajfrisbee.cz
        ;
  location / {
    proxy_pass http://192.168.123.22:8003;
  }
    listen 8443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/api-sync-dev.psmf.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/api-sync-dev.psmf.hrajfrisbee.cz/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
 }
 server {
    if ($host = api-sync-dev.psmf.hrajfrisbee.cz) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
  listen 80;
  server_name
        api-sync-dev.psmf.hrajfrisbee.cz
        ;
    return 404; # managed by Certbot
 }
--- a/servers/shadow/nginx/nginx-sites-enabled-default
+++ b/servers/shadow/nginx/nginx-sites-enabled-default
@@ -389,3 +389,33 @@ server {
    server_name fuj-management.home.hrajfrisbee.cz;
    return 404; # managed by Certbot
 }
 server {
    server_name czechultimate.cz;
    # let certbot http-01 land
    location /.well-known/acme-challenge/ { root /var/www/html; }
    location / { return 301 https://www.czechultimate.cz$request_uri; }
    listen 8443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/czechultimate.cz/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/czechultimate.cz/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
 }
 server {
    if ($host = czechultimate.cz) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
    listen 80;
    server_name czechultimate.cz;
    return 404; # managed by Certbot
 }
--- a/servers/shadow/nginx/nginx.conf
+++ b/servers/shadow/nginx/nginx.conf
@@ -66,6 +66,7 @@ stream {
        # Passthrough to K8s
 		ghost.lab.home.hrajfrisbee.cz   k8s_gatewayapi;
 		fujarna.lab.home.hrajfrisbee.cz   k8s_gatewayapi;
 		fuj-management.lab.home.hrajfrisbee.cz   k8s_gatewayapi;
 		operator-test.lab.home.hrajfrisbee.cz   k8s_gatewayapi;
        ~^.+\.lab\.home\.hrajfrisbee\.cz$    k8s_ingress;
--- a/vms-home/docker-29/readme.md
+++ b/vms-home/docker-29/readme.md
--- a/vms-home/docker-29/tracing/docker-compose.yaml
+++ b/vms-home/docker-29/tracing/docker-compose.yaml
@@ -0,0 +1,14 @@
 services:
  jaeger:
    image: jaegertracing/jaeger:2.18.0
    container_name: jaeger
    restart: unless-stopped
    user: root
    command: ["--config", "/jaeger/config.yaml"]
    volumes:
      - ./jaeger-config.yaml:/jaeger/config.yaml:ro
      - ./badger:/badger
    ports:
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP
      - "16686:16686"  # Jaeger UI
--- a/vms-home/docker-29/tracing/jaeger-config.yaml
+++ b/vms-home/docker-29/tracing/jaeger-config.yaml
@@ -0,0 +1,35 @@
 service:
  extensions: [jaeger_storage, jaeger_query]
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [jaeger_storage_exporter]
 extensions:
  jaeger_storage:
    backends:
      badger_store:
        badger:
          directories:
            keys: /badger/key
            values: /badger/data
          ephemeral: false
          ttl:
            spans: 4h
          metrics_update_interval: 5m
  jaeger_query:
    storage:
      traces: badger_store
 receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 exporters:
  jaeger_storage_exporter:
    trace_storage: badger_store
--- a/vms-home/docker-30/frisbee-drills/run-presejpacky.sh
+++ b/vms-home/docker-30/frisbee-drills/run-presejpacky.sh
@@ -0,0 +1,9 @@
 #!/bin/bash
 docker rm -f drills-presejpacky
 # gitea registry login with kacerr / token
 docker run -d --name drills-presejpacky \
  --restart=always \
  -p 3101:3000 \
  gitea.home.hrajfrisbee.cz/kacerr/flat-stack-presejpacky:presejpacky-v1
--- a/vms-home/docker-30/fuj-management/run.sh
+++ b/vms-home/docker-30/fuj-management/run.sh
--- a/vms-home/docker-30/garage/docker-compose.yml
+++ b/vms-home/docker-30/garage/docker-compose.yml
@@ -0,0 +1,42 @@
 services:
  garage:
    image: dxflrs/garage:v2.3.0
    container_name: garage
    restart: unless-stopped
    networks: [garage]
    ports:
      - "3900:3900"   # S3 API  (LAN-reachable for your other services)
      - "3902:3902"   # S3 web
      # - "3901:3901" # RPC — only publish once you go multi-node
    volumes:
      - /srv/docker/garage/garage.toml:/etc/garage.toml:ro
      - /srv/docker/garage/meta:/var/lib/garage/meta
      - /srv/docker/garage/data:/var/lib/garage/data
      - /srv/docker/garage/snapshots:/var/lib/garage/snapshots
    healthcheck:
      test: ["CMD", "/garage", "status"]   # image is distroless; use the binary, not curl
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s
  garage-webui:
    image: khairul169/garage-webui:1.1.0
    container_name: garage-webui
    restart: unless-stopped
    networks: [garage]
    depends_on:
      garage:
        condition: service_healthy
    ports:
      # - "127.0.0.1:3909:3909"   # bind localhost only — reverse-proxy in front for TLS/auth
      - "3909:3909"
    volumes:
      - /srv/docker/garage/garage.toml:/etc/garage.toml:ro
    environment:
      API_BASE_URL:    "http://garage:3903"   # override self-pointing default
      S3_ENDPOINT_URL: "http://garage:3900"
 networks:
  garage:
    driver: bridge
--- a/vms-home/docker-30/garage/garage.toml
+++ b/vms-home/docker-30/garage/garage.toml
@@ -0,0 +1,28 @@
 metadata_dir           = "/var/lib/garage/meta"
 data_dir               = "/var/lib/garage/data"
 metadata_snapshots_dir = "/var/lib/garage/snapshots"
 db_engine              = "lmdb"
 metadata_fsync                  = true
 metadata_auto_snapshot_interval = "6h"    # LMDB corruption insurance
 compression_level               = 2
 replication_factor              = 1        # single node; bump when you add nodes
 rpc_bind_addr   = "[::]:3901"
 rpc_public_addr = "127.0.0.1:3901"         # self-RPC only on single node
 rpc_secret      = "REPLACE_rpc_secret"     # openssl rand -hex 32
 [s3_api]
 s3_region     = "garage"
 api_bind_addr = "[::]:3900"
 root_domain   = ".s3.garage"
 [s3_web]
 bind_addr   = "[::]:3902"
 root_domain = ".web.garage"
 index       = "index.html"
 [admin]
 api_bind_addr = "[::]:3903"
 admin_token   = "REPLACE_admin_token"      # webui reads this for auth; openssl rand -hex 32
 metrics_token = "REPLACE_metrics_token"    # openssl rand -hex 32
--- a/vms-home/docker-30/garage/readme.md
+++ b/vms-home/docker-30/garage/readme.md
--- a/vms-home/docker-30/gitea/docker-compose.yaml
+++ b/vms-home/docker-30/gitea/docker-compose.yaml
--- a/vms-home/docker-30/gitea/readme.md
+++ b/vms-home/docker-30/gitea/readme.md
--- a/vms-home/docker-30/gitea/runner-config.yaml
+++ b/vms-home/docker-30/gitea/runner-config.yaml
@@ -0,0 +1,15 @@
 runner:
  file: /data/.runner
  fetch_interval: 10s
  fetch_timeout: 30s
  capacity: 4   # number of concurrent jobs this runner will accept
 cache:
  enabled: true
  dir: /data/cache
  host: gitea-runner
  port: 8088
 container:
  network: gitea-production_gitea-network
  force_pull: true
--- a/vms-home/docker-30/kanidm/gitea-action-kubernetes-access.md
+++ b/vms-home/docker-30/kanidm/gitea-action-kubernetes-access.md
--- a/vms-home/docker-30/kanidm/readme.md
+++ b/vms-home/docker-30/kanidm/readme.md
--- a/vms-home/docker-30/kanidm/run.sh
+++ b/vms-home/docker-30/kanidm/run.sh
--- a/vms-home/docker-30/kanidm/server.toml
+++ b/vms-home/docker-30/kanidm/server.toml
--- a/vms-home/docker-30/lab-proxy/nginx.conf
+++ b/vms-home/docker-30/lab-proxy/nginx.conf
--- a/vms-home/docker-30/lab-proxy/run.sh
+++ b/vms-home/docker-30/lab-proxy/run.sh
--- a/vms-home/docker-30/loki/.env
+++ b/vms-home/docker-30/loki/.env
@@ -0,0 +1,2 @@
 GARAGE_ACCESS_KEY=GK...
 GARAGE_SECRET_KEY=...
--- a/vms-home/docker-30/loki/docker-compose.yaml
+++ b/vms-home/docker-30/loki/docker-compose.yaml
@@ -0,0 +1,41 @@
 services:
  loki:
    image: grafana/loki:3.7.3
    container_name: loki
    restart: unless-stopped
    user: "10001:10001"
    command:
      - -config.file=/etc/loki/config.yaml
      - -config.expand-env=true                # REQUIRED — expands ${GARAGE_*}
    ports:
      - "3100:3100"                            # LAN IP, not 0.0.0.0 — push+query
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml:ro
      - ./loki-data:/loki                       # WAL / tsdb-index / compactor scratch
    env_file: [.env]                            # GARAGE_ACCESS_KEY / GARAGE_SECRET_KEY
    networks: [obs]
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:3100/ready || exit 1"]
      interval: 15s
      timeout: 5s
      retries: 10
      start_period: 30s
    logging:                                    # don't let Loki's own logs balloon
      driver: json-file
      options: { max-size: "10m", max-file: "3" }
  grafana:                                      # optional — drop if you query elsewhere
    image: grafana/grafana:13.1.0
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3001:3000" # LAN IP, not               # 3000 is taken by gitea
    volumes:
      - grafana-data:/var/lib/grafana
    networks: [obs]
 volumes:
  grafana-data:
 networks:
  obs: { driver: bridge }
--- a/vms-home/docker-30/loki/loki-config.yaml
+++ b/vms-home/docker-30/loki/loki-config.yaml
@@ -0,0 +1,47 @@
 auth_enabled: false                        # single-tenant; sources split by label
 server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: info
 common:
  path_prefix: /loki
  replication_factor: 1
  ring:
    kvstore: { store: inmemory }
  storage:
    s3:
      endpoint: 192.168.0.30:3900          # no scheme; insecure toggles http
      region: garage                       # MUST byte-match s3_region in garage.toml
      bucketnames: loki-chunks
      access_key_id: ${GARAGE_ACCESS_KEY}
      secret_access_key: ${GARAGE_SECRET_KEY}
      s3forcepathstyle: true               # mandatory for Garage
      insecure: true                       # http on LAN; drop if you TLS-front Garage
 schema_config:
  configs:
    - from: 2026-07-01                     # <= first-ingest date; never mutate historical
      store: tsdb
      object_store: s3
      schema: v13
      index: { prefix: index_, period: 24h }
 storage_config:
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache
 limits_config:
  allow_structured_metadata: true
  volume_enabled: true
  retention_period: 744h                   # 31d; compactor enforces
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_line_size: 256KB
  ingestion_rate_mb: 8                      # per-tenant; bump if you hit 429s
  ingestion_burst_size_mb: 16
 compactor:
  working_directory: /loki/compactor
--- a/vms-home/docker-30/maru-hleda-byt/run.sh
+++ b/vms-home/docker-30/maru-hleda-byt/run.sh
--- a/vms-home/docker-30/mimir/.env
+++ b/vms-home/docker-30/mimir/.env
@@ -0,0 +1,2 @@
 GARAGE_ACCESS_KEY=GK...
 GARAGE_SECRET_KEY=...
--- a/vms-home/docker-30/mimir/docker-compose.yaml
+++ b/vms-home/docker-30/mimir/docker-compose.yaml
@@ -0,0 +1,28 @@
 services:
  mimir:
    image: grafana/mimir:3.1.2
    container_name: mimir
    restart: unless-stopped
    command:
      - -config.file=/etc/mimir/config.yaml
      - -config.expand-env=true                # REQUIRED — expands ${GARAGE_*}
      - -target=all                             # monolithic single-binary mode
    ports:
      - "9009:9009"                            # LAN IP, not 0.0.0.0 — push+query
    volumes:
      - ./mimir-config.yaml:/etc/mimir/config.yaml:ro
      - ./mimir-data:/data                      # WAL / tsdb / compactor scratch
    env_file: [.env]                            # GARAGE_ACCESS_KEY / GARAGE_SECRET_KEY
    networks: [obs]
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:9009/ready || exit 1"]
      interval: 15s
      timeout: 5s
      retries: 10
      start_period: 30s
    logging:                                    # don't let Mimir's own logs balloon
      driver: json-file
      options: { max-size: "10m", max-file: "3" }
 networks:
  obs: { driver: bridge }
--- a/vms-home/docker-30/mimir/mimir-config.yaml
+++ b/vms-home/docker-30/mimir/mimir-config.yaml
@@ -0,0 +1,42 @@
 multitenancy_enabled: false            # single-tenant; mirrors Loki's auth_enabled:false
 server:
  http_listen_port: 9009
  grpc_listen_port: 9095
  log_level: info
 common:
  storage:
    backend: s3
    s3:
      endpoint: 192.168.0.30:3900      # no scheme; insecure toggles http
      region: garage                   # MUST byte-match s3_region in garage.toml
      access_key_id: ${GARAGE_ACCESS_KEY}
      secret_access_key: ${GARAGE_SECRET_KEY}
      insecure: true                   # http on LAN; drop if you TLS-front Garage
      bucket_lookup_type: path         # path-style — Garage requirement (== s3forcepathstyle)
 blocks_storage:
  s3: { bucket_name: mimir-blocks }
  tsdb: { dir: /data/tsdb }
  bucket_store: { sync_dir: /data/tsdb-sync }
 ruler_storage:
  s3: { bucket_name: mimir-ruler }
 alertmanager_storage:
  s3: { bucket_name: mimir-alertmanager }
 compactor:
  data_dir: /data/compactor
 ruler:
  rule_path: /data/ruler
 alertmanager:
  data_dir: /data/alertmanager
 limits:
  compactor_blocks_retention_period: 744h   # 31d; matches Loki's retention_period
  ingestion_rate: 50000                      # samples/s per tenant; bump if you hit 429s
  ingestion_burst_size: 100000
--- a/vms-home/docker-30/mimir/readme.md
+++ b/vms-home/docker-30/mimir/readme.md
@@ -0,0 +1,98 @@
 # Mimir
 Grafana Mimir, monolithic mode (`-target=all`), single-tenant, backed by Garage S3
 (same bucket-per-Garage-instance pattern as `../loki/`). Metrics ingest via
 Prometheus remote_write — Mimir does not scrape anything itself.
 - HTTP (query + push): `9009`
 - gRPC (internal): `9095`
 ## 1. Garage setup (do this first — before `docker compose up`)
 Mimir needs three buckets. Create them and grant the existing Loki Garage key
 read+write access (reuse the key rather than minting a new one, unless you
 want tighter separation):
 ```bash
 ssh novakj@192.168.0.30
 for b in mimir-blocks mimir-ruler mimir-alertmanager; do
  docker exec garage /garage bucket create "$b"
 done
 docker exec garage /garage key list           # find the key name Loki already uses
 for b in mimir-blocks mimir-ruler mimir-alertmanager; do
  docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
 done
 ```
 ## 2. `.env`
 Reuse the same Garage credentials Loki uses (see `../loki/.env`):
 ```
 GARAGE_ACCESS_KEY=GK...
 GARAGE_SECRET_KEY=...
 ```
 ## 3. Start
 ```bash
 cd /path/to/mimir
 docker compose up -d
 ```
 ## 4. Add Mimir as a Grafana datasource
 Reuses the Grafana already running from `../loki/docker-compose.yaml` (`:3001`)
 — no second Grafana instance.
 - Type: **Prometheus**
 - URL: `http://192.168.0.30:9009/prometheus`
 - No `X-Scope-OrgID` header needed (`multitenancy_enabled: false`)
 ## 5. Verify
 ```bash
 curl http://192.168.0.30:9009/ready                # -> ready
 docker compose ps                                   # mimir healthy
 docker compose logs -f mimir                         # no S3 auth/bucket errors
 # push endpoint reachable (expect a 4xx proto-decode error, not connection refused)
 curl http://192.168.0.30:9009/api/v1/push
 ```
 In Grafana → Explore → Mimir datasource, run `up` or `count({__name__!=""})`.
 Empty result is expected until a producer remote_writes into Mimir.
 After ~2h (first block flush interval), confirm blocks landed in Garage:
 ```bash
 docker exec garage /garage bucket info mimir-blocks
 ```
 ## 6. Wiring a producer (later, not done here)
 Point any Prometheus/Alloy remote_write at Mimir:
 ```yaml
 remote_write:
  - url: http://192.168.0.30:9009/api/v1/push
 ```
 `vms/utility-101-shadow/docker/monitoring/prometheus.yml` is the natural first
 candidate to wire up.
 ## Notes / gotchas
 - `bucket_lookup_type: path` in `mimir-config.yaml` is mandatory for Garage
  (equivalent to Loki's `s3forcepathstyle: true`).
 - `-config.expand-env=true` is required on the command line, or `${GARAGE_*}`
  in `mimir-config.yaml` won't expand.
 - First metrics won't appear in Garage until the first block flush (~2h) —
  don't panic if `mimir-blocks` looks empty right after startup.
 - Retention is 31d, set via `limits.compactor_blocks_retention_period` (matches
  Loki's `retention_period`).
 - `region: garage` in the config must byte-match `s3_region` in Garage's
  `garage.toml` — same requirement as Loki.
--- a/vms-home/docker-30/nginx/001-gitea.conf
+++ b/vms-home/docker-30/nginx/001-gitea.conf
--- a/vms-home/docker-30/nginx/002-jellyfin.conf
+++ b/vms-home/docker-30/nginx/002-jellyfin.conf
--- a/vms-home/docker-30/nginx/021-drills.conf
+++ b/vms-home/docker-30/nginx/021-drills.conf
@@ -0,0 +1,20 @@
 server {
    listen 443 ssl http2;
    server_name drills.home.hrajfrisbee.cz;
    ssl_certificate /etc/letsencrypt/live/drills.home.hrajfrisbee.cz/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/drills.home.hrajfrisbee.cz/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;
    location /presejpacky {
        proxy_pass http://192.168.0.30:3101;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
 }
--- a/vms-home/docker-30/readme.md
+++ b/vms-home/docker-30/readme.md
@@ -0,0 +1,43 @@
 # docker-30
 ## taiscale
 ```bash
 # Add signing key
 curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/$(lsb_release -cs).noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
 # Add repo
 echo "deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/tailscale.list
 # Install
 sudo apt update && sudo apt install tailscale
 # Start
 sudo tailscale up
 ```
 ## connect nvmeof volume
 ```bash
 nvme discover -t tcp -a 192.168.0.40 -s 4420
 nvme connect -t tcp -a 192.168.0.40  -s 4420 -n nqn.2011-06.com.truenas:uuid:6ac818fb-9f51-4e28-a5d6-41502eba031d:garage-s3
 nvme list
 mkfs.xfs -L garage /dev/nvme0n1
 mkdir -p /srv/garage
 mount /dev/disk/by-label/garage /srv/garage
 mkdir -p /srv/garage/{meta,data,snapshots}
 # add to fstab
 # LABEL=garage  /srv/garage  xfs  defaults,_netdev,nofail,x-systemd.requires=nvmf-autoconnect.service  0 0
 # /etc/nvme/discovery.conf
 -t tcp -a <TRUENAS_IP> -s 4420 -n nqn.2011-06.com.truenas:garage-s3
 systemctl enable nvmf-autoconnect.service
 ```
--- a/vms-home/docker-30/vault/backup.md
+++ b/vms-home/docker-30/vault/backup.md
--- a/vms-home/docker-30/vault/config.json
+++ b/vms-home/docker-30/vault/config.json
--- a/vms-home/docker-30/vault/docker-compose.yaml
+++ b/vms-home/docker-30/vault/docker-compose.yaml
--- a/vms-home/docker-30/vault/gitea-access-into-vault.md
+++ b/vms-home/docker-30/vault/gitea-access-into-vault.md
--- a/vms-home/docker-30/vault/readme.md
+++ b/vms-home/docker-30/vault/readme.md
--- a/vms-home/docker-30/vault/terraform/main.tf
+++ b/vms-home/docker-30/vault/terraform/main.tf
--- a/vms-home/docker-30/vault/terraform/terraform.tfstate
+++ b/vms-home/docker-30/vault/terraform/terraform.tfstate
--- a/vms-home/docker-30/vault/terraform/terraform.tfstate.backup
+++ b/vms-home/docker-30/vault/terraform/terraform.tfstate.backup
--- a/vms-home/docker-30/vault/terraform/versions.tf
+++ b/vms-home/docker-30/vault/terraform/versions.tf
--- a/vms-home/docker-30/vault/vault-backup.sh
+++ b/vms-home/docker-30/vault/vault-backup.sh
--- a/vms-home/docker-30/zot/config.yaml
+++ b/vms-home/docker-30/zot/config.yaml
--- a/vms-home/docker-30/zot/docker-compose.yaml
+++ b/vms-home/docker-30/zot/docker-compose.yaml
--- a/vms/docker-dev-22/psmf-data-sync/.env
+++ b/vms/docker-dev-22/psmf-data-sync/.env
@@ -0,0 +1,20 @@
 # Image (built by .gitea/workflows/build.yaml)
 # IMAGE_REPO is the Gitea repo path, e.g. "jan/psmf-data-sync"
 IMAGE_REPO=gitea.home.hrajfrisbee.cz/psmf/psmf-data-syncer
 IMAGE_TAG=0.2
 # Host port to publish the syncer on (container listens on 8002 internally)
 HOST_PORT=8003
 # External Postgres the syncer restores into
 DB_HOST=192.168.123.23
 DB_PORT=5432
 DB_NAME=psmf-dev
 DB_USER=psmf-dev
 DB_PASSWORD=** masked even when it is so secret **
 # Which Redis DB index to use (0-15)
 REDIS_DB=0
 # info | debug
 LOG_LEVEL=debug
--- a/vms/docker-dev-22/psmf-data-sync/docker-compose.yml
+++ b/vms/docker-dev-22/psmf-data-sync/docker-compose.yml
@@ -0,0 +1,28 @@
 services:
  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: ["redis-server", "--appendonly", "yes"]
    volumes:
      - redis-data:/data
  psmf-data-sync:
    image: ${IMAGE_REPO}:${IMAGE_TAG:-latest}
    restart: unless-stopped
    depends_on:
      - redis
    ports:
      - "${HOST_PORT:-8003}:8002"
    environment:
      DB_HOST: ${DB_HOST}
      DB_PORT: ${DB_PORT:-5432}
      DB_NAME: ${DB_NAME}
      DB_USER: ${DB_USER}
      DB_PASSWORD: ${DB_PASSWORD}
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_DB: ${REDIS_DB:-0}
      LOG_LEVEL: ${LOG_LEVEL:-info}
 volumes:
  redis-data:
--- a/vms/docker-dev-22/readme.md
+++ b/vms/docker-dev-22/readme.md
@@ -0,0 +1,22 @@
 ## docker
 ```bash
 # prereqs + keyring
 sudo apt-get update
 sudo apt-get install -y ca-certificates curl
 sudo install -m 0755 -d /etc/apt/keyrings
 sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
 sudo chmod a+r /etc/apt/keyrings/docker.asc
 # repo
 echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
 # install
 sudo apt-get update
 sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
 ```
--- a/vms/storage-23/pg_hba.conf
+++ b/vms/storage-23/pg_hba.conf
@@ -0,0 +1,22 @@
 # Database administrative login by Unix domain socket
 local   all             postgres                                peer
 # TYPE  DATABASE        USER            ADDRESS                 METHOD
 # "local" is for Unix domain socket connections only
 local   all             all                                     peer
 # IPv4 local connections:
 host    all             all             127.0.0.1/32            scram-sha-256
 # IPv6 local connections:
 host    all             all             ::1/128                 scram-sha-256
 # Allow replication connections from localhost, by a user with the
 # replication privilege.
 local   replication     all                                     peer
 host    replication     all             127.0.0.1/32            scram-sha-256
 host    replication     all             ::1/128                 scram-sha-256
 # TYPE  DATABASE        USER            ADDRESS                 METHOD
 host    psmf            psmf            192.168.123.21/32       md5
 host    psmfapi         psmf            192.168.123.21/32       md5
 host    postgres        psmf            192.168.123.21/32       md5
 host    psmf-dev        psmf-dev        192.168.123.22/32       md5
--- a/vms/utility-101-shadow/docker/monitoring/prometheus.yml
+++ b/vms/utility-101-shadow/docker/monitoring/prometheus.yml
@@ -45,7 +45,7 @@ scrape_configs:
          - https://gitea.home.hrajfrisbee.cz/
          - https://vault.hrajfrisbee.cz/
          - https://idm.home.hrajfrisbee.cz/
-          - https://maru-hleda-byt.home.hrajfrisbee.cz/mapa_bytu.html
+          # - https://maru-hleda-byt.home.hrajfrisbee.cz/mapa_bytu.html
          # - https://nonexistent.home.hrajfrisbee.cz/
    relabel_configs:
      - source_labels: [__address__]
Author	SHA1	Message	Date
Jan Novak	ed10cc837a	chore: add CLAUDE.md, kubectl notes, and settings/gitignore tweaks Document the repo for Claude Code, note a couple of handy kubectl one-liners, allow a few more Bash/WebFetch/WebSearch permissions used during this session, and ignore the relocated zot sync-credentials.json. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:56 +02:00
Jan Novak	b62b37542b	servers/shadow: open GitLab SSH port, add docker-dev/czechultimate nginx routes Forward tcp/33022 (FORWARD, LIBVIRT_INP/OUT, PREROUTING DNAT to 192.168.123.21) so the GitLab remote is reachable, add nginx vhosts for docker-dev-22 and czechultimate.cz, and route fuj-management through the k8s gateway. Also record saint's eno0 interface config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:49 +02:00
Jan Novak	4b39834f10	servers/barber: add barber server docs Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:42 +02:00
Jan Novak	6efa069b12	servers/psmf: add psmf server docs, nginx/tailscale config, and data-sync Document the psmf host (DNS, second tailscale netns instance, nginx vhosts, migration plan) and add the docker-dev-22 psmf-data-sync service plus the storage-23 pg_hba.conf entry it needs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:39 +02:00
Jan Novak	d541f2e1d2	infra: relocate docker-30 compose stack under vms-home/, add garage and frisbee-drills Move the docker-30 service definitions (gitea, kanidm, vault, zot, nginx, lab-proxy, maru-hleda-byt, fuj-management) into vms-home/ to match the naming convention used for other hosts, and add the new garage object-storage and frisbee-drills services alongside it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:30 +02:00
Jan Novak	d43ffd488e	observability: add k8s API/kubelet tracing, Alloy, Mimir and Loki Wire kube-apiserver and kubelet tracing to a Jaeger collector on docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and stand up Mimir + Loki on docker-30 as their backing stores. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:21 +02:00
Jan Novak	95355ef7a5	gitops: upgrade cert-manager to: v1.20.2	2026-06-17 12:08:07 +02:00
Jan Novak	e341a57018	gitops/fuj: add Flux Kustomization to reconcile fuj manifests Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 20:09:01 +02:00
kacerr	36344f7796	Merge pull request 'gitops: update fuj-management' (#6 ) from gitops/update-fuj-management-20260612-180146 into main Reviewed-on: #6	2026-06-12 20:02:08 +02:00
		`@@ -0,0 +1,2 @@`
							`# location: /etc/netns/ts2/resolv.conf`
							`nameserver 1.1.1.1`
		`@@ -0,0 +1,2 @@`
							`GARAGE_ACCESS_KEY=GK...`
							`GARAGE_SECRET_KEY=...`