chore: add CLAUDE.md, kubectl notes, and settings/gitignore tweaks

Document the repo for Claude Code, note a couple of handy kubectl one-liners, allow a few more Bash/WebFetch/WebSearch permissions used during this session, and ignore the relocated zot sync-credentials.json. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
servers/shadow: open GitLab SSH port, add docker-dev/czechultimate nginx routes
2026-07-09 00:05:56 +02:00 · 2026-07-09 00:05:49 +02:00 · 2026-07-09 00:05:42 +02:00 · 2026-07-09 00:05:39 +02:00 · 2026-07-09 00:05:30 +02:00 · 2026-07-09 00:05:21 +02:00
91 changed files with 2805 additions and 47 deletions
--- a/.claude/settings.json
+++ b/.claude/settings.json
@@ -4,7 +4,11 @@
      "Bash(for f:*)",
      "Bash(do echo:*)",
      "Read(//Users/jan.novak/srv/personal/home-kubernetes/**)",
-      "Bash(done)"
+      "Bash(done)",
+      "Bash(ssh docker-30 *)",
+      "Bash(git add *)",
+      "WebSearch",
+      "WebFetch(domain:cert-manager.io)"
    ]
  }
 }
--- a/.gitignore
+++ b/.gitignore
@@ -11,4 +11,5 @@ tmp/
 vms/utility-101-shadow/docker/monitoring/smtp_password

 docker-30/zot/sync-credentials.json
+vms-home/docker-30/zot/sync-credentials.json
 kubernetes-kvm-terraform/gke_gcloud_auth_plugin_cache
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -0,0 +1,105 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Repository Purpose
+
+Home Kubernetes lab infrastructure-as-code. Manages VM provisioning (Terraform/KVM), Kubernetes cluster configuration, and application deployments via Flux GitOps — all pointing at a self-hosted Gitea instance.
+
+## Common Commands
+
+### Terraform (kubernetes-kvm-terraform/)
+```bash
+cd kubernetes-kvm-terraform
+tofu init
+tofu plan
+tofu apply
+tofu destroy
+```
+
+### Flux GitOps
+```bash
+# Check reconciliation status
+kubectl get kustomizations -A
+kubectl get helmreleases -A
+
+# Force reconcile
+flux reconcile kustomization flux-system --with-source
+flux reconcile helmrelease <name> -n <namespace>
+
+# Watch logs
+flux logs --follow
+
+# Check source sync
+flux get sources git
+```
+
+### kubectl — common ops
+```bash
+export KUBECONFIG=kubernetes-kvm-terraform/kubeconfig
+kubectl get nodes
+kubectl get pods -A
+kubectl get secrets -A
+```
+
+### Docker Compose (docker-30/ services)
+```bash
+# These run on 192.168.0.30 (docker-30), accessed via SSH
+ssh novakj@192.168.0.30
+cd /path/to/service && docker compose up -d
+docker compose logs -f
+```
+
+## Architecture
+
+### Infrastructure Layer
+- **Hypervisors**: homer (192.168.0.7) and beelink (192.168.0.6) run KVM/libvirt
+- **Terraform provider**: primary `qemu+ssh://novakj@192.168.0.7/system`, secondary alias `kvm-beelink` for 192.168.0.6
+- **Kubernetes master**: kube-master-31 @ 192.168.0.31:6443, bootstrapped via kubeadm
+- **OS**: Ubuntu 24.04 Noble cloud images, network bridge `br0`, subnet 192.168.0.0/24
+
+### GitOps Layer (`gitops/home-kubernetes/`)
+Flux syncs from Gitea (`https://gitea.home.hrajfrisbee.cz`, main branch, every 10 minutes). The reconciliation order is enforced via `dependsOn` in `flux-system/extra-kustomizations.yaml`:
+
+```
+00-crds → 00-rbac → cilium → cert-manager → external-secrets → everything else
+```
+
+Each application lives in its own subdirectory under `gitops/home-kubernetes/` and is referenced as a Flux `Kustomization` resource. Prune is enabled — removing a manifest from git removes it from the cluster.
+
+### Secrets Flow
+Vault (docker-30) → External-Secrets controller (in-cluster) → Kubernetes `Secret` objects. Applications reference `ExternalSecret` CRs that pull from Vault paths. Do not put real secrets in git.
+
+### Networking
+- **CNI**: Cilium 1.19.x with Gateway API and Hubble UI enabled
+- **L2 LB**: Cilium `CiliumL2AnnouncementPolicy` + `CiliumLoadBalancerIPPool` for bare-metal load-balancer IPs (defined in `gitops/home-kubernetes/cilium/`)
+- **Ingress**: ingress-nginx for HTTP(S) workloads; Gateway API for newer apps
+- **TLS**: cert-manager issues wildcard cert (`*.home.hrajfrisbee.cz`) referenced by apps
+
+### Storage
+- **democratic-CSI**: iSCSI volumes backed by FreeNAS at 192.168.0.40
+- **Longhorn**: configured but disabled in Flux (directory present, not in `extra-kustomizations.yaml`)
+
+### Supporting Services on docker-30 (192.168.0.30)
+All managed with Docker Compose:
+- **Gitea** — Git server + act_runner (GitHub Actions-compatible CI)
+- **Vault** — secrets backend for External Secrets
+- **Zot** — private OCI/container registry
+- **Kanidm** — identity management / OIDC provider
+- **nginx** — reverse proxy for docker-30 services
+
+## Key Conventions
+
+- **New application**: create a directory under `gitops/home-kubernetes/<app-name>/`, add a `Kustomization` entry in `flux-system/extra-kustomizations.yaml` with appropriate `dependsOn`.
+- **Helm apps**: use a `HelmRepository` + `HelmRelease` pair; pin chart versions explicitly.
+- **Secrets**: add an `ExternalSecret` CR pointing to the Vault path; never commit actual secret values.
+- **Terraform state**: `kubernetes-kvm-terraform/terraform.tfstate*` is gitignored/sensitive; treat it carefully.
+- **kubeconfig**: `kubernetes-kvm-terraform/kubeconfig` is gitignored; obtain it from the master node after provisioning.
+
+
+## Plans
+
+When Claude Code's plan mode is used, save the plan file inside the repo at
+`docs/plans/YYYY-MM-DD-HHMM-<slug>.md` instead of the default `~/.claude/plans/`
+location. Get the timestamp with `date "+%Y-%m-%d-%H%M"` (matches the changelog
+convention). The `<slug>` should be a short kebab-case summary of the plan's topic.
--- a/docker-30/readme.md
+++ b/docker-30/readme.md
@@ -1,17 +0,0 @@
-# docker-30
-
-## taiscale
-
-```bash
-# Add signing key
-curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/$(lsb_release -cs).noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
-
-# Add repo
-echo "deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/tailscale.list
-
-# Install
-sudo apt update && sudo apt install tailscale
-
-# Start
-sudo tailscale up
-```
--- a/docs/kubernetes-extras.md
+++ b/docs/kubernetes-extras.md
@@ -0,0 +1,14 @@
+## kubectl
+
+```bash
+# condensed -o wide
+kubectl get pods -A -o wide --watch | awk '
+/NOMINATED NODE/ { sub(/[[:space:]]+NOMINATED NODE[[:space:]]+READINESS GATES[[:space:]]*$/,""); print; fflush(); next }
+{ sub(/[[:space:]]+[^[:space:]]+[[:space:]]+[^[:space:]]+[[:space:]]*$/,""); print; fflush() }
+'
+
+# condensed -o wide 
+kubectl get pods -A --watch -o custom-columns=\
+'NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase,RESTARTS:.status.containerStatuses[0].restartCount,IP:.status.podIP,NODE:.spec.nodeName'
+
+```
--- a/docs/kubernetes-tracing.md
+++ b/docs/kubernetes-tracing.md
@@ -0,0 +1,180 @@
+# Kubernetes component tracing
+
+OpenTelemetry tracing for kube-apiserver and kubelet, shipping OTLP spans to Jaeger on docker-29. Useful for measuring pod-creation latency broken down by component.
+
+Related: [plans/k8s-pod-creation-tracing.md](../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md) — setup rationale, storage sizing, etcd tracing notes.
+
+---
+
+## Where the toggles live
+
+| Component | Config on node | Restart trigger |
+|---|---|---|
+| kube-apiserver | `/etc/kubernetes/manifests/kube-apiserver.yaml` (flag) + `/etc/kubernetes/tracing-config.yaml` | kubelet detects manifest change, auto-restarts static pod (~5–10 s) |
+| kubelet | `/var/lib/kubelet/config.yaml` (`tracing:` block) + `kube-system/kubelet-config` CM | `systemctl restart kubelet` |
+
+Nodes: kube-master-31 @ 192.168.0.31, kube-node-32 @ 192.168.0.32, kube-node-33 @ 192.168.0.33.
+
+---
+
+## Quick suppress (no component restart)
+
+Stop the Jaeger backend — components keep building spans but have nowhere to send them. Lowest-risk; use this between measurement sessions.
+
+```bash
+ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose stop jaeger'
+```
+
+Resume:
+
+```bash
+ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose start jaeger'
+```
+
+Caveat: components still pay the (small) cost of building spans. For true zero-overhead, use **Full disable** below.
+
+---
+
+## Full enable (live cluster)
+
+### 1. Start Jaeger on docker-30
+
+```bash
+ssh novakj@192.168.0.29 'cd ~/docker-29/tracing && docker compose up -d'
+```
+
+### 2. kube-apiserver (master only — 192.168.0.31)
+
+```bash
+ssh novakj@192.168.0.31 bash -s <<'EOF'
+# TracingConfiguration file
+sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'YAML'
+apiVersion: apiserver.config.k8s.io/v1beta1
+kind: TracingConfiguration
+endpoint: 192.168.0.29:4317
+samplingRatePerMillion: 1000000
+YAML
+
+# Back up the manifest before touching it
+sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing
+
+# Add the flag (after --tls-private-key-file line)
+sudo sed -i '/--tls-private-key-file=/a\    - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' \
+    /etc/kubernetes/manifests/kube-apiserver.yaml
+
+# Wait for static pod to restart
+sleep 10
+sudo crictl ps | grep apiserver
+kubectl get --raw /livez
+EOF
+```
+
+### 3. kubelet (all three nodes)
+
+Run on each node in turn — wait for `Ready` before moving to the next to avoid simultaneous NotReady:
+
+```bash
+for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
+  echo "==> $NODE"
+  ssh ubuntu@$NODE bash -s <<'EOF'
+grep -q '^tracing:' /var/lib/kubelet/config.yaml && echo "already enabled, skipping" && exit 0
+sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'YAML'
+tracing:
+  endpoint: 192.168.0.29:4317
+  samplingRatePerMillion: 1000000
+YAML
+sudo systemctl restart kubelet
+EOF
+  # wait for node Ready before proceeding
+  kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
+done
+```
+
+### 4. Update the cluster ConfigMap (so future joins/reboots pick it up)
+
+```bash
+kubectl -n kube-system edit cm kubelet-config
+# under the `kubelet:` key, add:
+#   tracing:
+#     endpoint: 192.168.0.29:4317
+#     samplingRatePerMillion: 1000000
+```
+
+---
+
+## Full disable (live cluster)
+
+### kube-apiserver (master only)
+
+```bash
+ssh novakj@192.168.0.31 bash -s <<'EOF'
+# Restore the pre-tracing manifest (exact revert)
+sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
+# Optional: remove the config file
+sudo rm -f /etc/kubernetes/tracing-config.yaml
+EOF
+```
+
+Static pod restarts automatically. If the backup wasn't taken, remove the flag in-place instead:
+
+```bash
+sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml
+```
+
+### kubelet (all three nodes)
+
+```bash
+for NODE in 192.168.0.31 192.168.0.32 192.168.0.33; do
+  echo "==> $NODE"
+  ssh novakj@$NODE bash -s <<'EOF'
+sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
+sudo systemctl restart kubelet
+EOF
+  kubectl wait node --for=condition=Ready --timeout=60s $(kubectl get node -o wide | awk "/$NODE/{print \$1}")
+done
+```
+
+Remove the tracing block from the cluster ConfigMap too:
+
+```bash
+kubectl -n kube-system edit cm kubelet-config
+# remove the `tracing:` block under `kubelet:`
+```
+
+---
+
+## Verification
+
+### After enable
+
+1. Open Jaeger UI: `http://192.168.0.29:16686` — Services dropdown should show `apiserver` and `kubelet` (appear after first traced request).
+
+2. Generate a full pod-lifecycle trace:
+
+   ```bash
+   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
+   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
+   kubectl delete pod trace-probe
+   ```
+
+3. In Jaeger UI, search:
+   - Service `apiserver` — look for spans covering the create/watch flow for `trace-probe`.
+   - Service `kubelet` — look for `syncPod` spans tied to the same pod.
+
+4. The gap between apiserver spans and kubelet spans is scheduler time (kube-scheduler has no OTLP tracing in 1.32). Cross-check with `kubectl get events` or pod condition timestamps.
+
+### After disable
+
+Repeat step 2 above, then confirm no new spans appear in Jaeger for `apiserver` or `kubelet` within ~1 minute.
+
+---
+
+## Sampling rate cheatsheet
+
+Change `samplingRatePerMillion` in `/etc/kubernetes/tracing-config.yaml` (apiserver) and `/var/lib/kubelet/config.yaml` (kubelet) then restart each component.
+
+| Value | Effective rate | Use case |
+|---|---|---|
+| `1000000` | 100% | Active measurement |
+| `10000` | 1% | Steady-state background |
+| `0` | off | Disable without removing config |
--- a/docs/kubernetes/alloy.md
+++ b/docs/kubernetes/alloy.md
@@ -0,0 +1,142 @@
+# Alloy: deployment & operations
+
+Three Grafana Alloy instances run in the `monitoring` namespace, all deployed via
+the Grafana `alloy` Helm chart (`>=1.8.0 <2.0.0`, image `v1.17.1`) and managed by
+Flux with `driftDetection: enabled` (manual `kubectl edit`s get reverted — always
+change config via git). Logs and events ship to Loki; metrics ship to Mimir. Every
+instance stamps `cluster="homelab"` as an external label.
+
+---
+
+## Instances
+
+| Instance | Controller | Collects | Sink |
+|---|---|---|---|
+| `alloy-logs` | DaemonSet (`runAsUser: 0`, one pod per node) | pod logs + node journald | Loki `http://192.168.0.30:3100/loki/api/v1/push` |
+| `alloy-events` | Deployment (1 replica) | Kubernetes events | Loki (same endpoint) |
+| `alloy-metrics` | Deployment (1 replica) | all ServiceMonitors + PodMonitors cluster-wide | Mimir `http://192.168.0.30:9009/api/v1/push` |
+
+`alloy-logs` and `alloy-events` are one-per-node / cluster-scoped respectively —
+`alloy-metrics` is also a single Deployment, never a DaemonSet, since its
+Prometheus-Operator discovery is already cluster-wide (a DaemonSet would scrape
+every target once per node).
+
+---
+
+## Files & Flux wiring
+
+All manifests live in `gitops/home-kubernetes/alloy/`:
+
+- `helmrepository_grafana.yaml` — the shared `grafana` HelmRepository.
+- `helmrelease_alloy-logs.yaml` — logs DaemonSet.
+- `helmrelease_allow-events.yaml` — events Deployment (note the `allow` filename typo, kept as-is).
+- `helmrelease_alloy-metrics.yaml` — metrics Deployment.
+- `rbac_log-collector.yaml` — ClusterRole `alloy-log-reader`, bound to the `alloy-logs`/`alloy-events` ServiceAccounts.
+- `rbac_metrics-collector.yaml` — ClusterRole `alloy-metrics-reader`, bound to the `alloy-metrics` ServiceAccount.
+
+Each HelmRelease sets `rbac.create: false` — RBAC is supplied manually by the two
+files above rather than by the chart, since the chart's default ClusterRole
+doesn't cover everything each pipeline needs (journald/events reads, or
+ServiceMonitor/PodMonitor/kubelet-metrics reads).
+
+Reconciliation: a Flux `Kustomization` named `alloy` in
+`gitops/home-kubernetes/flux-system/extra-kustomizations.yaml`, pointing at
+`./gitops/home-kubernetes/alloy`, with `dependsOn: kube-prometheus` — it needs
+the `monitoring` namespace and the Prometheus Operator CRDs (ServiceMonitor/
+PodMonitor) to exist first.
+
+---
+
+## How metrics discovery works
+
+`alloy-metrics` runs two Alloy components — `prometheus.operator.servicemonitors`
+and `prometheus.operator.podmonitors` — which discover and scrape **every**
+ServiceMonitor/PodMonitor in the cluster, then forward to
+`prometheus.remote_write` targeting Mimir. There's no bespoke scrape config for
+"standard Kubernetes metrics": kube-prometheus-stack already ships
+ServiceMonitors for kubelet, cAdvisor, kube-state-metrics, node-exporter,
+apiserver, and coredns, so those are picked up automatically. Any future
+ServiceMonitor/PodMonitor added anywhere in the cluster is picked up the same way
+with no Alloy config change.
+
+This is **additive** — kube-prometheus-stack's bundled Prometheus keeps scraping
+the same targets into its own local 60d TSDB. Alloy scraping the same targets a
+second time and shipping to Mimir is expected double-collection, not a bug.
+
+---
+
+## Operating
+
+Status:
+
+```bash
+flux get helmreleases -n monitoring
+kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy -o wide
+```
+
+Force reconcile:
+
+```bash
+flux reconcile kustomization alloy --with-source
+flux reconcile helmrelease alloy-metrics -n monitoring
+```
+
+Logs:
+
+```bash
+kubectl -n monitoring logs deploy/alloy-metrics
+kubectl -n monitoring logs ds/alloy-logs
+kubectl -n monitoring logs deploy/alloy-events
+```
+
+Alloy UI (component graph, target health, remote_write queue status):
+
+```bash
+kubectl -n monitoring port-forward deploy/alloy-metrics 12345:12345
+# open http://localhost:12345
+```
+
+Editing config: change the inline `alloy.configMap.content` block in the
+relevant `helmrelease_*.yaml`, commit, push, then force-reconcile (above).
+Do not `kubectl edit` the generated ConfigMap directly — `driftDetection` will
+revert it on the next reconcile.
+
+---
+
+## Verification / smoke tests
+
+**Metrics reaching Mimir:**
+
+```bash
+curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=up' | jq '.data.result | length'
+# expect > 0
+```
+
+Spot-check a few standard series exist and carry the cluster label:
+
+```bash
+curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=kube_pod_info' | jq '.data.result[0].metric'
+curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=node_cpu_seconds_total' | jq '.data.result | length'
+curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=container_cpu_usage_seconds_total' | jq '.data.result | length'
+# confirm "cluster": "homelab" is present in the returned label sets
+```
+
+**Logs reaching Loki:**
+
+```bash
+curl -s http://192.168.0.30:3100/ready
+# then in Grafana / logcli, query {cluster="homelab"}
+```
+
+---
+
+## Troubleshooting
+
+| Symptom | Likely cause |
+|---|---|
+| `forbidden` errors in `alloy-metrics` logs | `alloy-metrics-reader` ClusterRole missing a resource/verb — check `rbac_metrics-collector.yaml` against what the component is trying to read |
+| No targets shown in the Alloy UI | Prometheus Operator CRDs not installed, or `kube-prometheus` Flux Kustomization not Ready yet (metrics depends on it) |
+| `prometheus.remote_write` shows failed sends / 4xx-5xx | Mimir down, or wrong endpoint — must be `http://192.168.0.30:9009/api/v1/push` exactly |
+| Loki push failing | Loki down at `192.168.0.30:3100`, check with `curl http://192.168.0.30:3100/ready` |
+| Same series appearing to be scraped twice | Expected — kube-prometheus-stack's bundled Prometheus and `alloy-metrics` both scrape the same ServiceMonitors by design (additive, not deduplicated) |
+| Manual `kubectl edit` on the Alloy ConfigMap reverts itself | `driftDetection: enabled` — edit the HelmRelease in git instead |
--- a/docs/plans/2026-05-21-2035-tracing-toggle-runbook.md
+++ b/docs/plans/2026-05-21-2035-tracing-toggle-runbook.md
@@ -0,0 +1,140 @@
+# Tracing enable/disable runbook for K8s components
+
+## Context
+
+The pod-creation tracing plan ([plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md](../../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md)) covers the *one-time setup* of OTLP tracing across kube-apiserver and kubelet, plus the Jaeger backend on docker-29. What's missing is a short operational doc for **toggling tracing on/off as quickly as possible** on the running cluster — useful for short measurement experiments where we want overhead only when we're actively recording.
+
+The Terraform/cloud-init changes already in flight (`master.tf`, `files/manifests/kube-apiserver.yaml`, `files/tracing-config.yaml`) cover *rebuilds*. This runbook covers the *live* cluster: kube-master-31 + kube-node-32 + kube-node-33, K8s 1.32.
+
+Output of this plan: a single new file `docs/kubernetes-tracing.md` containing three runbooks (quick suppress, full enable, full disable) and a verification section. No code or manifest changes — pure documentation.
+
+## What "as quickly as possible" means
+
+There are two speeds of toggling:
+
+1. **Quick suppress** — leave the tracing flags/config in place on the components; just stop Jaeger so span exports go nowhere. No K8s component restart, takes seconds, but components still pay the (small) cost of building spans and attempting to export.
+2. **Full enable / full disable** — edit configs and restart components. apiserver static-pod restart is automatic via kubelet manifest watch (~5–10s); kubelet itself needs `systemctl restart kubelet` on each node (brief NotReady blip per node, no pod evictions). This is what we'd use to truly remove overhead between experiments.
+
+The runbook will present both, with quick-suppress as the default for "I'm done measuring for now" and full-disable for "we don't need this on the cluster anymore".
+
+## Deliverable: `docs/kubernetes-tracing.md`
+
+### Structure
+
+```
+# Kubernetes component tracing
+
+## Components and where the toggles live
+- table: component | config file on node | restart trigger
+  - kube-apiserver | /etc/kubernetes/manifests/kube-apiserver.yaml (flag) +
+                     /etc/kubernetes/tracing-config.yaml (config) | kubelet auto-restarts static pod
+  - kubelet         | /var/lib/kubelet/config.yaml (`tracing:` block) +
+                     kube-system/kubelet-config CM (cluster-wide source of truth) | systemctl restart kubelet
+
+## Quick suppress (no component restart)
+1. `ssh novakj@192.168.0.29 'cd /path/to/tracing && docker compose stop jaeger'`
+2. Components continue running with tracing wired up but spans go nowhere.
+3. To resume: `docker compose start jaeger`.
+Caveat: components still build spans and attempt OTLP export — small CPU/network overhead. For a real "off", use full disable.
+
+## Full enable (live cluster)
+
+Prereqs: Jaeger is up on docker-29 (`docker compose up -d` in `docker-29/tracing/`).
+
+### kube-apiserver (master only)
+On 192.168.0.31:
+```bash
+# 1. Drop the TracingConfiguration file (same content as files/tracing-config.yaml)
+sudo tee /etc/kubernetes/tracing-config.yaml > /dev/null <<'EOF'
+apiVersion: apiserver.config.k8s.io/v1beta1
+kind: TracingConfiguration
+endpoint: 192.168.0.29:4317
+samplingRatePerMillion: 1000000
+EOF
+
+# 2. Back up the apiserver manifest
+sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.pre-tracing
+
+# 3. Add the flag (insert after --tls-private-key-file line)
+sudo sed -i '/--tls-private-key-file=/a\    - --tracing-config-file=/etc/kubernetes/tracing-config.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml
+
+# 4. Wait for kubelet to restart the static pod (~5-10s)
+sudo crictl ps | grep apiserver         # new container id == restarted
+kubectl get --raw /livez                 # should return "ok"
+```
+
+### kubelet (every node: master + workers)
+
+Per-node edit:
+```bash
+# Append the tracing block under kubelet config (idempotent: check first)
+grep -q '^tracing:' /var/lib/kubelet/config.yaml || sudo tee -a /var/lib/kubelet/config.yaml > /dev/null <<'EOF'
+tracing:
+  endpoint: 192.168.0.29:4317
+  samplingRatePerMillion: 1000000
+EOF
+
+sudo systemctl restart kubelet
+```
+
+Stagger across nodes (one at a time) so we don't NotReady the whole cluster at once.
+
+Then make it survive node rebuilds/joins by updating the cluster ConfigMap:
+```bash
+kubectl -n kube-system edit cm kubelet-config
+# add the same `tracing:` block under the `kubelet:` key
+```
+
+## Full disable (live cluster)
+
+### kube-apiserver
+```bash
+# Restore the pre-tracing manifest (preferred — exact revert)
+sudo cp /root/kube-apiserver.yaml.pre-tracing /etc/kubernetes/manifests/kube-apiserver.yaml
+# OR remove just the flag in place:
+sudo sed -i '/--tracing-config-file=/d' /etc/kubernetes/manifests/kube-apiserver.yaml
+
+# Optional: remove the now-unused config
+sudo rm /etc/kubernetes/tracing-config.yaml
+```
+kubelet restarts the static pod automatically.
+
+### kubelet (every node)
+```bash
+sudo sed -i '/^tracing:/,/samplingRatePerMillion:/d' /var/lib/kubelet/config.yaml
+sudo systemctl restart kubelet
+```
+Then strip the `tracing:` block from `kube-system/kubelet-config` CM so node rejoins don't reintroduce it.
+
+## Verification (after enable)
+1. Open `http://192.168.0.29:16686` — Services dropdown shows `apiserver` (appears after the first traced request) and `kubelet`.
+2. Generate end-to-end traffic:
+   ```bash
+   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
+   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
+   kubectl delete pod trace-probe
+   ```
+3. In Jaeger: filter service `apiserver`, look for spans tagged with the probe pod name; filter service `kubelet`, look for `syncPod` spans.
+4. To verify *disable*: after running disable steps, repeat (2). Within a minute, no new spans for `apiserver`/`kubelet` should appear in Jaeger.
+
+## Notes
+- 100% sampling is fine for short experiments; for steady-state leave tracing off (quick suppress is enough between sessions).
+- The flag/config are GA in K8s 1.32 — no `--feature-gates` needed.
+- This runbook only covers apiserver + kubelet. etcd and kube-scheduler tracing are documented as future steps in the parent plan.
+```
+
+## Files to change
+
+| File | Change |
+|---|---|
+| `docs/kubernetes-tracing.md` | **new** — the runbook described above |
+
+No other files modified. The Terraform/cloud-init side stays as already drafted in the parent plan.
+
+## Verification of this plan's output
+
+After the runbook is written:
+
+1. Skim `docs/kubernetes-tracing.md` end-to-end and check that someone unfamiliar with the setup can follow it without opening the parent plan.
+2. Cross-check the sed/tee commands against the actual files on the live master via SSH — confirm line patterns match (e.g., the `--tls-private-key-file=` line is present so the sed insertion lands in the right spot).
+3. Dry-run the quick-suppress flow first (it's the lowest-risk path) and confirm Jaeger UI stops receiving new spans within ~30 s.
--- a/docs/plans/2026-05-21-2310-tracing-host-move-to-docker-29.md
+++ b/docs/plans/2026-05-21-2310-tracing-host-move-to-docker-29.md
@@ -0,0 +1,91 @@
+# Move Jaeger/OTLP host from 192.168.0.30 (docker-30) → 192.168.0.29 (docker-29)
+
+## Context
+
+The Jaeger backend that receives OTLP spans from kube-apiserver and kubelet has been relocated from the `docker-30` host (192.168.0.30) to `docker-29` (192.168.0.29). The new compose stack is already present at [vms-home/docker-29/tracing/](../../vms-home/docker-29/tracing/) and binds `0.0.0.0:4317`/`0.0.0.0:4318`, so the backend side requires no changes.
+
+The OTLP exporter endpoints on the Kubernetes side, plus the runbooks/plans that document them, still point at `192.168.0.30:4317`. They need to be repointed to `192.168.0.29:4317`. Without this change, apiserver and kubelet spans go nowhere once docker-30's Jaeger is shut down.
+
+The other docker-30 references in the repo (Vault, Gitea, Zot registry mirror, Kanidm, nginx) are unrelated to tracing and must be left alone.
+
+## Scope
+
+In-repo file edits only. Live cluster updates are documented as a follow-up checklist; no live execution is part of this plan.
+
+## File-level changes
+
+### 1. Live OTEL config (load-bearing)
+
+**[kubernetes-kvm-terraform/files/tracing-config.yaml:3](../../kubernetes-kvm-terraform/files/tracing-config.yaml#L3)**
+- `endpoint: 192.168.0.30:4317` → `endpoint: 192.168.0.29:4317`
+
+**[kubernetes-kvm-terraform/master.tf](../../kubernetes-kvm-terraform/master.tf)** — two occurrences inside cloud-init heredocs:
+- Line 122 (apiserver `TracingConfiguration` written to `/etc/kubernetes/tracing-config.yaml`): `192.168.0.30:4317` → `192.168.0.29:4317`
+- Line 169 (`KubeletConfiguration.tracing.endpoint` in the kubeadm config): `192.168.0.30:4317` → `192.168.0.29:4317`
+- **Do NOT touch line 8** (`zot_registry_ip = "192.168.0.30"`) — that's the Zot container-registry mirror, not tracing.
+
+### 2. Runbook docs
+
+**[docs/kubernetes-tracing.md](../kubernetes-tracing.md)** — replace tracing-related references throughout (≈10 lines):
+- All occurrences of `192.168.0.30:4317` → `192.168.0.29:4317` (lines 54, 83, 99)
+- `http://192.168.0.30:16686` → `http://192.168.0.29:16686` (line 150)
+- SSH targets `novakj@192.168.0.30` → `novakj@192.168.0.29` (lines 25, 31, 43)
+- Remote directory `~/docker-30/tracing` → `~/docker-29/tracing` (lines 25, 31, 43)
+- Prose "Jaeger on docker-30" → "Jaeger on docker-29" (lines 3, 40)
+
+### 3. Plan files (historical record — also requested)
+
+**[plans/2026-05-21 20:15 - k8s-pod-creation-tracing.md](../../plans/2026-05-21%2020%3A15%20-%20k8s-pod-creation-tracing.md)** — ~12 references:
+- `192.168.0.30:4317` (lines 38, 76, 101, 111) and `http://192.168.0.30:16686` (line 141) → `.29`
+- `docker-30/tracing/` directory path (lines 13, 16, 117, 128, 155) → `docker-29/tracing/`
+- Prose mentions of "docker-30" in tracing context (lines 5, 11) → "docker-29"
+
+**[docs/plans/2026-05-21-2035-tracing-toggle-runbook.md](2026-05-21-2035-tracing-toggle-runbook.md)** — ~6 references:
+- `192.168.0.30:4317` (lines 51, 73) and `http://192.168.0.30:16686` (line 110) → `.29`
+- SSH target `novakj@192.168.0.30` (line 35) → `novakj@192.168.0.29`
+- Prose mentions of "docker-30" in tracing context (lines 5, 42) → "docker-29"
+
+### Search safety
+
+Use a per-file targeted replace with surrounding context (e.g. `endpoint: 192.168.0.30:4317`, `novakj@192.168.0.30`, `docker-30/tracing`), not a blanket repo-wide `sed`. The leave-alone list is large (Zot, Vault, Gitea, Kanidm, nginx, gitignore, claude settings, generated tfstate, commented-out blackbox/prometheus examples) — see Phase 1 exploration for the full inventory.
+
+## Verification
+
+### In-repo
+```bash
+# Confirm no tracing-related .30 references remain
+grep -nE '192\.168\.0\.30:(4317|16686)' -r .
+grep -nE 'docker-30/tracing' -r .
+# Confirm new IP is in place
+grep -nE '192\.168\.0\.29:4317' kubernetes-kvm-terraform/
+```
+Both `grep` calls in step 1 should return empty. The leave-alone Zot/Vault/Gitea/nginx hits at port 3000/8200/9443/etc. should still be present (sanity check that the surgical replace didn't over-match).
+
+Re-run `tofu plan` in [kubernetes-kvm-terraform/](../../kubernetes-kvm-terraform/) to confirm the diff only shows the two endpoint changes and no unrelated drift.
+
+### Live cluster follow-up (out of scope for this plan's execution, documented for the operator)
+
+The in-repo changes only affect newly-bootstrapped nodes. The existing master + workers still have the old endpoint baked into their live files. After merging the repo changes, run:
+
+1. **kube-apiserver** (master, 192.168.0.31):
+   ```bash
+   ssh novakj@192.168.0.31 'sudo sed -i s/192.168.0.30:4317/192.168.0.29:4317/ /etc/kubernetes/tracing-config.yaml'
+   # kubelet auto-restarts the static pod within ~10s
+   ```
+2. **kubelet** on each node (192.168.0.31, .32, .33), serially:
+   ```bash
+   ssh novakj@$NODE 'sudo sed -i s/192.168.0.30:4317/192.168.0.29:4317/ /var/lib/kubelet/config.yaml && sudo systemctl restart kubelet'
+   kubectl wait node --for=condition=Ready --timeout=60s <node-name>
+   ```
+3. **Cluster ConfigMap** so future joins/reboots pick it up:
+   ```bash
+   kubectl -n kube-system edit cm kubelet-config
+   # update the tracing.endpoint value under kubelet:
+   ```
+4. **End-to-end smoke test** (from [docs/kubernetes-tracing.md](../kubernetes-tracing.md) "Verification" section):
+   ```bash
+   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
+   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
+   kubectl delete pod trace-probe
+   ```
+   Then open `http://192.168.0.29:16686` and confirm `apiserver` and `kubelet` services show recent spans for `trace-probe`.
--- a/docs/plans/2026-07-08-0034-deploy-mimir.md
+++ b/docs/plans/2026-07-08-0034-deploy-mimir.md
@@ -0,0 +1,165 @@
+# Deploy Mimir on docker-30 (mirror the Loki setup)
+
+## Context
+
+Loki is already deployed on the docker host (`192.168.0.30`) under
+`vms-home/docker-30/loki/`, storing chunks in Garage S3 and queried through a
+shared Grafana. The user wants Mimir deployed for **metrics** in the same style:
+self-contained compose dir, Garage S3 backend, `.env`-supplied credentials,
+env-expanded config.
+
+Decisions made:
+- **Mimir only for now.** No producer is wired up in this plan; Mimir simply
+  exposes its remote_write endpoint (`http://192.168.0.30:9009/api/v1/push`) for
+  a Prometheus/Alloy to point at later.
+- **Reuse the existing Grafana** (the one in the Loki compose) by adding a
+  Prometheus-type datasource — no second Grafana.
+
+Mimir runs in **monolithic mode** (`-target=all`, single binary), which is the
+right shape for a single-node home lab and matches Loki's single-process model.
+
+## Reference: how Loki does it (the pattern to copy)
+
+`vms-home/docker-30/loki/loki-config.yaml` + `docker-compose.yaml`:
+- Garage S3 at `192.168.0.30:3900`, `region: garage`, `s3forcepathstyle: true`,
+  `insecure: true`, bucket `loki-chunks`.
+- Creds via `${GARAGE_ACCESS_KEY}` / `${GARAGE_SECRET_KEY}` from `.env`,
+  expanded with `-config.expand-env=true`.
+- `obs` bridge network; Grafana published on `3001`.
+
+## Files to create (new dir `vms-home/docker-30/mimir/`)
+
+### 1. `mimir-config.yaml` — monolithic + Garage S3
+
+```yaml
+multitenancy_enabled: false            # single-tenant; Loki mirrors this (auth_enabled:false)
+
+server:
+  http_listen_port: 9009
+  grpc_listen_port: 9095
+  log_level: info
+
+common:
+  storage:
+    backend: s3
+    s3:
+      endpoint: 192.168.0.30:3900      # no scheme; insecure toggles http
+      region: garage                   # MUST byte-match s3_region in garage.toml
+      access_key_id: ${GARAGE_ACCESS_KEY}
+      secret_access_key: ${GARAGE_SECRET_KEY}
+      insecure: true                   # http on LAN
+      bucket_lookup_type: path         # path-style — Garage requirement (== s3forcepathstyle)
+
+blocks_storage:
+  s3: { bucket_name: mimir-blocks }
+  tsdb:        { dir: /data/tsdb }
+  bucket_store:{ sync_dir: /data/tsdb-sync }
+
+ruler_storage:
+  s3: { bucket_name: mimir-ruler }
+
+alertmanager_storage:
+  s3: { bucket_name: mimir-alertmanager }
+
+compactor:
+  data_dir: /data/compactor
+
+ruler:
+  rule_path: /data/ruler
+
+alertmanager:
+  data_dir: /data/alertmanager
+
+limits:
+  compactor_blocks_retention_period: 744h   # 31d, matches Loki's retention
+  ingestion_rate: 50000                      # samples/s per tenant; bump on 429s
+  ingestion_burst_size: 100000
+```
+
+Note: the three storage components inherit endpoint/creds from `common.storage`;
+only `bucket_name` is overridden per component. Buckets can be collapsed to one
+if preferred, but three is cleaner and Garage bucket creation is cheap.
+
+### 2. `docker-compose.yaml` — copy Loki's, swap image/ports/paths
+
+```yaml
+services:
+  mimir:
+    image: grafana/mimir:2.16.1          # pin latest stable — verify tag before apply
+    container_name: mimir
+    restart: unless-stopped
+    command:
+      - -config.file=/etc/mimir/config.yaml
+      - -config.expand-env=true          # REQUIRED — expands ${GARAGE_*}
+      - -target=all                      # monolithic single-binary mode
+    ports:
+      - "9009:9009"                      # push (/api/v1/push) + query (/prometheus)
+    volumes:
+      - ./mimir-config.yaml:/etc/mimir/config.yaml:ro
+      - ./mimir-data:/data
+    env_file: [.env]
+    networks: [obs]
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://localhost:9009/ready || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 10
+      start_period: 30s
+    logging:
+      driver: json-file
+      options: { max-size: "10m", max-file: "3" }
+
+networks:
+  obs: { driver: bridge }
+```
+
+### 3. `.env` — reuse the same Garage credentials as Loki
+
+```
+GARAGE_ACCESS_KEY=<same key id Loki uses>
+GARAGE_SECRET_KEY=<same secret>
+```
+
+## Manual steps on docker-30 (execution)
+
+1. **Create Garage buckets** and grant the existing Loki key access:
+   ```bash
+   ssh novakj@192.168.0.30
+   for b in mimir-blocks mimir-ruler mimir-alertmanager; do
+     docker exec garage /garage bucket create "$b"
+   done
+   docker exec garage /garage key list           # get the key name Loki already uses
+   for b in mimir-blocks mimir-ruler mimir-alertmanager; do
+     docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
+   done
+   ```
+2. Copy the `mimir/` dir to the host (or `git pull` if this repo is checked out
+   there), fill `.env`, then `cd mimir && docker compose up -d`.
+3. **Add Mimir datasource to the existing Grafana** (`:3001`):
+   - Type: **Prometheus**
+   - URL: `http://192.168.0.30:9009/prometheus`
+   - No `X-Scope-OrgID` header needed (multitenancy disabled).
+
+## Verification
+
+- `curl http://192.168.0.30:9009/ready` → `ready` (also gates the healthcheck).
+- `docker compose ps` shows `mimir` healthy; `docker compose logs -f mimir`
+  clean (no S3 auth / bucket errors — confirms Garage wiring).
+- Smoke-test the write path without a producer:
+  ```bash
+  curl http://192.168.0.30:9009/api/v1/push   # expect 4xx (proto body), NOT conn-refused
+  ```
+- In Grafana → Explore → Mimir datasource, run `up` or
+  `count({__name__!=""})`. Empty until a producer remote_writes — expected.
+- Confirm blocks land in Garage after ~2h (first block flush):
+  `docker exec garage /garage bucket info mimir-blocks`.
+
+## Later (out of scope here): wiring a producer
+
+Point any Prometheus/Alloy at:
+```yaml
+remote_write:
+  - url: http://192.168.0.30:9009/api/v1/push
+```
+The existing `vms/utility-101-shadow/docker/monitoring/prometheus.yml` is the
+natural first candidate.
--- a/docs/plans/2026-07-08-2346-alloy-metrics-to-mimir.md
+++ b/docs/plans/2026-07-08-2346-alloy-metrics-to-mimir.md
@@ -0,0 +1,116 @@
+# Alloy → Mimir: scrape Kubernetes metrics + Prometheus Operator monitors
+
+> On execution, also copy this file into the repo per CLAUDE.md convention:
+> `docs/plans/2026-07-08-HHMM-alloy-metrics-to-mimir.md` (get `HHMM` via `date "+%H%M"`).
+
+## Context
+
+Mimir is now up on docker-30 (`http://192.168.0.30:9009`, single-tenant, plain
+HTTP over LAN) but nothing in the cluster pushes metrics to it. Grafana Alloy
+already exists in the repo but is **logs-only** (ships pod/journal logs + k8s
+events to Loki) and — importantly — the entire `gitops/home-kubernetes/alloy/`
+directory is **not wired into Flux** yet.
+
+kube-prometheus-stack is deployed and active: it provides the Prometheus
+Operator CRDs (ServiceMonitor / PodMonitor), kube-state-metrics, node-exporter,
+and ServiceMonitors for all standard targets (kubelet, cAdvisor, apiserver,
+coredns, etc.). Its bundled Prometheus scrapes everything into a local 60d TSDB
+but does **not** remote_write anywhere.
+
+**Goal:** add a dedicated `alloy-metrics` instance that scrapes standard
+Kubernetes metrics and reads ServiceMonitors/PodMonitors from the Prometheus
+Operator, remote-writing to Mimir — additive alongside the existing Prometheus
+(no changes to kube-prometheus-stack). Because kube-prometheus-stack already
+ships ServiceMonitors for every standard target, a single
+`prometheus.operator.servicemonitors` + `prometheus.operator.podmonitors` pair
+satisfies **both** requirements at once. As part of wiring the `alloy/` dir into
+Flux, the existing `alloy-logs` and `alloy-events` releases also come under
+GitOps management (confirmed desired).
+
+**Decisions confirmed:** additive/coexist with bundled Prometheus; activate all
+three Alloy instances (logs, events, metrics).
+
+## Design
+
+- Metrics scraping runs as a **single Deployment** (`controller.type: deployment`,
+  1 replica) — never a DaemonSet, or every cluster-wide target would be scraped
+  once per node. This is why it must be a separate release from `alloy-logs`
+  (which is a DaemonSet).
+- Mirror the conventions already in `helmrelease_alloy-logs.yaml`: chart `alloy`
+  `>=1.8.0 <2.0.0`, image `tag: v1.17.1`, `rbac.create: false`,
+  `alloy.enableReporting: false`, `driftDetection: enabled`, install/upgrade
+  `retries: 3`, `interval: 30m`, namespace `monitoring`.
+- River config = three components:
+  1. `prometheus.operator.servicemonitors "sm"` — discovers/scrapes **all**
+     ServiceMonitors cluster-wide (covers kubelet, cAdvisor, kube-state-metrics,
+     node-exporter, apiserver, coredns, controller-manager, scheduler — i.e. the
+     "standard kubernetes metrics").
+  2. `prometheus.operator.podmonitors "pm"` — discovers/scrapes all PodMonitors.
+  3. `prometheus.remote_write "mimir"` → `http://192.168.0.30:9009/api/v1/push`,
+     `external_labels = { cluster = "homelab" }` (matches the logs config).
+  Both operator components `forward_to = [prometheus.remote_write.mimir.receiver]`.
+- Reuse `cluster = "homelab"` external label for consistency with the Loki side.
+
+## Files
+
+### New — `gitops/home-kubernetes/alloy/helmrelease_alloy-metrics.yaml`
+HelmRelease `alloy-metrics` (ns `monitoring`), Deployment, serviceAccount
+`alloy-metrics`, inline `alloy.configMap.content` with the three components
+above. Modest resources (e.g. requests cpu 100m / mem 256Mi, limit mem 768Mi —
+operator discovery + scrape buffers use more than the log shipper).
+
+### New — `gitops/home-kubernetes/alloy/rbac_metrics-collector.yaml`
+The `prometheus.operator.*` components need read access beyond the existing
+`alloy-log-reader` role. ClusterRole `alloy-metrics-reader` + ClusterRoleBinding
+to SA `alloy-metrics` (monitoring), granting:
+- core: `namespaces, nodes, nodes/metrics, nodes/proxy, services, endpoints, pods` — get/list/watch
+- `discovery.k8s.io`: `endpointslices` — get/list/watch
+- `networking.k8s.io`: `ingresses` — get/list/watch
+- `monitoring.coreos.com`: `servicemonitors, podmonitors, probes, scrapeconfigs` — get/list/watch
+- nonResourceURLs `/metrics`, `/metrics/cadvisor` — get (kubelet scraping)
+
+(Kept as a separate file so the metrics RBAC is self-contained; the existing
+`rbac_log-collector.yaml` stays untouched.)
+
+### New — `gitops/home-kubernetes/alloy/kustomization.yaml`
+`kind: Kustomization` listing all manifests in the dir: `helmrepository_grafana.yaml`,
+`helmrelease_alloy-logs.yaml`, `helmrelease_allow-events.yaml`,
+`helmrelease_alloy-metrics.yaml`, `rbac_log-collector.yaml`,
+`rbac_metrics-collector.yaml`. Makes the Flux build explicit/deterministic.
+
+### Modified — `gitops/home-kubernetes/flux-system/extra-kustomizations.yaml`
+Append an `alloy` Flux Kustomization:
+```yaml
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: alloy
+  namespace: flux-system
+spec:
+  interval: 10m0s
+  path: ./gitops/home-kubernetes/alloy
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  dependsOn:
+    - name: kube-prometheus   # needs the monitoring ns + operator CRDs/ServiceMonitors
+```
+
+## Verification
+
+1. Commit + push; force reconcile: `flux reconcile kustomization flux-system --with-source`.
+2. `kubectl get kustomizations -A` — `alloy` Ready=True.
+3. `kubectl get helmreleases -n monitoring` — `alloy-metrics` (+ logs/events) Ready.
+4. `kubectl -n monitoring get pods -l app.kubernetes.io/instance=alloy-metrics`
+   Running; `kubectl -n monitoring logs deploy/alloy-metrics` shows no RBAC
+   `forbidden` errors and targets discovered.
+5. Alloy UI: `kubectl -n monitoring port-forward deploy/alloy-metrics 12345:12345`
+   → open `http://localhost:12345`; confirm the two operator components list
+   healthy targets and `prometheus.remote_write.mimir` shows successful sends.
+6. Confirm Mimir is receiving: query it directly, e.g.
+   `curl -s 'http://192.168.0.30:9009/prometheus/api/v1/query?query=up' | jq '.data.result | length'`
+   returns > 0 (or check the Grafana Mimir datasource for `up`,
+   `kube_pod_info`, `node_cpu_seconds_total`, `container_cpu_usage_seconds_total`).
+7. Sanity: a `cluster="homelab"` label is present on the ingested series.
--- a/gitops/home-kubernetes/alloy/helmrelease_allow-events.yaml
+++ b/gitops/home-kubernetes/alloy/helmrelease_allow-events.yaml
@@ -0,0 +1,42 @@
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: alloy-events
+  namespace: monitoring
+spec:
+  interval: 30m
+  chart:
+    spec:
+      chart: alloy
+      version: ">=1.8.0 <2.0.0"
+      sourceRef: { kind: HelmRepository, name: grafana, namespace: monitoring }
+      interval: 12h
+  install: { remediation: { retries: 3 } }
+  upgrade: { remediation: { retries: 3 } }
+  driftDetection: { mode: enabled }
+  values:
+    controller:
+      type: deployment
+      replicas: 1                          # cluster-scoped → exactly one
+    image:
+      tag: v1.17.1
+    rbac:           { create: false }
+    serviceAccount: { create: true, name: alloy-events }
+    alloy:
+      enableReporting: false
+      resources:
+        requests: { cpu: 50m, memory: 96Mi }
+        limits:   { memory: 256Mi }
+      configMap:
+        content: |-
+          loki.source.kubernetes_events "events" {
+            forward_to = [loki.process.events.receiver]
+          }
+          loki.process "events" {
+            stage.static_labels { values = { source = "kubernetes-events" } }
+            forward_to = [loki.write.loki.receiver]
+          }
+          loki.write "loki" {
+            endpoint { url = "http://192.168.0.30:3100/loki/api/v1/push" }
+            external_labels = { cluster = "homelab" }
+          }
--- a/gitops/home-kubernetes/alloy/helmrelease_alloy-logs.yaml
+++ b/gitops/home-kubernetes/alloy/helmrelease_alloy-logs.yaml
@@ -0,0 +1,106 @@
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: alloy-logs
+  namespace: monitoring
+spec:
+  interval: 30m
+  chart:
+    spec:
+      chart: alloy
+      version: ">=1.8.0 <2.0.0"          # newest 1.x each reconcile; pin exact if you want determinism
+      sourceRef: { kind: HelmRepository, name: grafana, namespace: monitoring }
+      interval: 12h                        # poll cadence for new chart versions
+  install: { remediation: { retries: 3 } }
+  upgrade: { remediation: { retries: 3 } }
+  driftDetection: { mode: enabled }        # revert manual kubectl edits
+  values:
+    controller:
+      type: daemonset
+    image:
+      tag: v1.17.1
+    rbac:           { create: false }
+    serviceAccount: { create: true, name: alloy-logs }
+    alloy:
+      enableReporting: false
+      securityContext: { runAsUser: 0, runAsGroup: 0 }   # journald read
+      mounts:
+        varlog: true
+      resources:
+        requests: { cpu: 100m, memory: 128Mi }
+        limits:   { memory: 512Mi }
+      extraEnv:
+        - name: NODE_NAME
+          valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
+      configMap:
+        content: |-
+          discovery.kubernetes "pods" {
+            role = "pod"
+            selectors {
+              role  = "pod"
+              field = "spec.nodeName=" + sys.env("NODE_NAME")
+            }
+          }
+          discovery.relabel "pods" {
+            targets = discovery.kubernetes.pods.targets
+            rule {
+              source_labels = ["__meta_kubernetes_namespace"]
+              target_label  = "namespace"
+            }
+            rule {
+              source_labels = ["__meta_kubernetes_pod_label_app"]
+              target_label  = "app"
+            }
+            rule {
+              source_labels = ["__meta_kubernetes_pod_container_name"]
+              target_label  = "container"
+            }
+            rule {
+              source_labels = ["__meta_kubernetes_pod_node_name"]
+              target_label  = "node"
+            }
+            rule {
+              source_labels = ["__meta_kubernetes_pod_name"]
+              target_label  = "pod"
+            }
+          }
+          loki.source.kubernetes "pods" {
+            targets    = discovery.relabel.pods.output
+            forward_to = [loki.process.pods.receiver]
+          }
+          loki.process "pods" {
+            stage.static_labels {
+              values = { source = "kubernetes" }
+            }
+            forward_to = [loki.write.loki.receiver]
+          }
+
+          discovery.relabel "journal" {
+            targets = []
+            rule {
+              source_labels = ["__journal__systemd_unit"]
+              target_label  = "unit"
+            }
+            rule {
+              source_labels = ["__journal__hostname"]
+              target_label  = "node"
+            }
+            rule {
+              source_labels = ["__journal_priority_keyword"]
+              target_label  = "level"
+            }
+          }
+          loki.source.journal "node" {
+            path          = "/var/log/journal"
+            max_age       = "12h"
+            labels        = { source = "node-journal" }
+            relabel_rules = discovery.relabel.journal.rules
+            forward_to    = [loki.write.loki.receiver]
+          }
+
+          loki.write "loki" {
+            endpoint {
+              url = "http://192.168.0.30:3100/loki/api/v1/push"
+            }
+            external_labels = { cluster = "homelab" }
+          }
--- a/gitops/home-kubernetes/alloy/helmrelease_alloy-metrics.yaml
+++ b/gitops/home-kubernetes/alloy/helmrelease_alloy-metrics.yaml
@@ -0,0 +1,49 @@
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: alloy-metrics
+  namespace: monitoring
+spec:
+  interval: 30m
+  chart:
+    spec:
+      chart: alloy
+      version: ">=1.8.0 <2.0.0"
+      sourceRef: { kind: HelmRepository, name: grafana, namespace: monitoring }
+      interval: 12h
+  install: { remediation: { retries: 3 } }
+  upgrade: { remediation: { retries: 3 } }
+  driftDetection: { mode: enabled }
+  values:
+    controller:
+      type: deployment
+      replicas: 1                          # cluster-scoped scrape target discovery → exactly one
+    image:
+      tag: v1.17.1
+    rbac:           { create: false }
+    serviceAccount: { create: true, name: alloy-metrics }
+    alloy:
+      enableReporting: false
+      resources:
+        requests: { cpu: 100m, memory: 256Mi }
+        limits:   { memory: 768Mi }
+      configMap:
+        content: |-
+          // Discovers every ServiceMonitor cluster-wide. kube-prometheus-stack already ships
+          // ServiceMonitors for kubelet/cAdvisor, kube-state-metrics, node-exporter, apiserver,
+          // coredns, controller-manager and scheduler — so this alone covers "standard k8s metrics".
+          prometheus.operator.servicemonitors "sm" {
+            forward_to = [prometheus.remote_write.mimir.receiver]
+          }
+
+          // Discovers every PodMonitor cluster-wide, same selector scope as above.
+          prometheus.operator.podmonitors "pm" {
+            forward_to = [prometheus.remote_write.mimir.receiver]
+          }
+
+          prometheus.remote_write "mimir" {
+            endpoint {
+              url = "http://192.168.0.30:9009/api/v1/push"
+            }
+            external_labels = { cluster = "homelab" }
+          }
--- a/gitops/home-kubernetes/alloy/helmrepository_grafana.yaml
+++ b/gitops/home-kubernetes/alloy/helmrepository_grafana.yaml
@@ -0,0 +1,8 @@
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: grafana
+  namespace: monitoring
+spec:
+  interval: 1h
+  url: https://grafana.github.io/helm-charts
--- a/gitops/home-kubernetes/alloy/rbac_log-collector.yaml
+++ b/gitops/home-kubernetes/alloy/rbac_log-collector.yaml
@@ -0,0 +1,15 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata: { name: alloy-log-reader }
+rules:
+  - apiGroups: [""]
+    resources: ["pods", "pods/log", "namespaces", "nodes", "nodes/proxy", "events"]
+    verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata: { name: alloy-log-reader }
+roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: alloy-log-reader }
+subjects:
+  - { kind: ServiceAccount, name: alloy-logs,   namespace: monitoring }
+  - { kind: ServiceAccount, name: alloy-events, namespace: monitoring }
--- a/gitops/home-kubernetes/alloy/rbac_metrics-collector.yaml
+++ b/gitops/home-kubernetes/alloy/rbac_metrics-collector.yaml
@@ -0,0 +1,25 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata: { name: alloy-metrics-reader }
+rules:
+  - apiGroups: [""]
+    resources: ["namespaces", "nodes", "nodes/metrics", "nodes/proxy", "services", "endpoints", "pods"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["discovery.k8s.io"]
+    resources: ["endpointslices"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["networking.k8s.io"]
+    resources: ["ingresses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["monitoring.coreos.com"]
+    resources: ["servicemonitors", "podmonitors", "probes", "scrapeconfigs"]
+    verbs: ["get", "list", "watch"]
+  - nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
+    verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata: { name: alloy-metrics-reader }
+roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: alloy-metrics-reader }
+subjects:
+  - { kind: ServiceAccount, name: alloy-metrics, namespace: monitoring }
--- a/gitops/home-kubernetes/alloy/readme.md
+++ b/gitops/home-kubernetes/alloy/readme.md
@@ -0,0 +1,10 @@
+## debugging alloy
+
+```bash
+POD=$(kubectl -n monitoring get pod -l app.kubernetes.io/name=alloy -o name | head -1)
+kubectl -n monitoring debug -it $POD --image=nicolaka/netshoot --target=alloy -- bash
+
+
+curl -v http://192.168.0.30:3100/ready
+
+```
--- a/gitops/home-kubernetes/flux-system/extra-kustomizations.yaml
+++ b/gitops/home-kubernetes/flux-system/extra-kustomizations.yaml
@@ -81,6 +81,21 @@ spec:
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
+metadata:
+  name: alloy
+  namespace: flux-system
+spec:
+  interval: 10m0s
+  path: ./gitops/home-kubernetes/alloy
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  dependsOn:
+    - name: kube-prometheus   # needs the monitoring ns + operator CRDs/ServiceMonitors
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
 metadata:
  name: ingress-nginx
  namespace: flux-system
--- a/kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml
+++ b/kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml
@@ -44,6 +44,7 @@ spec:
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
+    - --tracing-config-file=/etc/kubernetes/tracing-config.yaml
    image: registry.k8s.io/kube-apiserver:v1.32.11
    imagePullPolicy: IfNotPresent
    livenessProbe:
--- a/kubernetes-kvm-terraform/files/tracing-config.yaml
+++ b/kubernetes-kvm-terraform/files/tracing-config.yaml
@@ -0,0 +1,4 @@
+apiVersion: apiserver.config.k8s.io/v1beta1
+kind: TracingConfiguration
+endpoint: 192.168.0.29:4317
+samplingRatePerMillion: 1000000
--- a/kubernetes-kvm-terraform/master.tf
+++ b/kubernetes-kvm-terraform/master.tf
@@ -115,6 +115,13 @@ locals {
            skip_verify = true
            override_path = true              

+      - path: /etc/kubernetes/tracing-config.yaml
+        content: |
+          apiVersion: apiserver.config.k8s.io/v1beta1
+          kind: TracingConfiguration
+          endpoint: 192.168.0.29:4317
+          samplingRatePerMillion: 1000000
+
      - path: /etc/kubernetes/auth-config.yaml
        content: |
          apiVersion: apiserver.config.k8s.io/v1beta1
@@ -158,6 +165,9 @@ locals {
          apiVersion: kubelet.config.k8s.io/v1beta1
          kind: KubeletConfiguration
          cgroupDriver: systemd
+          tracing:
+            endpoint: 192.168.0.29:4317
+            samplingRatePerMillion: 1000000

      - path: /etc/profile.d/kubectl.sh
        content: |
--- a/k8s-pod-creation-tracing.md
+++ b/k8s-pod-creation-tracing.md
@@ -0,0 +1,157 @@
+# Pod-creation latency tracing with OpenTelemetry
+
+## Context
+
+Goal: measure end-to-end latency of `kubectl apply` → pod running, broken down by component. The approach is to enable native OpenTelemetry tracing in Kubernetes 1.32 components (kube-apiserver, kubelet, etcd) and ship spans to a Jaeger backend running outside the cluster on `docker-29`. With all three reporting under the same trace id (object-resource correlation), Jaeger will let us see how much time is spent in admission/etcd, on the wire, and inside the kubelet's pod sync loop.
+
+Constraints / known gaps:
+- **kube-scheduler has no native OTLP tracing yet**; there will be a gap between "object stored" and "kubelet sees assignment". We can infer it from event timestamps (`kubectl get events`) or `.status.conditions` PodScheduled/Initialized/ContainersReady/Ready timestamps.
+- Cluster is K8s 1.32, kubeadm-bootstrapped via Terraform/cloud-init. Per Phase-3 decision, the canonical source-of-truth is the Terraform cloud-init; changes there cover future rebuilds. To activate tracing on the *currently running* nodes we also need a one-time manual application step (documented at the bottom).
+
+## Backend: Jaeger on docker-29
+
+New directory `docker-29/tracing/` containing `docker-compose.yaml`. Uses Jaeger all-in-one v1 with Badger on-disk storage and the built-in OTLP receiver enabled.
+
+```yaml
+# docker-29/tracing/docker-compose.yaml
+services:
+  jaeger:
+    image: jaegertracing/all-in-one:1.62
+    container_name: jaeger
+    restart: unless-stopped
+    environment:
+      SPAN_STORAGE_TYPE: badger
+      BADGER_EPHEMERAL: "false"
+      BADGER_DIRECTORY_VALUE: /badger/data
+      BADGER_DIRECTORY_KEY: /badger/key
+      BADGER_SPAN_STORE_TTL: 4h       # auto-expire traces older than 4h
+      BADGER_MAINTENANCE_INTERVAL: 5m # GC cycle that reclaims disk
+      COLLECTOR_OTLP_ENABLED: "true"
+    volumes:
+      - ./badger:/badger
+    ports:
+      - "4317:4317"    # OTLP gRPC (apiserver + kubelet exporters)
+      - "4318:4318"    # OTLP HTTP (optional)
+      - "16686:16686"  # Jaeger UI
+```
+
+Endpoint that the cluster will send to: `192.168.0.29:4317`. Jaeger UI: `http://192.168.0.29:16686`.
+
+The OTLP receiver in Jaeger all-in-one accepts **plaintext** OTLP/gRPC by default — matches `endpoint: ...:4317` on the K8s side with no TLS. Acceptable for a home lab on a private LAN.
+
+### Storage sizing and retention
+
+Two complementary levers control how much disk Badger uses:
+
+1. **TTL** — `BADGER_SPAN_STORE_TTL=4h` deletes spans older than 4 hours; Badger's value-log GC reclaims disk on `BADGER_MAINTENANCE_INTERVAL` (default 5m). This is the hard upper bound on retention.
+2. **Sampling rate** — controls *how many* spans get written in the first place. Set in the apiserver/kubelet TracingConfiguration (`samplingRatePerMillion`).
+
+Rough estimate for this cluster (1 master + 2 workers, mostly idle, Flux reconciling continuously):
+
+| Sampling | Span ingest rate | Disk footprint at 4h TTL |
+|---|---|---|
+| 100% (`1000000`) | ~500–3000 spans/s | ~6–40 GB steady-state |
+| 1% (`10000`) | ~5–30 spans/s | ~60–400 MB steady-state |
+| 0.1% (`1000`) | ~0.5–3 spans/s | ~6–40 MB steady-state |
+
+Note: 100% sampling on this cluster is sized for short experiments, not long-term operation. The 4h TTL prevents runaway growth if I forget to lower it, but Badger has no built-in *size* cap — only TTL + GC. If a hard size cap is needed, mount `./badger/` from a fixed-size LVM volume or zfs dataset with a quota.
+
+**Recommended progression**:
+- Start with `samplingRatePerMillion: 1000000` (100%) and `TTL=4h` so the first measurements are easy to find.
+- After the first round of experiments, drop sampling to `10000` (1%) for steady-state and adjust `TTL` upward if a longer history is useful.
+
+## kube-apiserver tracing
+
+Two changes, both in the Terraform cloud-init / referenced manifest:
+
+### 1. Write the TracingConfiguration file (master.tf cloud-init)
+
+Add to `write_files` in [kubernetes-kvm-terraform/master.tf](kubernetes-kvm-terraform/master.tf):
+
+```yaml
+- path: /etc/kubernetes/tracing-config.yaml
+  content: |
+    apiVersion: apiserver.config.k8s.io/v1beta1
+    kind: TracingConfiguration
+    endpoint: 192.168.0.29:4317
+    samplingRatePerMillion: 1000000  # 100% sampling for measurement work
+```
+
+The apiserver static pod already mounts `/etc/kubernetes` read-only via the `k8s-config` volume, so the file becomes visible inside the container automatically — no new volume mount needed.
+
+### 2. Add the apiserver flag
+
+Edit [kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml](kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml) — add one new flag in the `command:` list (alphabetical placement after `--tls-private-key-file` works; ordering isn't enforced):
+
+```yaml
+- --tracing-config-file=/etc/kubernetes/tracing-config.yaml
+```
+
+`APIServerTracing` is GA in 1.32 — no feature-gate flag required.
+
+## kubelet tracing
+
+Add a `tracing:` block to the existing `KubeletConfiguration` document inside the kubeadm-config heredoc in [kubernetes-kvm-terraform/master.tf](kubernetes-kvm-terraform/master.tf) (around line 158):
+
+```yaml
+apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+cgroupDriver: systemd
+tracing:
+  endpoint: 192.168.0.29:4317
+  samplingRatePerMillion: 1000000
+```
+
+`KubeletTracing` is beta and on by default in 1.32 — no feature gate needed.
+
+Kubeadm stores this `KubeletConfiguration` in the `kubelet-config` ConfigMap in `kube-system`, so workers joining via `kubeadm join` pick the tracing block up automatically. The two existing workers (kube-node-32, kube-node-33) need a one-time manual nudge (see below) because they were already joined before the change.
+
+## etcd tracing (optional, second-step)
+
+etcd supports OTLP via flags `--experimental-enable-distributed-tracing`, `--experimental-distributed-tracing-address=192.168.0.29:4317`, and `--experimental-distributed-tracing-sampling-rate=1000000`. Enabling these requires editing the etcd static-pod manifest on the master — propose deferring until after the apiserver+kubelet path is verified, so we keep the first iteration narrow.
+
+## Files to change
+
+| File | Change |
+|---|---|
+| `docker-29/tracing/docker-compose.yaml` | **new** — Jaeger all-in-one + Badger |
+| `kubernetes-kvm-terraform/master.tf` | add `tracing-config.yaml` to `write_files`; add `tracing:` block to `KubeletConfiguration` |
+| `kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml` | add `--tracing-config-file=/etc/kubernetes/tracing-config.yaml` |
+| `kubernetes-kvm-terraform/files/tracing-config.yaml` | **new** — reference copy of the TracingConfiguration (mirrors the inline content in master.tf, matches the existing `auth-config.yaml` convention) |
+
+No changes to `nodes-on-homer.tf` / `nodes-on-beelink.tf` are needed — workers receive kubelet config from the cluster ConfigMap on join.
+
+## Applying to the currently running cluster
+
+The Terraform changes cover future rebuilds. To activate tracing **now** without rebuilding:
+
+1. **Start Jaeger** on docker-29: `cd docker-29/tracing && docker compose up -d` (creates `./badger/` for storage).
+2. **On the master (kube-master-31, 192.168.0.31)**:
+   - `scp` or write `/etc/kubernetes/tracing-config.yaml` with the TracingConfiguration shown above.
+   - Edit `/etc/kubernetes/manifests/kube-apiserver.yaml` to add `--tracing-config-file=/etc/kubernetes/tracing-config.yaml`. Kubelet will detect the manifest change and restart the apiserver static pod within seconds.
+   - Watch: `crictl ps | grep apiserver`, then `kubectl get --raw /livez`.
+3. **Update the kubelet-config ConfigMap** in `kube-system`:
+   - `kubectl -n kube-system edit cm kubelet-config` and add the same `tracing:` block under `kubelet:`.
+4. **On every node** (master + both workers): edit `/var/lib/kubelet/config.yaml` to append the `tracing:` block, then `systemctl restart kubelet`. Pods will *not* be evicted by a kubelet restart, but expect a brief NotReady blip per node — stagger the restarts.
+
+## Verification
+
+End-to-end test once everything is live:
+
+1. UI reachable: open `http://192.168.0.29:16686` and check "Services" dropdown populates with `apiserver` (it will appear only after the first traced request).
+2. Generate traffic that exercises the full path:
+   ```
+   kubectl run trace-probe --image=registry.k8s.io/pause:3.10 --restart=Never
+   kubectl wait --for=condition=Ready pod/trace-probe --timeout=60s
+   kubectl delete pod trace-probe
+   ```
+3. In Jaeger UI, search service `apiserver`, operation `KubernetesAPI` (or similar) — you should see a trace covering admission/etcd write. Search service `kubelet`, operation `syncPod` — kubelet-side pod sync spans should be visible.
+4. Cross-check the gap (scheduler) by comparing pod `metadata.creationTimestamp`, the `PodScheduled` condition `lastTransitionTime`, and the kubelet span start. The diff between those values quantifies the scheduler portion that we cannot trace natively.
+5. Confirm Badger persistence: `docker compose restart jaeger`, reload UI — earlier traces should still be present.
+
+## Notes / risks
+
+- 100% sampling rate is fine for measurement; once we're done we should drop to e.g. `samplingRatePerMillion: 10000` (1%) so storage doesn't explode.
+- Badger volume `./badger/` on docker-29 will grow with span volume; size budget worth watching but not urgent.
+- `tracing.endpoint` is plaintext OTLP/gRPC and traverses the LAN — fine for a home lab but explicitly *not* encrypted.
+- If apiserver fails to come back after the manifest edit, the most common cause is a typo in the new flag; revert is straightforward — keep a copy of the original `/etc/kubernetes/manifests/kube-apiserver.yaml` in `/root/` before editing.
--- a/servers/barber/readme.md
+++ b/servers/barber/readme.md
@@ -0,0 +1,136 @@
+# Barber
+
+## network
+
+```bash
+# ip: 87.236.197.158, 87.236.197.157
+```
+
+
+## TODO
+
+- firewall
+- kvm
+
+## KVM
+
+```bash
+egrep -c '(vmx|svm)' /proc/cpuinfo      # > 0 expected (Intel VT-x / AMD-V)
+
+# qemu-kvm is a transitional metapackage on 26.04; qemu-system-x86 is the real binary.
+apt update
+apt install -y \
+  qemu-system-x86 qemu-utils \
+  libvirt-daemon-system libvirt-clients \
+  virtinst cpu-checker ovmf
+
+apt install virt-manager
+# optional: libguestfs-tools (image surgery), virt-manager (GUI over X-fwd)
+
+
+# Verify acceleration + service
+kvm-ok                  # expect: "KVM acceleration can be used"
+lsmod | grep kvm        # kvm + kvm_intel|kvm_amd loaded
+
+sudo systemctl enable --now libvirtd
+sudo systemctl is-active libvirtd
+
+# non-root management
+sudo usermod -aG libvirt,kvm $USER
+newgrp libvirt          # apply to current shell; otherwise log out/in
+virsh list --all        # should run without sudo
+
+
+# default libvirt network configuration changed to: 192.168.124.*
+```
+
+
+## Firewall
+
+```bash
+ufw default deny incoming
+ufw default allow outgoing
+ufw limit 22/tcp        # SSH — limit, not allow (see below)
+ufw enable              # answer 'y' — safe, the rule's already in
+ufw status verbose
+
+# Pin IP forwarding
+echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-ip-forward.conf
+sudo sysctl --system
+
+# Fix the FORWARD policy (mandatory for NAT guests)
+sudo sed -i 's/^DEFAULT_FORWARD_POLICY=.*/DEFAULT_FORWARD_POLICY="ACCEPT"/' /etc/default/ufw
+sudo ufw reload
+
+# DHCP/DNS to guests (usually automatic)
+sudo ufw allow in on virbr0 to any port 53 proto udp
+sudo ufw allow in on virbr0 to any port 67 proto udp
+
+# dns on utility-101
+ufw route allow proto udp from any to 192.168.124.101 port 53
+ufw route allow proto tcp from any to 192.168.124.101 port 53
+```
+
+```bash
+# ufw fights with libvirtd about some iptables rules
+
+# /etc/libvirt/hooks/network
+cat > /etc/libvirt/hooks/network <<'EOF'
+#!/bin/bash
+# Re-inject inbound DNS ACCEPT into libvirt's LIBVIRT_FWI chain.
+# libvirt flushes/rebuilds this chain on network start, so the rule
+# must be re-applied each time the network comes up.
+
+NET="$1"   # network name (arg 1)
+OP="$2"    # operation   (arg 2)
+
+TARGET_NET="default"      # <-- set to your `virsh net-list` name
+VM_IP="192.168.124.101"
+WAN_IF="enp5s0"
+BR_IF="virbr0"
+
+OSPROVIDER_NET="osprovider"
+OSPROVIDER_BR="virbr2"
+OSPROVIDER_SUBNET="10.10.50.0/24"
+
+[ "$OP" = "started" ] || exit 0
+
+if [ "$NET" = "$TARGET_NET" ]; then
+    for proto in udp tcp; do
+        iptables -C LIBVIRT_FWI -i "$WAN_IF" -o "$BR_IF" -d "$VM_IP" -p "$proto" --dport 53 -j ACCEPT 2>/dev/null \
+          || iptables -I LIBVIRT_FWI 1 -i "$WAN_IF" -o "$BR_IF" -d "$VM_IP" -p "$proto" --dport 53 -j ACCEPT
+    done
+fi
+
+if [ "$NET" = "$OSPROVIDER_NET" ]; then
+    # libvirt's LIBVIRT_FWO chain REJECTs all forwarding from virbr2 because osprovider is
+    # defined as an isolated network. Insert these rules at the top of FORWARD (before the
+    # LIBVIRT_* chains run) so provider-network traffic can reach POSTROUTING/MASQUERADE.
+    iptables -C FORWARD -i "$OSPROVIDER_BR" -s "$OSPROVIDER_SUBNET" -j ACCEPT 2>/dev/null \
+      || iptables -I FORWARD 1 -i "$OSPROVIDER_BR" -s "$OSPROVIDER_SUBNET" -j ACCEPT
+    iptables -C FORWARD -o "$OSPROVIDER_BR" -d "$OSPROVIDER_SUBNET" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2>/dev/null \
+      || iptables -I FORWARD 1 -o "$OSPROVIDER_BR" -d "$OSPROVIDER_SUBNET" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+fi
+EOF
+chmod +x /etc/libvirt/hooks/network 
+
+```
+
+## VNC remote desktop
+
+```bash
+apt install tigervnc-standalone-server xfce4 xfce4-goodies
+apt install dbus-x11
+
+# Fix the xstartup to launch Xfce explicitly inside a fresh dbus session:
+cat > ~/.vnc/xstartup <<'EOF'
+#!/bin/sh
+unset SESSION_MANAGER
+unset DBUS_SESSION_BUS_ADDRESS
+export XDG_SESSION_TYPE=x11
+export XDG_CURRENT_DESKTOP=XFCE
+[ -r "$HOME/.Xresources" ] && xrdb "$HOME/.Xresources"
+exec dbus-run-session -- startxfce4
+EOF
+chmod +x ~/.vnc/xstartup
+```
--- a/servers/barber/vms/utility-101-barber.md
+++ b/servers/barber/vms/utility-101-barber.md
--- a/servers/psmf/dns-configuration.md
+++ b/servers/psmf/dns-configuration.md
@@ -0,0 +1,157 @@
+## disable systemd-resolved
+
+```bash
+sudo mkdir -p /etc/systemd/resolved.conf.d
+sudo tee /etc/systemd/resolved.conf.d/disable-stub.conf <<EOF
+[Resolve]
+DNSStubListener=no
+EOF
+
+# Fix /etc/resolv.conf symlink so the box itself still resolves
+sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
+
+sudo systemctl restart systemd-resolved
+```
+
+
+## dnsmasq in ts2 namespace
+
+```bash
+sudo mkdir -p /etc/dnsmasq-ts2
+sudo tee /etc/dnsmasq-ts2/dnsmasq.conf <<'EOF'
+# Inside ts2 netns — tailscale0 is the only real interface.
+# Bind to all (namespace is isolated, so this is safe).
+no-resolv
+no-hosts
+
+# Upstreams (explicit, no /etc/resolv.conf dependency inside netns)
+server=1.1.1.1
+server=9.9.9.9
+
+# Zone
+local=/intranet/
+domain=intranet
+
+address=/psmf.intranet/100.90.25.77
+address=/psmf-new.intranet/100.90.25.77
+
+cache-size=1000
+log-facility=/var/log/dnsmasq-ts2.log
+
+user=dnsmasq
+pid-file=/run/dnsmasq-ts2.pid
+EOF
+```
+
+```bash
+# systemd-unit
+sudo tee /etc/systemd/system/dnsmasq-ts2.service <<'EOF'
+[Unit]
+Description=dnsmasq inside ts2 netns (intranet zone)
+After=network-online.target
+Wants=network-online.target
+# If you have a unit that sets up the ts2 namespace + tailscaled inside it,
+# add it here, e.g.:
+# Requires=tailscaled-ts2.service
+# After=tailscaled-ts2.service
+
+[Service]
+Type=simple
+NetworkNamespacePath=/var/run/netns/ts2
+ExecStartPre=/usr/sbin/dnsmasq --test -C /etc/dnsmasq-ts2/dnsmasq.conf
+ExecStart=/usr/sbin/dnsmasq -k -C /etc/dnsmasq-ts2/dnsmasq.conf
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+EOF
+```
+
+```bash
+# validate and start
+sudo systemctl daemon-reload
+sudo systemctl enable --now dnsmasq-ts2
+sudo systemctl status dnsmasq-ts2 --no-pager
+
+
+sudo ip netns exec ts2 ss -lnup | grep :53
+sudo ip netns exec ts2 dig @127.0.0.1 psmf.intranet +short
+sudo ip netns exec ts2 dig @127.0.0.1 cloudflare.com +short
+```
+
+
+## dnsmasq in root namespace
+
+```bash
+sudo mkdir -p /etc/dnsmasq-root
+sudo tee /etc/dnsmasq-root/dnsmasq.conf <<'EOF'
+# Root netns instance. Stock dnsmasq.service must be disabled/masked
+# so it doesn't fight us for port 53.
+# systemd-resolved stub listener is disabled (see top of this file).
+
+# Only listen on these — don't grab :53 on every interface.
+bind-interfaces
+interface=br0
+interface=tailscale0
+
+# Upstreams (explicit, no /etc/resolv.conf lookup)
+no-resolv
+no-hosts
+server=1.1.1.1
+server=9.9.9.9
+
+# Zone
+local=/intranet/
+domain=intranet
+
+address=/psmf.intranet/100.90.25.77
+address=/psmf-new.intranet/100.90.25.77
+
+cache-size=1000
+log-facility=/var/log/dnsmasq-root.log
+
+user=dnsmasq
+pid-file=/run/dnsmasq-root.pid
+EOF
+```
+
+```bash
+# systemd-unit
+sudo tee /etc/systemd/system/dnsmasq-root.service <<'EOF'
+[Unit]
+Description=dnsmasq in root netns (intranet zone)
+After=network-online.target
+Wants=network-online.target
+# Make sure the stock apt dnsmasq is out of the way:
+# sudo systemctl disable --now dnsmasq
+# sudo systemctl mask dnsmasq
+
+[Service]
+Type=simple
+ExecStartPre=/usr/sbin/dnsmasq --test -C /etc/dnsmasq-root/dnsmasq.conf
+ExecStart=/usr/sbin/dnsmasq -k -C /etc/dnsmasq-root/dnsmasq.conf
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+EOF
+```
+
+```bash
+# validate and start
+sudo systemctl daemon-reload
+sudo systemctl enable --now dnsmasq-root
+sudo systemctl status dnsmasq-root --no-pager
+
+# verify it's bound only on br0 + tailscale0 (not on lo, not 0.0.0.0)
+sudo ss -lnup | grep :53
+
+# smoke test against the host's own br0 / tailscale0 IP
+BR0_IP=$(ip -4 -o addr show br0        | awk '{print $4}' | cut -d/ -f1)
+TS0_IP=$(ip -4 -o addr show tailscale0  | awk '{print $4}' | cut -d/ -f1)
+dig @"$BR0_IP" psmf.intranet  +short
+dig @"$TS0_IP" psmf.intranet  +short
+dig @"$BR0_IP" cloudflare.com +short
+```
--- a/servers/psmf/files/nginx/default
+++ b/servers/psmf/files/nginx/default
@@ -0,0 +1,28 @@
+# Reverse proxy: horst-2 default site -> horst-1 Apache (10.0.0.5)
+# Deployed during the horst-1 -> horst-2 migration.
+# See servers/psmf/migration-plan.md.
+
+server {
+    listen 80 default_server;
+    listen [::]:80 default_server;
+
+    server_name _;
+
+    # Long bodies / uploads from the legacy PHP intranet app.
+    client_max_body_size 64m;
+
+    location / {
+        proxy_pass http://10.0.0.5;
+
+        proxy_http_version 1.1;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host  $host;
+
+        proxy_connect_timeout 10s;
+        proxy_read_timeout    300s;
+        proxy_send_timeout    300s;
+    }
+}
--- a/servers/psmf/files/nginx/psmf-new.intranet
+++ b/servers/psmf/files/nginx/psmf-new.intranet
@@ -0,0 +1,27 @@
+# Reverse proxy: psmf-new.intranet -> dockerized PHP intranet app on horst-2 (127.0.0.1:8080)
+# See servers/psmf/migration-plan.md and servers/psmf/dns-configuration.md.
+
+server {
+    listen 80;
+    listen [::]:80;
+
+    server_name psmf-new.intranet;
+
+    # Long bodies / uploads from the PHP intranet app.
+    client_max_body_size 64m;
+
+    location / {
+        proxy_pass http://127.0.0.1:8080;
+
+        proxy_http_version 1.1;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host  $host;
+
+        proxy_connect_timeout 10s;
+        proxy_read_timeout    300s;
+        proxy_send_timeout    300s;
+    }
+}
--- a/servers/psmf/files/nginx/psmf.intranet
+++ b/servers/psmf/files/nginx/psmf.intranet
@@ -0,0 +1,28 @@
+# Reverse proxy: psmf.intranet -> horst-1 Apache (10.0.0.5)
+# Same target as the default site, but bound to the psmf.intranet host header.
+# See servers/psmf/migration-plan.md and servers/psmf/dns-configuration.md.
+
+server {
+    listen 80;
+    listen [::]:80;
+
+    server_name psmf.intranet;
+
+    # Long bodies / uploads from the legacy PHP intranet app.
+    client_max_body_size 64m;
+
+    location / {
+        proxy_pass http://10.0.0.5;
+
+        proxy_http_version 1.1;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host  $host;
+
+        proxy_connect_timeout 10s;
+        proxy_read_timeout    300s;
+        proxy_send_timeout    300s;
+    }
+}
--- a/servers/psmf/files/srv/psmf/docker-compose.yml
+++ b/servers/psmf/files/srv/psmf/docker-compose.yml
@@ -0,0 +1,15 @@
+services:
+  app:
+    image: gitea.home.hrajfrisbee.cz/psmf/psmf-intranet:1.03
+    ports:
+      - "8080:8080"
+    environment:
+      DB_HOST: ${DB_HOST:-host.docker.internal}
+      DB_NAME: ${DB_NAME:-psmf}
+      DB_USER: ${DB_USER:-psmf}
+      DB_PASS: ${DB_PASS:-psmf}
+      # PSMF_BASE_URL: http://127.0.0.1:8080  # override for PDF export when behind a proxy
+      PSMF_HEADER_TEXT: "Vítejte na intranetu PSMF (PHP {PHP_VERSION})"
+      PSMF_HEADER_BG: "#a0b000"
+      PSMF_SHOW_PHP_VERSION: "1"
+    restart: unless-stopped
--- a/servers/psmf/files/tailscaled-ts2.service
+++ b/servers/psmf/files/tailscaled-ts2.service
@@ -0,0 +1,21 @@
+# location: /etc/systemd/system/tailscaled-ts2.service
+[Unit]
+Description=tailscaled in ts2 netns
+After=ts2-netns.service network-online.target
+Requires=ts2-netns.service
+BindsTo=ts2-netns.service
+
+[Service]
+ExecStart=/usr/sbin/ip netns exec ts2 /usr/sbin/tailscaled \
+  --state=/var/lib/tailscale-ts2/tailscaled.state \
+  --socket=/run/tailscale-ts2/tailscaled.sock \
+  --port=41642 \
+  --tun=tailscale0
+Restart=on-failure
+RestartSec=5s
+
+RuntimeDirectory=tailscale-ts2
+StateDirectory=tailscale-ts2
+
+[Install]
+WantedBy=multi-user.target
--- a/servers/psmf/files/ts2-netns-down
+++ b/servers/psmf/files/ts2-netns-down
@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# best-effort cleanup, don't fail on missing pieces
+# location: /usr/local/sbin/ts2-netns-down
+
+iptables -t nat -D POSTROUTING -s 10.200.0.0/30 -o br0 -j MASQUERADE 2>/dev/null || true
+iptables -D FORWARD -i veth-ts2 -o br0 -j ACCEPT 2>/dev/null || true
+iptables -D FORWARD -o veth-ts2 -i br0 -m state --state RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || true
+
+ip link del veth-ts2 2>/dev/null || true
+ip netns del ts2 2>/dev/null || true
+rm -rf /etc/netns/ts2
--- a/servers/psmf/files/ts2-netns-up
+++ b/servers/psmf/files/ts2-netns-up
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+# location: /usr/local/sbin/ts2-netns-up
+set -e
+
+# bail out clean if anything already exists from a previous run
+ip netns list | grep -qw ts2 || ip netns add ts2
+
+ip link show veth-ts2 &>/dev/null || \
+  ip link add veth-ts2 type veth peer name veth-ts2-ns
+
+# move peer into ns only if it's still on the host
+ip link show veth-ts2-ns &>/dev/null && \
+  ip link set veth-ts2-ns netns ts2
+
+ip addr add 10.200.0.1/30 dev veth-ts2 2>/dev/null || true
+ip link set veth-ts2 up
+
+ip -n ts2 addr add 10.200.0.2/30 dev veth-ts2-ns 2>/dev/null || true
+ip -n ts2 link set veth-ts2-ns up
+ip -n ts2 link set lo up
+ip -n ts2 route add default via 10.200.0.1 2>/dev/null || true
+
+mkdir -p /etc/netns/ts2
+echo "nameserver 1.1.1.1" > /etc/netns/ts2/resolv.conf
+
+sysctl -wq net.ipv4.ip_forward=1
+
+# -C checks if rule exists, -A appends only if missing
+iptables -t nat -C POSTROUTING -s 10.200.0.0/30 -o br0 -j MASQUERADE 2>/dev/null || \
+  iptables -t nat -A POSTROUTING -s 10.200.0.0/30 -o br0 -j MASQUERADE
+
+iptables -C FORWARD -i veth-ts2 -o br0 -j ACCEPT 2>/dev/null || \
+  iptables -A FORWARD -i veth-ts2 -o br0 -j ACCEPT
+
+iptables -C FORWARD -o veth-ts2 -i br0 -m state --state RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || \
+  iptables -A FORWARD -o veth-ts2 -i br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
+
+# iptables rules + sysctl inside ts2 ns
+# Forwarding inside ts2
+ip netns exec ts2 sysctl -wq net.ipv4.ip_forward=1
+
+# DNAT incoming ssh to host
+ip netns exec ts2 iptables -t nat -C PREROUTING -i tailscale0 -p tcp --dport 22 -j DNAT --to-destination 10.200.0.1:22 2>/dev/null || \
+  ip netns exec ts2 iptables -t nat -A PREROUTING -i tailscale0 -p tcp --dport 22 -j DNAT --to-destination 10.200.0.1:22
+
+# DNAT incoming http to host nginx
+ip netns exec ts2 iptables -t nat -C PREROUTING -i tailscale0 -p tcp --dport 80 -j DNAT --to-destination 10.200.0.1:80 2>/dev/null || \
+  ip netns exec ts2 iptables -t nat -A PREROUTING -i tailscale0 -p tcp --dport 80 -j DNAT --to-destination 10.200.0.1:80
+
+# DNAT incoming 8080 to docker web app in root ns
+ip netns exec ts2 iptables -t nat -C PREROUTING -i tailscale0 -p tcp --dport 8080 -j DNAT --to-destination 10.200.0.1:8080 2>/dev/null || \
+  ip netns exec ts2 iptables -t nat -A PREROUTING -i tailscale0 -p tcp --dport 8080 -j DNAT --to-destination 10.200.0.1:8080
+
+ip netns exec ts2 iptables -t nat -C POSTROUTING -o veth-ts2-ns -j MASQUERADE 2>/dev/null || \
+  ip netns exec ts2 iptables -t nat -A POSTROUTING -o veth-ts2-ns -j MASQUERADE
--- a/servers/psmf/files/ts2-netns.service
+++ b/servers/psmf/files/ts2-netns.service
@@ -0,0 +1,17 @@
+# /etc/systemd/system/ts2-netns.service
+
+[Unit]
+Description=ts2 netns + veth + NAT
+After=network-online.target
+Wants=network-online.target
+# tie tailscaled lifecycle to this
+Before=tailscaled-ts2.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/local/sbin/ts2-netns-up
+ExecStop=/usr/local/sbin/ts2-netns-down
+
+[Install]
+WantedBy=multi-user.target
--- a/servers/psmf/files/ts2-resolv.conf
+++ b/servers/psmf/files/ts2-resolv.conf
@@ -0,0 +1,2 @@
+# location: /etc/netns/ts2/resolv.conf
+nameserver 1.1.1.1
--- a/servers/psmf/horst-2/crontab
+++ b/servers/psmf/horst-2/crontab
@@ -0,0 +1,6 @@
+# backups
+15 11,15,19,22 * * * root /srv/bin/psmf-cli db backup --env-file ~/.psmf/horst-2.env --db psmf --dir /srv/backups/horst-2 -v >> /var/log/psmf-backup.log 2>&1
+17 11,15,19,22 * * * root /srv/bin/psmf-cli db backup --env-file ~/.psmf/horst.env --db psmf --dir /srv/backups/horst -v >> /var/log/psmf-backup.log 2>&1
+# prune backups
+20 11,15,19,22 * * * root /srv/bin/psmf-cli db prune-backups --db psmf --dir /srv/backups/horst-2 --keep-within 2w --keep-weekly 8 --keep-monthly 120 >> /var/log/psmf-backup.log 2>&1
+21 11,15,19,22 * * * root /srv/bin/psmf-cli db prune-backups --db psmf --dir /srv/backups/horst --keep-within 2w --keep-weekly 8 --keep-monthly 120 >> /var/log/psmf-backup.log 2>&1
--- a/servers/psmf/migration-plan.md
+++ b/servers/psmf/migration-plan.md
@@ -0,0 +1,93 @@
+# Migration from horst-1 to horst-2
+
+## postgres database
+
+```bash
+# [DONE] install postgres
+apt update
+apt install -y curl ca-certificates gnupg lsb-release
+
+# PGDG signing key (keyring-style, not the deprecated apt-key)
+sudo install -d /usr/share/postgresql-common/pgdg
+sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
+  -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc
+
+# repo
+echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] \
+https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \
+| sudo tee /etc/apt/sources.list.d/pgdg.list
+
+sudo apt update
+sudo apt install -y postgresql-18 postgresql-contrib-18
+
+# backup database from horst-1
+# users backup
+pg_dumpall --globals-only -h 10.0.0.5 -U postgres > /tmp/pg-dumpall-globals
+# data backup
+pg_dump -h 10.0.0.5 -U postgres -C psmf > /tmp/psmf.pgsql
+
+# restore backup of database from horst-1
+# create users 
+cat /tmp/pg-dumpall-globals | psql
+# restore psmf database
+cat /tmp/psmf.pgsql | psql
+
+# install docker oin horst-2
+# Docker's GPG key (keyring, not apt-key)
+sudo install -m 0755 -d /etc/apt/keyrings
+sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
+  -o /etc/apt/keyrings/docker.asc
+sudo chmod a+r /etc/apt/keyrings/docker.asc
+
+# repo
+echo \
+"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
+https://download.docker.com/linux/ubuntu \
+$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
+  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+
+sudo apt-get update
+sudo apt-get install -y \
+  docker-ce docker-ce-cli containerd.io \
+  docker-buildx-plugin docker-compose-plugin
+
+
+# [DONE] dockerize php intranet app
+# dockerized + upgraded to PHP 8.3.31
+
+# [DONE] start psmf app on horst-2 and point it to database on horst
+
+# [TODO] point php app on horst-1 to horst-2
+
+# validate application is still working
+# [OK] - wkhtml - PDF printing of html pages
+# prepare local backups
+# validate if pg sync to api still works from horst-1
+# set sync to api on horst-2
+# install nginx and configure default site as reverse proxy to horst-1
+sudo apt-get update
+sudo apt-get install -y nginx
+
+# drop the proxy config in place (file lives in this repo at servers/psmf/files/nginx/default)
+sudo install -m 0644 -o root -g root \
+    servers/psmf/files/nginx/default /etc/nginx/sites-available/default
+
+# sites-enabled/default is already a symlink to ../sites-available/default on a stock Ubuntu install
+sudo nginx -t
+sudo systemctl reload nginx
+sudo systemctl enable nginx
+
+# intranet vhosts: psmf.intranet -> horst-1, psmf-new.intranet -> local docker app
+# (depends on the dnsmasq setup in servers/psmf/dns-configuration.md)
+sudo install -m 0644 -o root -g root \
+    servers/psmf/files/nginx/psmf.intranet     /etc/nginx/sites-available/psmf.intranet
+sudo install -m 0644 -o root -g root \
+    servers/psmf/files/nginx/psmf-new.intranet /etc/nginx/sites-available/psmf-new.intranet
+
+sudo ln -sf ../sites-available/psmf.intranet     /etc/nginx/sites-enabled/psmf.intranet
+sudo ln -sf ../sites-available/psmf-new.intranet /etc/nginx/sites-enabled/psmf-new.intranet
+
+sudo nginx -t
+sudo systemctl reload nginx
+
+```
--- a/servers/psmf/readme.md
+++ b/servers/psmf/readme.md
@@ -0,0 +1,33 @@
+# Servers
+
+### Horst
+
+- ip: 10.0.0.5
+- os: CentOS release 6.9 (Final)
+- services:
+    - apache tcp/80
+    - postgres tcp/5432
+    - samba (not really in use)
+    - cronjobs
+        - /srv/bin/pg_psmf_dump.sh
+        - /srv/bin/psmf-prepare-tables.sh 2023 2027
+        - /srv/bin/psmf-api-sync.sh
+
+- specials:
+    -   some binaries/libraries relevant to PDF printing from the app WKHTML        
+
+```bash
+# crontab content
+00 11,15,19,22 * * * root /srv/bin/pg_psmf_dump.sh
+55 10,14,18,21 * * * root /srv/bin/psmf-prepare-tables.sh 2023 2027
+05 11,15,19,22 * * * root /srv/bin/psmf-api-sync.sh
+```
+
+#### Problems
+
+- operating system is already that old, that there is not reasonable and safe upgrade path to something current
+
+
+### Horst-2
+
+- ip: 10.0.0.6
--- a/servers/psmf/tailscale-second-instance.md
+++ b/servers/psmf/tailscale-second-instance.md
@@ -0,0 +1,39 @@
+```bash
+# Create namespace
+sudo ip netns add ts2
+
+# veth pair: host side <-> ns side
+sudo ip link add veth-ts2 type veth peer name veth-ts2-ns
+sudo ip link set veth-ts2-ns netns ts2
+
+# Host side
+sudo ip addr add 10.200.0.1/30 dev veth-ts2
+sudo ip link set veth-ts2 up
+
+# Namespace side
+sudo ip -n ts2 addr add 10.200.0.2/30 dev veth-ts2-ns
+sudo ip -n ts2 link set veth-ts2-ns up
+sudo ip -n ts2 link set lo up
+sudo ip -n ts2 route add default via 10.200.0.1
+
+# DNS inside the netns
+sudo mkdir -p /etc/netns/ts2
+echo "nameserver 1.1.1.1" | sudo tee /etc/netns/ts2/resolv.conf
+
+# NAT from the netns out to the world (replace eth0 with your egress iface)
+sudo sysctl -w net.ipv4.ip_forward=1
+sudo iptables -t nat -A POSTROUTING -s 10.200.0.0/30 -o br0 -j MASQUERADE
+sudo iptables -A FORWARD -i veth-ts2 -o br0 -j ACCEPT
+sudo iptables -A FORWARD -o veth-ts2 -i br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
+
+
+# quick sanity check
+sudo ip netns exec ts2 curl https://controlplane.tailscale.com
+# expect: "OK"
+```
+
+```bash
+# ts2 alias
+echo 'alias ts2="sudo ip netns exec ts2 tailscale --socket=/run/tailscale-ts2/tailscaled.sock"' \
+  | sudo tee /etc/profile.d/tailscale-ts2.sh
+```
--- a/servers/saint/etc/ifcfg-eno0
+++ b/servers/saint/etc/ifcfg-eno0
@@ -0,0 +1,22 @@
+TYPE="Ethernet"
+BOOTPROTO="none"
+DEFROUTE="yes"
+IPV4_FAILURE_FATAL="no"
+IPV6INIT="yes"
+IPV6_AUTOCONF="yes"
+IPV6_DEFROUTE="yes"
+IPV6_FAILURE_FATAL="no"
+NAME="eno1"
+UUID="2f881122-85e5-4078-8cec-cf4a7e465321"
+DEVICE="eno1"
+ONBOOT="yes"
+IPADDR="87.236.197.83"
+PREFIX="23"
+GATEWAY="87.236.196.1"
+DNS1="87.236.198.210"
+#DNS1="87.236.198.211"
+DNS2="89.187.142.8"
+IPV6_PEERDNS="yes"
+IPV6_PEERROUTES="yes"
+IPV6_PRIVACY="no"
+ZONE=public
--- a/servers/shadow/iptables/rules.v4
+++ b/servers/shadow/iptables/rules.v4
@@ -49,6 +49,7 @@ COMMIT
 -A FORWARD -i eno1 -p udp -m udp --dport 5353 -j ACCEPT
 -A FORWARD -i eno1 -p udp -m udp --dport 51820 -j ACCEPT
 -A FORWARD -i eno1 -p udp -m udp --dport 1194 -j ACCEPT
+-A FORWARD -i eno1 -p tcp -m tcp --dport 33022 -j ACCEPT
 -A FORWARD -j DOCKER-USER
 -A FORWARD -j DOCKER-ISOLATION-STAGE-1
 -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
@@ -92,12 +93,14 @@ COMMIT
 -A LIBVIRT_INP -p tcp -m tcp --dport 53 -j ACCEPT
 -A LIBVIRT_INP -p udp -m udp --dport 5353 -j ACCEPT
 -A LIBVIRT_INP -p tcp -m tcp --dport 5353 -j ACCEPT
+-A LIBVIRT_INP -p tcp -m tcp --dport 33022 -j ACCEPT
 -A LIBVIRT_INP -p udp -m udp --dport 67 -j ACCEPT
 -A LIBVIRT_INP -p tcp -m tcp --dport 67 -j ACCEPT
 -A LIBVIRT_OUT -p udp -m udp --dport 53 -j ACCEPT
 -A LIBVIRT_OUT -p tcp -m tcp --dport 53 -j ACCEPT
 -A LIBVIRT_OUT -p udp -m udp --dport 5353 -j ACCEPT
 -A LIBVIRT_OUT -p tcp -m tcp --dport 5353 -j ACCEPT
+-A LIBVIRT_OUT -p tcp -m tcp --dport 33022 -j ACCEPT
 -A LIBVIRT_OUT -p udp -m udp --dport 68 -j ACCEPT
 -A LIBVIRT_OUT -p tcp -m tcp --dport 68 -j ACCEPT
 -A f2b-sshd -j RETURN
@@ -118,6 +121,7 @@ COMMIT
 -A PREROUTING -i eno1 -p udp -m udp --dport 51820 -j DNAT --to-destination 192.168.123.101:51820
 -A PREROUTING -i eno1 -p udp -m udp --dport 1194 -j DNAT --to-destination 192.168.123.101:1194
 -A PREROUTING -i eno1 -p tcp -m tcp --dport 21080 -j DNAT --to-destination 192.168.123.141:80
+-A PREROUTING -i eno1 -p tcp -m tcp --dport 33022 -j DNAT --to-destination 192.168.123.21:33022
 -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
 -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
 -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
--- a/servers/shadow/nginx/21_docker_hrajfrisbee.cz.conf
+++ b/servers/shadow/nginx/21_docker_hrajfrisbee.cz.conf
@@ -36,12 +36,6 @@ server {
 # ----------------
 # api-sync.psmf.hrajfrisbee.cz
 server {
-  #  if ($host = api-sync.psmf.hrajfrisbee.cz) {
-  #      return 301 https://$host$request_uri;
-  #  } # managed by Certbot
-
-
-  listen 80;
  client_max_body_size 15m;                                                                                                                                                              proxy_read_timeout 600;
  server_name
        api-sync.psmf.hrajfrisbee.cz
@@ -49,19 +43,12 @@ server {
  location / {
    proxy_pass http://192.168.123.21:8003;
  }
-}
-server {
-    if ($host = api.psmf.hrajfrisbee.cz) {
-        return 301 https://$host$request_uri;
-    } # managed by Certbot
-
-
-  server_name
-        api.psmf.hrajfrisbee.cz
-        ;
-    listen 80;
-    return 404; # managed by Certbot

+    listen 8443 ssl; # managed by Certbot
+    ssl_certificate /etc/letsencrypt/live/api-sync.psmf.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
+    ssl_certificate_key /etc/letsencrypt/live/api-sync.psmf.hrajfrisbee.cz/privkey.pem; # managed by Certbot
+    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

 }

@@ -75,6 +62,10 @@ server {
                gitlab.hrajfrisbee.cz
        ;

+        # let certbot http-01 land
+        location /.well-known/acme-challenge/ { root /var/www/html; }
+
+
          location / {
            proxy_pass http://192.168.123.21:80;
            proxy_set_header        X-Real-IP       $remote_addr;
@@ -92,16 +83,17 @@ server {

          }

-    #listen 443 ssl; # managed by Certbot
-    #ssl_certificate /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
-    #ssl_certificate_key /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
-    #include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
-    #ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
+    listen 8443 ssl; # managed by Certbot
+    ssl_certificate /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
+    ssl_certificate_key /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
+    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    } # managed by Certbot
+
 }

 server {
@@ -123,14 +115,79 @@ server {
          }


-    #listen 443 ssl; # managed by Certbot
-    #ssl_certificate /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
-    #ssl_certificate_key /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
-    #include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
-    #ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
+    listen 8443 ssl; # managed by Certbot
+    ssl_certificate /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
+    ssl_certificate_key /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
+    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    } # managed by Certbot
 }
+
+
+server {
+    if ($host = api-sync.psmf.hrajfrisbee.cz) {
+        return 301 https://$host$request_uri;
+    } # managed by Certbot
+
+
+  listen 80;
+  server_name
+        api-sync.psmf.hrajfrisbee.cz
+        ;
+    return 404; # managed by Certbot
+
+
+}
+
+
+server {
+    if ($host = registry.gitlab.hrajfrisbee.cz) {
+        return 301 https://$host$request_uri;
+    } # managed by Certbot
+
+
+        server_name
+                registry.gitlab.hrajfrisbee.cz
+        ;
+    listen 80;
+    return 404; # managed by Certbot
+}
+
+
+server {
+    if ($host = gitlab.hrajfrisbee.cz) {
+        return 301 https://$host$request_uri;
+    } # managed by Certbot
+
+
+        server_name
+                gitlab.hrajfrisbee.cz
+        ;
+    listen 80;
+    return 404; # managed by Certbot
+}
+
+# ----------------
+# legacy sites
+# ----------------
+
+server {
+        server_name
+                vegtral.cz www.vegtral.cz
+        ;
+
+        location / {
+                proxy_pass       http://192.168.123.21:8080;
+                proxy_set_header Host            $host;
+                proxy_set_header X-Forwarded-For $remote_addr;
+                proxy_set_header X-Forwarded-Proto https;
+                proxy_set_header X-Forwarded-Ssl on;
+
+                client_max_body_size 3000M;
+                client_body_buffer_size 200000k;
+          }
+}
--- a/servers/shadow/nginx/22_docker-dev_hrajfrisbee.cz.conf
+++ b/servers/shadow/nginx/22_docker-dev_hrajfrisbee.cz.conf
@@ -0,0 +1,35 @@
+# ----------------
+# api-sync-dev.psmf.hrajfrisbee.cz
+server {
+  client_max_body_size 15m;                                                                                                                                                              proxy_read_timeout 600;
+  server_name
+        api-sync-dev.psmf.hrajfrisbee.cz
+        ;
+  location / {
+    proxy_pass http://192.168.123.22:8003;
+  }
+
+    listen 8443 ssl; # managed by Certbot
+    ssl_certificate /etc/letsencrypt/live/api-sync-dev.psmf.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
+    ssl_certificate_key /etc/letsencrypt/live/api-sync-dev.psmf.hrajfrisbee.cz/privkey.pem; # managed by Certbot
+    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
+
+}
+
+
+
+server {
+    if ($host = api-sync-dev.psmf.hrajfrisbee.cz) {
+        return 301 https://$host$request_uri;
+    } # managed by Certbot
+
+
+  listen 80;
+  server_name
+        api-sync-dev.psmf.hrajfrisbee.cz
+        ;
+    return 404; # managed by Certbot
+
+
+}
--- a/servers/shadow/nginx/nginx-sites-enabled-default
+++ b/servers/shadow/nginx/nginx-sites-enabled-default
@@ -389,3 +389,33 @@ server {
    server_name fuj-management.home.hrajfrisbee.cz;
    return 404; # managed by Certbot
 }
+
+server {
+    server_name czechultimate.cz;
+
+    # let certbot http-01 land
+    location /.well-known/acme-challenge/ { root /var/www/html; }
+
+    location / { return 301 https://www.czechultimate.cz$request_uri; }
+
+    listen 8443 ssl; # managed by Certbot
+    ssl_certificate /etc/letsencrypt/live/czechultimate.cz/fullchain.pem; # managed by Certbot
+    ssl_certificate_key /etc/letsencrypt/live/czechultimate.cz/privkey.pem; # managed by Certbot
+    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
+
+}
+
+
+server {
+    if ($host = czechultimate.cz) {
+        return 301 https://$host$request_uri;
+    } # managed by Certbot
+
+
+    listen 80;
+    server_name czechultimate.cz;
+    return 404; # managed by Certbot
+
+
+}
--- a/servers/shadow/nginx/nginx.conf
+++ b/servers/shadow/nginx/nginx.conf
@@ -66,6 +66,7 @@ stream {
        # Passthrough to K8s
 		ghost.lab.home.hrajfrisbee.cz   k8s_gatewayapi;
 		fujarna.lab.home.hrajfrisbee.cz   k8s_gatewayapi;
+		fuj-management.lab.home.hrajfrisbee.cz   k8s_gatewayapi;
 		operator-test.lab.home.hrajfrisbee.cz   k8s_gatewayapi;

        ~^.+\.lab\.home\.hrajfrisbee\.cz$    k8s_ingress;
--- a/vms-home/docker-29/readme.md
+++ b/vms-home/docker-29/readme.md
--- a/vms-home/docker-29/tracing/docker-compose.yaml
+++ b/vms-home/docker-29/tracing/docker-compose.yaml
@@ -0,0 +1,14 @@
+services:
+  jaeger:
+    image: jaegertracing/jaeger:2.18.0
+    container_name: jaeger
+    restart: unless-stopped
+    user: root
+    command: ["--config", "/jaeger/config.yaml"]
+    volumes:
+      - ./jaeger-config.yaml:/jaeger/config.yaml:ro
+      - ./badger:/badger
+    ports:
+      - "4317:4317"    # OTLP gRPC
+      - "4318:4318"    # OTLP HTTP
+      - "16686:16686"  # Jaeger UI
--- a/vms-home/docker-29/tracing/jaeger-config.yaml
+++ b/vms-home/docker-29/tracing/jaeger-config.yaml
@@ -0,0 +1,35 @@
+service:
+  extensions: [jaeger_storage, jaeger_query]
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: []
+      exporters: [jaeger_storage_exporter]
+
+extensions:
+  jaeger_storage:
+    backends:
+      badger_store:
+        badger:
+          directories:
+            keys: /badger/key
+            values: /badger/data
+          ephemeral: false
+          ttl:
+            spans: 4h
+          metrics_update_interval: 5m
+  jaeger_query:
+    storage:
+      traces: badger_store
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+
+exporters:
+  jaeger_storage_exporter:
+    trace_storage: badger_store
--- a/vms-home/docker-30/frisbee-drills/run-presejpacky.sh
+++ b/vms-home/docker-30/frisbee-drills/run-presejpacky.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+docker rm -f drills-presejpacky
+
+# gitea registry login with kacerr / token
+docker run -d --name drills-presejpacky \
+  --restart=always \
+  -p 3101:3000 \
+  gitea.home.hrajfrisbee.cz/kacerr/flat-stack-presejpacky:presejpacky-v1
--- a/vms-home/docker-30/fuj-management/run.sh
+++ b/vms-home/docker-30/fuj-management/run.sh
--- a/vms-home/docker-30/garage/docker-compose.yml
+++ b/vms-home/docker-30/garage/docker-compose.yml
@@ -0,0 +1,42 @@
+services:
+  garage:
+    image: dxflrs/garage:v2.3.0
+    container_name: garage
+    restart: unless-stopped
+    networks: [garage]
+    ports:
+      - "3900:3900"   # S3 API  (LAN-reachable for your other services)
+      - "3902:3902"   # S3 web
+      # - "3901:3901" # RPC — only publish once you go multi-node
+    volumes:
+      - /srv/docker/garage/garage.toml:/etc/garage.toml:ro
+      - /srv/docker/garage/meta:/var/lib/garage/meta
+      - /srv/docker/garage/data:/var/lib/garage/data
+      - /srv/docker/garage/snapshots:/var/lib/garage/snapshots
+    healthcheck:
+      test: ["CMD", "/garage", "status"]   # image is distroless; use the binary, not curl
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 20s
+
+  garage-webui:
+    image: khairul169/garage-webui:1.1.0
+    container_name: garage-webui
+    restart: unless-stopped
+    networks: [garage]
+    depends_on:
+      garage:
+        condition: service_healthy
+    ports:
+      # - "127.0.0.1:3909:3909"   # bind localhost only — reverse-proxy in front for TLS/auth
+      - "3909:3909"
+    volumes:
+      - /srv/docker/garage/garage.toml:/etc/garage.toml:ro
+    environment:
+      API_BASE_URL:    "http://garage:3903"   # override self-pointing default
+      S3_ENDPOINT_URL: "http://garage:3900"
+
+networks:
+  garage:
+    driver: bridge
--- a/vms-home/docker-30/garage/garage.toml
+++ b/vms-home/docker-30/garage/garage.toml
@@ -0,0 +1,28 @@
+metadata_dir           = "/var/lib/garage/meta"
+data_dir               = "/var/lib/garage/data"
+metadata_snapshots_dir = "/var/lib/garage/snapshots"
+db_engine              = "lmdb"
+
+metadata_fsync                  = true
+metadata_auto_snapshot_interval = "6h"    # LMDB corruption insurance
+compression_level               = 2
+replication_factor              = 1        # single node; bump when you add nodes
+
+rpc_bind_addr   = "[::]:3901"
+rpc_public_addr = "127.0.0.1:3901"         # self-RPC only on single node
+rpc_secret      = "REPLACE_rpc_secret"     # openssl rand -hex 32
+
+[s3_api]
+s3_region     = "garage"
+api_bind_addr = "[::]:3900"
+root_domain   = ".s3.garage"
+
+[s3_web]
+bind_addr   = "[::]:3902"
+root_domain = ".web.garage"
+index       = "index.html"
+
+[admin]
+api_bind_addr = "[::]:3903"
+admin_token   = "REPLACE_admin_token"      # webui reads this for auth; openssl rand -hex 32
+metrics_token = "REPLACE_metrics_token"    # openssl rand -hex 32
--- a/vms-home/docker-30/garage/readme.md
+++ b/vms-home/docker-30/garage/readme.md
--- a/vms-home/docker-30/gitea/docker-compose.yaml
+++ b/vms-home/docker-30/gitea/docker-compose.yaml
--- a/vms-home/docker-30/gitea/readme.md
+++ b/vms-home/docker-30/gitea/readme.md
--- a/vms-home/docker-30/gitea/runner-config.yaml
+++ b/vms-home/docker-30/gitea/runner-config.yaml
@@ -0,0 +1,15 @@
+runner:
+  file: /data/.runner
+  fetch_interval: 10s
+  fetch_timeout: 30s
+  capacity: 4   # number of concurrent jobs this runner will accept
+
+cache:
+  enabled: true
+  dir: /data/cache
+  host: gitea-runner
+  port: 8088
+
+container:
+  network: gitea-production_gitea-network
+  force_pull: true
--- a/vms-home/docker-30/kanidm/gitea-action-kubernetes-access.md
+++ b/vms-home/docker-30/kanidm/gitea-action-kubernetes-access.md
--- a/vms-home/docker-30/kanidm/readme.md
+++ b/vms-home/docker-30/kanidm/readme.md
--- a/vms-home/docker-30/kanidm/run.sh
+++ b/vms-home/docker-30/kanidm/run.sh
--- a/vms-home/docker-30/kanidm/server.toml
+++ b/vms-home/docker-30/kanidm/server.toml
--- a/vms-home/docker-30/lab-proxy/nginx.conf
+++ b/vms-home/docker-30/lab-proxy/nginx.conf
--- a/vms-home/docker-30/lab-proxy/run.sh
+++ b/vms-home/docker-30/lab-proxy/run.sh
--- a/vms-home/docker-30/loki/.env
+++ b/vms-home/docker-30/loki/.env
@@ -0,0 +1,2 @@
+GARAGE_ACCESS_KEY=GK...
+GARAGE_SECRET_KEY=...
--- a/vms-home/docker-30/loki/docker-compose.yaml
+++ b/vms-home/docker-30/loki/docker-compose.yaml
@@ -0,0 +1,41 @@
+services:
+  loki:
+    image: grafana/loki:3.7.3
+    container_name: loki
+    restart: unless-stopped
+    user: "10001:10001"
+    command:
+      - -config.file=/etc/loki/config.yaml
+      - -config.expand-env=true                # REQUIRED — expands ${GARAGE_*}
+    ports:
+      - "3100:3100"                            # LAN IP, not 0.0.0.0 — push+query
+    volumes:
+      - ./loki-config.yaml:/etc/loki/config.yaml:ro
+      - ./loki-data:/loki                       # WAL / tsdb-index / compactor scratch
+    env_file: [.env]                            # GARAGE_ACCESS_KEY / GARAGE_SECRET_KEY
+    networks: [obs]
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://localhost:3100/ready || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 10
+      start_period: 30s
+    logging:                                    # don't let Loki's own logs balloon
+      driver: json-file
+      options: { max-size: "10m", max-file: "3" }
+
+  grafana:                                      # optional — drop if you query elsewhere
+    image: grafana/grafana:13.1.0
+    container_name: grafana
+    restart: unless-stopped
+    ports:
+      - "3001:3000" # LAN IP, not               # 3000 is taken by gitea
+    volumes:
+      - grafana-data:/var/lib/grafana
+    networks: [obs]
+
+volumes:
+  grafana-data:
+
+networks:
+  obs: { driver: bridge }
--- a/vms-home/docker-30/loki/loki-config.yaml
+++ b/vms-home/docker-30/loki/loki-config.yaml
@@ -0,0 +1,47 @@
+auth_enabled: false                        # single-tenant; sources split by label
+
+server:
+  http_listen_port: 3100
+  grpc_listen_port: 9096
+  log_level: info
+
+common:
+  path_prefix: /loki
+  replication_factor: 1
+  ring:
+    kvstore: { store: inmemory }
+  storage:
+    s3:
+      endpoint: 192.168.0.30:3900          # no scheme; insecure toggles http
+      region: garage                       # MUST byte-match s3_region in garage.toml
+      bucketnames: loki-chunks
+      access_key_id: ${GARAGE_ACCESS_KEY}
+      secret_access_key: ${GARAGE_SECRET_KEY}
+      s3forcepathstyle: true               # mandatory for Garage
+      insecure: true                       # http on LAN; drop if you TLS-front Garage
+
+schema_config:
+  configs:
+    - from: 2026-07-01                     # <= first-ingest date; never mutate historical
+      store: tsdb
+      object_store: s3
+      schema: v13
+      index: { prefix: index_, period: 24h }
+
+storage_config:
+  tsdb_shipper:
+    active_index_directory: /loki/tsdb-index
+    cache_location: /loki/tsdb-cache
+
+limits_config:
+  allow_structured_metadata: true
+  volume_enabled: true
+  retention_period: 744h                   # 31d; compactor enforces
+  reject_old_samples: true
+  reject_old_samples_max_age: 168h
+  max_line_size: 256KB
+  ingestion_rate_mb: 8                      # per-tenant; bump if you hit 429s
+  ingestion_burst_size_mb: 16
+
+compactor:
+  working_directory: /loki/compactor
--- a/vms-home/docker-30/maru-hleda-byt/run.sh
+++ b/vms-home/docker-30/maru-hleda-byt/run.sh
--- a/vms-home/docker-30/mimir/.env
+++ b/vms-home/docker-30/mimir/.env
@@ -0,0 +1,2 @@
+GARAGE_ACCESS_KEY=GK...
+GARAGE_SECRET_KEY=...
--- a/vms-home/docker-30/mimir/docker-compose.yaml
+++ b/vms-home/docker-30/mimir/docker-compose.yaml
@@ -0,0 +1,28 @@
+services:
+  mimir:
+    image: grafana/mimir:3.1.2
+    container_name: mimir
+    restart: unless-stopped
+    command:
+      - -config.file=/etc/mimir/config.yaml
+      - -config.expand-env=true                # REQUIRED — expands ${GARAGE_*}
+      - -target=all                             # monolithic single-binary mode
+    ports:
+      - "9009:9009"                            # LAN IP, not 0.0.0.0 — push+query
+    volumes:
+      - ./mimir-config.yaml:/etc/mimir/config.yaml:ro
+      - ./mimir-data:/data                      # WAL / tsdb / compactor scratch
+    env_file: [.env]                            # GARAGE_ACCESS_KEY / GARAGE_SECRET_KEY
+    networks: [obs]
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://localhost:9009/ready || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 10
+      start_period: 30s
+    logging:                                    # don't let Mimir's own logs balloon
+      driver: json-file
+      options: { max-size: "10m", max-file: "3" }
+
+networks:
+  obs: { driver: bridge }
--- a/vms-home/docker-30/mimir/mimir-config.yaml
+++ b/vms-home/docker-30/mimir/mimir-config.yaml
@@ -0,0 +1,42 @@
+multitenancy_enabled: false            # single-tenant; mirrors Loki's auth_enabled:false
+
+server:
+  http_listen_port: 9009
+  grpc_listen_port: 9095
+  log_level: info
+
+common:
+  storage:
+    backend: s3
+    s3:
+      endpoint: 192.168.0.30:3900      # no scheme; insecure toggles http
+      region: garage                   # MUST byte-match s3_region in garage.toml
+      access_key_id: ${GARAGE_ACCESS_KEY}
+      secret_access_key: ${GARAGE_SECRET_KEY}
+      insecure: true                   # http on LAN; drop if you TLS-front Garage
+      bucket_lookup_type: path         # path-style — Garage requirement (== s3forcepathstyle)
+
+blocks_storage:
+  s3: { bucket_name: mimir-blocks }
+  tsdb: { dir: /data/tsdb }
+  bucket_store: { sync_dir: /data/tsdb-sync }
+
+ruler_storage:
+  s3: { bucket_name: mimir-ruler }
+
+alertmanager_storage:
+  s3: { bucket_name: mimir-alertmanager }
+
+compactor:
+  data_dir: /data/compactor
+
+ruler:
+  rule_path: /data/ruler
+
+alertmanager:
+  data_dir: /data/alertmanager
+
+limits:
+  compactor_blocks_retention_period: 744h   # 31d; matches Loki's retention_period
+  ingestion_rate: 50000                      # samples/s per tenant; bump if you hit 429s
+  ingestion_burst_size: 100000
--- a/vms-home/docker-30/mimir/readme.md
+++ b/vms-home/docker-30/mimir/readme.md
@@ -0,0 +1,98 @@
+# Mimir
+
+Grafana Mimir, monolithic mode (`-target=all`), single-tenant, backed by Garage S3
+(same bucket-per-Garage-instance pattern as `../loki/`). Metrics ingest via
+Prometheus remote_write — Mimir does not scrape anything itself.
+
+- HTTP (query + push): `9009`
+- gRPC (internal): `9095`
+
+## 1. Garage setup (do this first — before `docker compose up`)
+
+Mimir needs three buckets. Create them and grant the existing Loki Garage key
+read+write access (reuse the key rather than minting a new one, unless you
+want tighter separation):
+
+```bash
+ssh novakj@192.168.0.30
+
+for b in mimir-blocks mimir-ruler mimir-alertmanager; do
+  docker exec garage /garage bucket create "$b"
+done
+
+docker exec garage /garage key list           # find the key name Loki already uses
+
+for b in mimir-blocks mimir-ruler mimir-alertmanager; do
+  docker exec garage /garage bucket allow --read --write "$b" --key <keyname>
+done
+```
+
+## 2. `.env`
+
+Reuse the same Garage credentials Loki uses (see `../loki/.env`):
+
+```
+GARAGE_ACCESS_KEY=GK...
+GARAGE_SECRET_KEY=...
+```
+
+## 3. Start
+
+```bash
+cd /path/to/mimir
+docker compose up -d
+```
+
+## 4. Add Mimir as a Grafana datasource
+
+Reuses the Grafana already running from `../loki/docker-compose.yaml` (`:3001`)
+— no second Grafana instance.
+
+- Type: **Prometheus**
+- URL: `http://192.168.0.30:9009/prometheus`
+- No `X-Scope-OrgID` header needed (`multitenancy_enabled: false`)
+
+## 5. Verify
+
+```bash
+curl http://192.168.0.30:9009/ready                # -> ready
+docker compose ps                                   # mimir healthy
+docker compose logs -f mimir                         # no S3 auth/bucket errors
+
+# push endpoint reachable (expect a 4xx proto-decode error, not connection refused)
+curl http://192.168.0.30:9009/api/v1/push
+```
+
+In Grafana → Explore → Mimir datasource, run `up` or `count({__name__!=""})`.
+Empty result is expected until a producer remote_writes into Mimir.
+
+After ~2h (first block flush interval), confirm blocks landed in Garage:
+
+```bash
+docker exec garage /garage bucket info mimir-blocks
+```
+
+## 6. Wiring a producer (later, not done here)
+
+Point any Prometheus/Alloy remote_write at Mimir:
+
+```yaml
+remote_write:
+  - url: http://192.168.0.30:9009/api/v1/push
+```
+
+`vms/utility-101-shadow/docker/monitoring/prometheus.yml` is the natural first
+candidate to wire up.
+
+## Notes / gotchas
+
+- `bucket_lookup_type: path` in `mimir-config.yaml` is mandatory for Garage
+  (equivalent to Loki's `s3forcepathstyle: true`).
+- `-config.expand-env=true` is required on the command line, or `${GARAGE_*}`
+  in `mimir-config.yaml` won't expand.
+- First metrics won't appear in Garage until the first block flush (~2h) —
+  don't panic if `mimir-blocks` looks empty right after startup.
+- Retention is 31d, set via `limits.compactor_blocks_retention_period` (matches
+  Loki's `retention_period`).
+- `region: garage` in the config must byte-match `s3_region` in Garage's
+  `garage.toml` — same requirement as Loki.
--- a/vms-home/docker-30/nginx/001-gitea.conf
+++ b/vms-home/docker-30/nginx/001-gitea.conf
--- a/vms-home/docker-30/nginx/002-jellyfin.conf
+++ b/vms-home/docker-30/nginx/002-jellyfin.conf
--- a/vms-home/docker-30/nginx/021-drills.conf
+++ b/vms-home/docker-30/nginx/021-drills.conf
@@ -0,0 +1,20 @@
+server {
+    listen 443 ssl http2;
+    server_name drills.home.hrajfrisbee.cz;
+
+    ssl_certificate /etc/letsencrypt/live/drills.home.hrajfrisbee.cz/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/drills.home.hrajfrisbee.cz/privkey.pem;
+
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
+    ssl_prefer_server_ciphers off;
+
+    location /presejpacky {
+        proxy_pass http://192.168.0.30:3101;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+
+}
--- a/vms-home/docker-30/readme.md
+++ b/vms-home/docker-30/readme.md
@@ -0,0 +1,43 @@
+# docker-30
+
+## taiscale
+
+```bash
+# Add signing key
+curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/$(lsb_release -cs).noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
+
+# Add repo
+echo "deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/tailscale.list
+
+# Install
+sudo apt update && sudo apt install tailscale
+
+# Start
+sudo tailscale up
+```
+
+
+## connect nvmeof volume
+
+```bash
+nvme discover -t tcp -a 192.168.0.40 -s 4420
+
+nvme connect -t tcp -a 192.168.0.40  -s 4420 -n nqn.2011-06.com.truenas:uuid:6ac818fb-9f51-4e28-a5d6-41502eba031d:garage-s3
+
+nvme list
+
+mkfs.xfs -L garage /dev/nvme0n1
+mkdir -p /srv/garage
+mount /dev/disk/by-label/garage /srv/garage
+mkdir -p /srv/garage/{meta,data,snapshots}
+
+# add to fstab
+# LABEL=garage  /srv/garage  xfs  defaults,_netdev,nofail,x-systemd.requires=nvmf-autoconnect.service  0 0
+
+# /etc/nvme/discovery.conf
+-t tcp -a <TRUENAS_IP> -s 4420 -n nqn.2011-06.com.truenas:garage-s3
+systemctl enable nvmf-autoconnect.service
+
+
+
+```
--- a/vms-home/docker-30/vault/backup.md
+++ b/vms-home/docker-30/vault/backup.md
--- a/vms-home/docker-30/vault/config.json
+++ b/vms-home/docker-30/vault/config.json
--- a/vms-home/docker-30/vault/docker-compose.yaml
+++ b/vms-home/docker-30/vault/docker-compose.yaml
--- a/vms-home/docker-30/vault/gitea-access-into-vault.md
+++ b/vms-home/docker-30/vault/gitea-access-into-vault.md
--- a/vms-home/docker-30/vault/readme.md
+++ b/vms-home/docker-30/vault/readme.md
--- a/vms-home/docker-30/vault/terraform/main.tf
+++ b/vms-home/docker-30/vault/terraform/main.tf
--- a/vms-home/docker-30/vault/terraform/terraform.tfstate
+++ b/vms-home/docker-30/vault/terraform/terraform.tfstate
--- a/vms-home/docker-30/vault/terraform/terraform.tfstate.backup
+++ b/vms-home/docker-30/vault/terraform/terraform.tfstate.backup
--- a/vms-home/docker-30/vault/terraform/versions.tf
+++ b/vms-home/docker-30/vault/terraform/versions.tf
--- a/vms-home/docker-30/vault/vault-backup.sh
+++ b/vms-home/docker-30/vault/vault-backup.sh
--- a/vms-home/docker-30/zot/config.yaml
+++ b/vms-home/docker-30/zot/config.yaml
--- a/vms-home/docker-30/zot/docker-compose.yaml
+++ b/vms-home/docker-30/zot/docker-compose.yaml
--- a/vms/docker-dev-22/psmf-data-sync/.env
+++ b/vms/docker-dev-22/psmf-data-sync/.env
@@ -0,0 +1,20 @@
+# Image (built by .gitea/workflows/build.yaml)
+# IMAGE_REPO is the Gitea repo path, e.g. "jan/psmf-data-sync"
+IMAGE_REPO=gitea.home.hrajfrisbee.cz/psmf/psmf-data-syncer
+IMAGE_TAG=0.2
+
+# Host port to publish the syncer on (container listens on 8002 internally)
+HOST_PORT=8003
+
+# External Postgres the syncer restores into
+DB_HOST=192.168.123.23
+DB_PORT=5432
+DB_NAME=psmf-dev
+DB_USER=psmf-dev
+DB_PASSWORD=** masked even when it is so secret **
+
+# Which Redis DB index to use (0-15)
+REDIS_DB=0
+
+# info | debug
+LOG_LEVEL=debug
--- a/vms/docker-dev-22/psmf-data-sync/docker-compose.yml
+++ b/vms/docker-dev-22/psmf-data-sync/docker-compose.yml
@@ -0,0 +1,28 @@
+services:
+  redis:
+    image: redis:7-alpine
+    restart: unless-stopped
+    command: ["redis-server", "--appendonly", "yes"]
+    volumes:
+      - redis-data:/data
+
+  psmf-data-sync:
+    image: ${IMAGE_REPO}:${IMAGE_TAG:-latest}
+    restart: unless-stopped
+    depends_on:
+      - redis
+    ports:
+      - "${HOST_PORT:-8003}:8002"
+    environment:
+      DB_HOST: ${DB_HOST}
+      DB_PORT: ${DB_PORT:-5432}
+      DB_NAME: ${DB_NAME}
+      DB_USER: ${DB_USER}
+      DB_PASSWORD: ${DB_PASSWORD}
+      REDIS_HOST: redis
+      REDIS_PORT: 6379
+      REDIS_DB: ${REDIS_DB:-0}
+      LOG_LEVEL: ${LOG_LEVEL:-info}
+
+volumes:
+  redis-data:
--- a/vms/docker-dev-22/readme.md
+++ b/vms/docker-dev-22/readme.md
@@ -0,0 +1,22 @@
+## docker
+
+```bash
+# prereqs + keyring
+sudo apt-get update
+sudo apt-get install -y ca-certificates curl
+sudo install -m 0755 -d /etc/apt/keyrings
+sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
+sudo chmod a+r /etc/apt/keyrings/docker.asc
+
+# repo
+echo \
+  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
+  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
+  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+
+# install
+sudo apt-get update
+sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+
+
+```
--- a/vms/storage-23/pg_hba.conf
+++ b/vms/storage-23/pg_hba.conf
@@ -0,0 +1,22 @@
+# Database administrative login by Unix domain socket
+local   all             postgres                                peer
+
+# TYPE  DATABASE        USER            ADDRESS                 METHOD
+
+# "local" is for Unix domain socket connections only
+local   all             all                                     peer
+# IPv4 local connections:
+host    all             all             127.0.0.1/32            scram-sha-256
+# IPv6 local connections:
+host    all             all             ::1/128                 scram-sha-256
+# Allow replication connections from localhost, by a user with the
+# replication privilege.
+local   replication     all                                     peer
+host    replication     all             127.0.0.1/32            scram-sha-256
+host    replication     all             ::1/128                 scram-sha-256
+
+# TYPE  DATABASE        USER            ADDRESS                 METHOD
+host    psmf            psmf            192.168.123.21/32       md5
+host    psmfapi         psmf            192.168.123.21/32       md5
+host    postgres        psmf            192.168.123.21/32       md5
+host    psmf-dev        psmf-dev        192.168.123.22/32       md5
--- a/vms/utility-101-shadow/docker/monitoring/prometheus.yml
+++ b/vms/utility-101-shadow/docker/monitoring/prometheus.yml
@@ -45,7 +45,7 @@ scrape_configs:
          - https://gitea.home.hrajfrisbee.cz/
          - https://vault.hrajfrisbee.cz/
          - https://idm.home.hrajfrisbee.cz/
-          - https://maru-hleda-byt.home.hrajfrisbee.cz/mapa_bytu.html
+          # - https://maru-hleda-byt.home.hrajfrisbee.cz/mapa_bytu.html
          # - https://nonexistent.home.hrajfrisbee.cz/
    relabel_configs:
      - source_labels: [__address__]
Author	SHA1	Message	Date
Jan Novak	ed10cc837a	chore: add CLAUDE.md, kubectl notes, and settings/gitignore tweaks Document the repo for Claude Code, note a couple of handy kubectl one-liners, allow a few more Bash/WebFetch/WebSearch permissions used during this session, and ignore the relocated zot sync-credentials.json. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:56 +02:00
Jan Novak	b62b37542b	servers/shadow: open GitLab SSH port, add docker-dev/czechultimate nginx routes Forward tcp/33022 (FORWARD, LIBVIRT_INP/OUT, PREROUTING DNAT to 192.168.123.21) so the GitLab remote is reachable, add nginx vhosts for docker-dev-22 and czechultimate.cz, and route fuj-management through the k8s gateway. Also record saint's eno0 interface config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:49 +02:00
Jan Novak	4b39834f10	servers/barber: add barber server docs Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:42 +02:00
Jan Novak	6efa069b12	servers/psmf: add psmf server docs, nginx/tailscale config, and data-sync Document the psmf host (DNS, second tailscale netns instance, nginx vhosts, migration plan) and add the docker-dev-22 psmf-data-sync service plus the storage-23 pg_hba.conf entry it needs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:39 +02:00
Jan Novak	d541f2e1d2	infra: relocate docker-30 compose stack under vms-home/, add garage and frisbee-drills Move the docker-30 service definitions (gitea, kanidm, vault, zot, nginx, lab-proxy, maru-hleda-byt, fuj-management) into vms-home/ to match the naming convention used for other hosts, and add the new garage object-storage and frisbee-drills services alongside it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:30 +02:00
Jan Novak	d43ffd488e	observability: add k8s API/kubelet tracing, Alloy, Mimir and Loki Wire kube-apiserver and kubelet tracing to a Jaeger collector on docker-29, deploy Grafana Alloy in-cluster to ship logs/metrics, and stand up Mimir + Loki on docker-30 as their backing stores. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-09 00:05:21 +02:00