lab1 workers on both PVE nodes: per-node template + MD, balanced CA

PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage: - template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50; lab1-workers renamed for symmetry (decision #18, #7 superseded) - CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19). Scale test: 16 pods -> 4 + 4 across both nodes. - runbook, image-builder copy procedure, fixes tables, future task to report three CA clusterapi bugs upstream, lab facts, changelog, execution log Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lab1: 2 GiB workers, CAPMOX memory overcommit, CA install (Step 6), future-tasks
2026-09-30 11:07:17 +02:00 · 2026-09-30 11:05:39 +02:00 · 2026-09-30 08:39:12 +02:00 · 2026-09-30 08:36:03 +02:00 · 2026-09-29 23:16:49 +02:00 · 2026-09-28 09:23:41 +02:00
25 changed files with 1608 additions and 90 deletions
@@ -1,36 +1,74 @@
 ---
 name: cluster-inspector
-description: Read-only inspection and triage of the lab management cluster (CAPI, CAPMOX, IPAM, cluster-autoscaler) and workload clusters. Use when a cluster, Machine, ProxmoxMachine, node or autoscaling behaviour needs diagnosing. Never mutates anything.
-tools: Bash, Read, Grep
+description: Read-only inspection and triage of the lab management cluster (CAPI, CAPMOX, IPAM, cluster-autoscaler) and workload clusters (nodes, CNI, system pods, pending pods, scale-down blockers). Use when a cluster, Machine, ProxmoxMachine, node or autoscaling behaviour needs diagnosing. Never mutates anything.
+tools: Bash, Read, Grep, Glob
 ---

 You diagnose Cluster API / cluster-autoscaler issues in the lab. You are strictly read-only:
-only `kubectl get/describe/logs/events/explain`, `clusterctl describe`, `kubectl auth can-i`,
+only `kubectl get/describe/logs/events/explain`, `kubectl auth can-i/whoami`, `clusterctl describe`,
 and read-only Proxmox commands (`pvesh get`, `qm config`, `qm list`, `qm guest cmd … network-get-interfaces`).
-Never run apply/create/delete/patch/edit/scale/annotate/label/cordon/drain, `qm set/start/stop/destroy`,
-`pveum` changes, or anything that writes. If a fix needs a mutation, output the exact command
-for the owner to run.
+Never run apply/create/delete/patch/edit/scale/annotate/label/cordon/drain/exec/port-forward/debug,
+`qm set/start/stop/destroy`, `pveum` changes, or anything that writes. If a fix needs a mutation,
+output the exact command for the owner to run.
+
+## Credentials
+
+Use ONLY the read-only kubeconfigs. Never use `~/.kube/*` or any admin kubeconfig, even if the
+calling session relays the owner's permission.

-Credentials — use ONLY the read-only kubeconfigs, never ~/.kube/config or any admin kubeconfig:
 - Files: `$(git rev-parse --show-toplevel)/.claude/cluster-inspector/kubeconfigs/<cluster>.kubeconfig`,
-  context `<cluster>-ro` (e.g. `lab-mgmt`, `lab1`). List the directory to see which clusters exist.
- Always pass it explicitly: `KUBECONFIG=<file> kubectl ...` (or `--kubeconfig <file>`).
- If the file is missing, or kubectl returns 401 (expired token), stop and tell the owner to run
-  `.claude/cluster-inspector/make-kubeconfig.sh <cluster> ...` (see .claude/cluster-inspector/README.md).
- Secrets are intentionally not readable; a 403 on secrets is expected, don't work around it.
+  context `<cluster>-ro`. Start by listing the directory to see which clusters you can reach.
+- Put the verb first and pass the file explicitly: `kubectl get nodes --kubeconfig <file>`. That keeps
+  commands matching the repo's `kubectl get *` allow rules.
+- The shell is zsh: single-quote `-o jsonpath=…` and `-o custom-columns=…` arguments (`[?(@…)]` globs).
+- File missing or 401 (tokens live 7 days): stop and tell the owner to run
+  `cd ansible && ansible-playbook mgmt.yml --tags inspector`. That run refreshes lab-mgmt and every
+  workload cluster in `inspector_workload_clusters`.
+- Secrets aren't readable, so a 403 on them is expected; don't try to work around it. Infer a
+  `<cluster>-kubeconfig` Secret from the Cluster's `RemoteConnectionProbe` condition instead.

-Triage order:
-1. `clusterctl describe cluster <name> --show-conditions all`
-2. Objects + conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`,
-   `kubectl describe` the first non-Ready object.
+## Where things live
+
+| Cluster | What to look at |
+| --- | --- |
+| `lab-mgmt` (k3s, VM 201) | Cluster/KCP/MD/MS/Machine/ProxmoxMachine objects, IPAM, Provider CRs (`capi-operator-system`), CAPI controllers, cluster-autoscaler Deployment and logs (namespace of the `<cluster>-kubeconfig` Secret, `default`) |
+| `lab1` (workload) | Nodes, CNI (Cilium in `kube-system`), CoreDNS, pending pods and their events, PDBs, `cluster-autoscaler-status` ConfigMap (check both clusters if it isn't in `kube-system` here) |
+
+Machine → Node → VM: `Machine.status.nodeRef.name`, and `providerID` = `proxmox://<SMBIOS uuid>`
+(the same on Machine, ProxmoxMachine and Node). `ProxmoxMachine.spec.virtualMachineID` gives the Proxmox VMID.
+
+## Triage order
+
+1. `clusterctl describe cluster <name> --show-conditions all --kubeconfig <mgmt file>`
+2. Objects and conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`,
+   then `kubectl describe` the first non-Ready object.
 3. Controller logs: `capi-system/capi-controller-manager`, `capmox-system/capmox-controller-manager`,
-   `capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses -A`).
-4. Node side: Node vs Machine providerID match
-   (`kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID`), kubelet/cloud-init
-   status if SSH access is given.
-5. Autoscaler: CA logs (node group discovery, scale decisions), pending pod events
-   (`TriggeredScaleUp` / `NotTriggerScaleUp`), MachineDeployment min/max + capacity annotations,
-   whether `spec.replicas` is being fought by a re-apply.
+   `capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io -A`;
+   plain `ipaddresses` is the built-in networking.k8s.io type).
+4. Workload side: Node Ready state, kubelet version and taints; Node vs Machine providerID; Cilium
+   DaemonSet/operator and CoreDNS; kube-system pods that aren't Running, and which node each runs on.
+   Kubelet/cloud-init status only if the owner gives SSH access.
+5. Autoscaler:
+   - CA logs: node group discovery (`MachineDeployment/<ns>/<name>` with min/max) and scale decisions.
+   - Pending pod events (`TriggeredScaleUp` / `NotTriggerScaleUp`).
+   - MD min/max and capacity annotations, matching the `ProxmoxMachineTemplate` size.
+   - Who owns `spec.replicas` (`--show-managed-fields`): a re-apply must not own it.
+6. Scale-down blockers on each worker. The last worker never goes to zero if any of these hold:
+   - a non-DaemonSet `kube-system` pod without a PDB (CA default `skip-nodes-with-system-pods=true`);
+   - a pod with no controller, or with local storage (`emptyDir`/`hostPath`);
+   - a PDB with `ALLOWED DISRUPTIONS 0`;
+   - `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` on a pod, or `…/scale-down-disabled`
+     on the node.

-Known failure → fix mappings live in docs/phase-1-proxmox-capi.md and docs/image-builder-proxmox.md;
-check them first. Report: root cause (with evidence lines), fix command(s), and anything unverified.
+## Report
+
+Known symptom → fix mappings are in the troubleshooting tables of docs/phase-1-proxmox-capi.md
+and docs/image-builder-proxmox.md; check them first.
+
+Reply in this order:
+
+1. Anything abnormal: root cause with evidence lines, and the fix command(s) for the owner.
+2. A short health summary; tables are fine.
+3. Anything you couldn't verify, and why (missing kubeconfig, RBAC 403, no SSH).
+
+Prefer `-o custom-columns`/`jsonpath` over dumping full YAML. Never print kubeconfig contents or tokens.
@@ -26,15 +26,23 @@ rm /tmp/lab-mgmt-admin.yaml

 ## 2. Workload clusters (after phase 1 step 3)

+Add the cluster to `inspector_workload_clusters` in `ansible/group_vars/all.yml`. Then
+`ansible-playbook mgmt.yml --tags inspector` mints a read-only kubeconfig for lab-mgmt and for every
+listed cluster whose `<name>-kubeconfig` Secret exists. Clusters without that Secret are skipped.
+
+Doing it by hand gives the same result. `--capi-cluster` reads the workload admin kubeconfig from
+the management cluster's Secret into a temp file that is deleted on exit:
+
 ```bash
-clusterctl get kubeconfig lab1 > /tmp/lab1-admin.yaml     # run against the mgmt cluster (admin)
-.claude/cluster-inspector/make-kubeconfig.sh lab1 --admin-kubeconfig /tmp/lab1-admin.yaml --apply-rbac
-rm /tmp/lab1-admin.yaml
+.claude/cluster-inspector/make-kubeconfig.sh lab1 \
+  --admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig \
+  --capi-cluster default/lab1 \
+  --apply-rbac
 ```

 ## Expected verification output

-```
+```text
  list pods                                     yes
  get machines.cluster.x-k8s.io                 yes    # "no" on workload clusters (no CAPI CRDs), fine
  delete machines.cluster.x-k8s.io              no
@@ -46,7 +54,8 @@ Anything else means the RBAC is wrong. Don't use the agent until it matches.

 ## Refresh / revoke

- **Refresh:** re-run the same command without `--apply-rbac`. The default lifetime is 7 days
+- **Refresh:** `ansible-playbook mgmt.yml --tags inspector` refreshes all clusters. You can also
+  re-run the same script command, with or without `--apply-rbac`; it's idempotent. The default lifetime is 7 days
  (`--duration`); the API server may cap it (`--service-account-max-token-expiration`).
 - **Revoke everything:** `kubectl delete sa -n kube-system readonly-inspector` (as admin). That
  invalidates every minted token; re-apply `rbac.yaml` to start over.
@@ -5,29 +5,43 @@
 #   <name>                   cluster name, e.g. lab-mgmt or lab1 -> kubeconfigs/<name>.kubeconfig
 #   --admin-kubeconfig PATH  admin kubeconfig used to apply RBAC / mint the token (default: $KUBECONFIG or ~/.kube/config)
 #   --context CTX            context inside the admin kubeconfig
+#   --capi-cluster [NS/]NAME workload cluster: --admin-kubeconfig is the management cluster; the workload
+#                            admin kubeconfig is read from Secret NS/NAME-kubeconfig (NS default: default)
+#                            into a temp file that is removed on exit
 #   --server URL             override API server URL (k3s.yaml says https://127.0.0.1:6443)
 #   --duration DUR           token lifetime (default 168h); re-run the script to refresh
 #   --apply-rbac             kubectl apply rbac.yaml first (needed once per cluster)
 set -euo pipefail
 here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; server=""; duration="168h"; apply=0
+name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; capi=""; server=""; duration="168h"; apply=0
 while [[ $# -gt 0 ]]; do
  case "$1" in
    --admin-kubeconfig) admin="$2"; shift 2 ;;
    --context)          ctx="$2"; shift 2 ;;
+    --capi-cluster)     capi="$2"; shift 2 ;;
    --server)           server="$2"; shift 2 ;;
    --duration)         duration="$2"; shift 2 ;;
    --apply-rbac)       apply=1; shift ;;
-    -h|--help)          sed -n '2,10p' "$0"; exit 0 ;;
+    -h|--help)          sed -n '2,13p' "$0"; exit 0 ;;
    -*)                 echo "unknown option $1" >&2; exit 1 ;;
    *)                  name="$1"; shift ;;
  esac
 done
-[[ -n "$name" ]] || { sed -n '2,10p' "$0" >&2; exit 1; }
+[[ -n "$name" ]] || { sed -n '2,13p' "$0" >&2; exit 1; }

 k=(kubectl --kubeconfig "$admin")
 [[ -n "$ctx" ]] && k+=(--context "$ctx")

+if [[ -n "$capi" ]]; then
+  ns="default"; cluster="$capi"
+  [[ "$capi" == */* ]] && { ns="${capi%%/*}"; cluster="${capi#*/}"; }
+  workload="$(mktemp)"                               # mktemp creates it 0600
+  trap 'rm -f "$workload"' EXIT
+  "${k[@]}" -n "$ns" get secret "$cluster-kubeconfig" -o jsonpath='{.data.value}' | base64 -d > "$workload"
+  [[ -s "$workload" ]] || { echo "Secret $ns/$cluster-kubeconfig is empty or missing" >&2; exit 1; }
+  k=(kubectl --kubeconfig "$workload")
+fi
+
 if (( apply )); then "${k[@]}" apply -f "$here/rbac.yaml"; fi

 token="$("${k[@]}" -n kube-system create token readonly-inspector --duration "$duration")"
@@ -33,6 +33,10 @@ rules:
  - apiGroups: [""]
    resources: ["nodes", "namespaces"]
    verbs: ["get", "list", "watch"]
+  # clusterctl describe checks the CAPI CRD storage version
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["get", "list", "watch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
@@ -18,7 +18,35 @@
      "Bash(clusterctl generate *)",
      "Bash(ansible mgmt *)",
      "Bash(kubectl get *)",
-      "Bash(kubectl auth *)"
+      "Bash(kubectl auth *)",
+      "Bash(kubectl describe *)",
+      "Bash(kubectl logs *)",
+      "Bash(kubectl events *)",
+      "Bash(clusterctl describe *)",
+      "Bash(ansible pve *)",
+      "Read(//Users/jan.novak/**)",
+      "Bash(export KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig)",
+      "Bash(kubectl explain *)",
+      "Bash(grep -v '^ *$')",
+      "Bash(git ls-remote *)",
+      "Bash(awk -F/ '{print $3}')",
+      "Bash(curl -fsSL https://kubernetes.github.io/autoscaler/index.yaml)",
+      "Bash(awk '/^  cluster-autoscaler:/{f=1} f&&/appVersion:/{a=$2} f&&/^    version:/{print $2, a; n++} n>=6{exit}')",
+      "Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get md lab1-workers-pv1-1 -n default -o json)",
+      "Bash(clusterctl --kubeconfig /Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig describe cluster lab1 -n default --show-conditions all)",
+      "Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get md lab1-workers-pv1-1 -n default -o json --show-managed-fields)",
+      "Bash(export KUBECONFIG=$K)",
+      "Bash(helm show *)",
+      "Bash(grep -rniE 'backlog|future tasks|TODO' --include='*.md' . | head; sed -n '/## Key decisions/,/^## /p' README.md | head -5; grep -n 'docs/' README.md | head -20)",
+      "Bash(helm template *)",
+      "Bash(sed -e 's|^\\\\*\\\\*Created:\\\\*\\\\* .*|**Created:** 2026-09-29 23:15 CEST|' /Users/jan.novak/.claude/plans/purrfect-frolicking-lovelace.md > docs/plans/2026-09-29-2315-pv1-2-workers.md && head -3 docs/plans/2026-09-29-2315-pv1-2-workers.md)",
+      "WebSearch",
+      "WebFetch(domain:github.com)",
+      "Bash(git apply *)"
+    ],
+    "additionalDirectories": [
+      "/Users/jan.novak/srv/go/cluster-autoscaler-with-capi",
+      "/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/docs/plans"
    ]
  }
 }
@@ -0,0 +1,107 @@
+# AGENTS.md
+
+Home-lab research project: cluster-autoscaler + Cluster API on Proxmox VE (phase 1), then a
+KVM/libvirt provider (phase 2) and Kamaji hosted control planes / "poor man's KaaS" (phase 3).
+Start with README.md; details live in docs/.
+
+## Owner context
+
+Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux, distributed systems).
+
+## Lab facts (don't re-derive)
+
+- PVE cluster `homelab`: nodes `pv1-1` (192.168.0.8) and `pv1-2` (192.168.0.10); API
+  `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0). VMIDs are cluster-wide.
+- VM network: `vmbr0` is VLAN-aware, VMs in **VLAN 4** → always `bridge=vmbr0` + tag/`vlan: 4`,
+  never `vmbr0.4@vmbr0` (that is an `ip link` name, not a bridge).
+- Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2.
+- Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds).
+- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55), VMID 101 on `pv1-1` and copy
+  9101 on `pv1-2` (local-lvm can't clone cross-node → one worker MD per PVE node). Login only via
+  cloud-init.
+- Management cluster: VM 201 `lab-mgmt`, 192.168.4.21 (`ssh ubuntu@`), built by the idempotent
+  `ansible/mgmt.yml` (re-run must be `changed=0`); admin kubeconfig `~/.kube/lab-mgmt.kubeconfig`.
+- Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`,
+  PVESDNUser on `/sdn/zones/localnetwork`.
+
+## Version rules
+
+- `docs/versions.md` is the source of truth. Any version change updates it in the same change.
+- Before proposing any deployment or upgrade, verify versions are current (use the
+  `k8s-version-check` skill if available). Go module proxy / git ls-remote / Helm index.yaml work
+  even when the GitHub API is unavailable.
+- CAPI core is pinned to **v1.12.x** because CAPMOX v0.9 is only tested on CAPI 1.11/1.12. This caps
+  workload Kubernetes at 1.35 and cluster-autoscaler at 1.35.x. Don't bump CAPI unless the CAPMOX
+  compatibility table allows it.
+- Always pin versions explicitly (management cluster: `ansible/group_vars/all.yml`; `helm install
+  --version`); never rely on "latest".
+- CAPI providers are managed by cluster-api-operator (Provider CRs from `ansible/mgmt.yml`). Never
+  run `clusterctl init/upgrade/move/delete` against the management cluster; `clusterctl` is only for
+  `generate cluster` (always with `--infrastructure proxmox:v0.9.1`), `describe`, `get kubeconfig`.
+- Check a provider's `metadata.yaml` contract and CAPI version (go.mod) before adding it to the
+  management cluster — all providers share one CAPI core (known skew: Kamaji CP provider v0.21.0 is
+  on CAPI 1.13).
+
+## Facts about providers: verify in source
+
+READMEs lag. For "does provider X support Y" read the API types and templates of the exact tagged
+release (Go proxy zip: `https://proxy.golang.org/<module>/@v/<ver>.zip`). Delegate to the
+`upstream-researcher` agent for anything non-trivial.
+
+## Safety
+
+- Never commit secrets: PVE token secret, kubeconfigs, `clusterctl.yaml` (see .gitignore).
+- Cluster inspection is read-only via the `cluster-inspector` agent and read-only kubeconfigs
+  (`.Codex/cluster-inspector/`, see its README). Mutations (`kubectl apply/delete/scale`, `qm`, `pveum`)
+  are proposed as commands for the owner to run unless explicitly asked to execute them.
+- Don't delete Proxmox VMs/templates or CAPI objects on your own initiative.
+
+## Conventions
+
+- Docs in English, Markdown, Mermaid for diagrams (Gitea renders it).
+- Troubleshooting findings go into the relevant doc's troubleshooting/fixes table:
+  symptom → root cause → fix.
+- Example configs live in `configs/` as `*.example` or non-secret values files.
+
+## Project history
+
+Research repo: plans, conversations and decisions are kept in the repo, dated. Timestamps always
+come from `date` (`+%Y-%m-%d-%H%M` for file names, `+%Y-%m-%d %H:%M %Z` for headings), never guessed.
+Summaries only, never raw transcripts; mask secrets and check the staged diff before committing.
+
+| What | Where | When |
+| --- | --- | --- |
+| Plans | `docs/plans/YYYY-MM-DD-HHMM-<slug>.md` | every plan-mode plan |
+| Execution logs | `docs/plans-executions/<plan's file name>` | after each step of an approved plan |
+| Session notes | `docs/sessions/YYYY-MM-DD-HHMM-<slug>.md` | research/design/debugging conversations without a plan |
+| Decisions | `docs/decisions.md` (one table) | a direction chosen, rejected or reversed |
+| Prompts | `docs/prompts/YYYY-MM-DD-HHMM-<slug>.md` | long crafted prompts/specs that kick off work (verbatim) |
+| Reviews | `docs/reviews/YYYY-MM-DD-HHMM-<slug>.md` | review findings, most severe first, `- [ ]` ticked as processed |
+| Changelog | `CHANGELOG.md`, newest on top | lab milestone, version change, design direction, new runbook/config — after the owner confirms it works |
+
+- **Plans:** header `# Plan: <title>` + `**Created:** <timestamp>`. Plan mode can only write to
+  `~/.Codex/plans/`, so right after `ExitPlanMode` the first action is copying the plan into
+  `docs/plans/` and committing it on its own, before any work starts.
+- **Execution logs:** a `- [ ] Step N — <title>` checklist on top (the authoritative status; a new
+  session resumes from this file alone), then one section per step in chronological order,
+  committed with that step's work. Light prose; exact commands as fenced blocks (`clusterctl`,
+  `helm`, `kubectl apply`, `qm`, Packer), each with one line on why it was run that way, including
+  commands the owner reports running on the lab and outputs that mattered (VM IDs, IPs, resolved
+  versions, errors). Call out deviations from the plan; end with "Worth noting". Problems also go
+  into the relevant fixes table, linked from the log.
+- **Session notes:** write one unprompted when a conversation produced a finding worth keeping
+  (option compared or rejected, compatibility check, root cause, open question) that no plan
+  covers; mention it in the reply. Sections: Question, What was checked (sources, versions with
+  date checked), Findings, Open questions, Follow-ups. Extend the note if the topic continues.
+- **Decisions:** never rewrite a row; a reversal is a new row and the old one becomes
+  `superseded by #N`. Design-doc suggestions are `proposed` until lab work confirms them. Link
+  the plan/session/doc with the reasoning. README "Key decisions so far" stays the short summary.
+
+## Git
+
+- Remote: Gitea `gitea.home.hrajfrisbee.cz/kacerr/cluster-autoscaler-with-capi`.
+- Docs, configs and history files go straight to `main`. Phase-3 Go code or anything the owner
+  calls a feature: `feat/<slug>` branch, `git push -u`, MR via
+  `tea pr create --title … --description … --base main --head <branch>` (fallback: print
+  `…/compare/main...<branch>`). Never merge or delete branches from the CLI.
+- Commit messages end with a `Co-Authored-By:` trailer for Codex. `/ship` commits and pushes.
@@ -1,5 +1,33 @@
 # Changelog

+## 2026-09-30 11:05 CEST — lab1 workers on both PVE nodes, balanced autoscaling
+
+- **PVE cluster.** It is now the 2-node cluster `homelab`: `pv1-1` plus `pv1-2` (192.168.0.10).
+  - `vmbr0` on `pv1-2` is VLAN-aware.
+  - Template 101 is copied to `pv1-2` as 9101 (clone + offline migrate). `local-lvm` can't clone
+    across nodes.
+- **lab1 workers.** One MachineDeployment per PVE node: `lab1-workers-pv1-1` (max 6) and
+  `lab1-workers-pv1-2` (max 12).
+  - Each is pinned by `allowedNodes`.
+  - The node IP pool is `.31–.50`.
+  - The control plane stays on `pv1-1`.
+  - The old `lab1-workers` was replaced to get symmetric names.
+- **cluster-autoscaler.** Balances the two groups (a scale-up of 16 pods gave 4 + 4).
+  - Settings: `balancing-label: node.cluster.x-k8s.io/pool`, `scan-interval: 1m` and `v: 2`.
+  - The MDs got the pool label, `capacity…/labels` (os/arch) and `maxPods`.
+  - The request rate is left at the default. `scan-interval` cuts the discovery calls the
+    clusterapi provider makes without a cache.
+- **Findings.** Three CA 1.35.2 clusterapi bugs: uncached discovery, labels read from a wrong
+  v1beta2 path, and empty os/arch labels for CAPMOX. They are worked around in config and logged
+  in `docs/future-tasks.md` for reporting upstream.
+- **Files:**
+  - `configs/capi/lab1.yaml`, `configs/cluster-autoscaler/values.yaml`,
+    `configs/capi/clusterctl.yaml.example`
+  - `docs/phase-1-proxmox-capi.md`, `docs/image-builder-proxmox.md`, `docs/decisions.md` (#18, #19;
+    #7 superseded), `docs/future-tasks.md`
+  - plan, execution log and session note
+  - `README.md`, `CLAUDE.md`, `AGENTS.md`
+
 ## 2026-09-27 10:40 CEST — Management cluster running, built by an idempotent Ansible playbook

 - Phase 1 Step 2 done: `ansible/mgmt.yml` creates VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`
@@ -10,12 +10,15 @@ Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux

 ## Lab facts (don't re-derive)

- PVE node `pv1-1`, API `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0).
+- PVE cluster `homelab`: nodes `pv1-1` (192.168.0.8) and `pv1-2` (192.168.0.10); API
+  `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0). VMIDs are cluster-wide.
 - VM network: `vmbr0` is VLAN-aware, VMs in **VLAN 4** → always `bridge=vmbr0` + tag/`vlan: 4`,
  never `vmbr0.4@vmbr0` (that is an `ip link` name, not a bridge).
 - Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2.
 - Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds).
- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55). Login only via cloud-init.
+- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55), VMID 101 on `pv1-1` and copy
+  9101 on `pv1-2` (local-lvm can't clone cross-node → one worker MD per PVE node). Login only via
+  cloud-init.
 - Management cluster: VM 201 `lab-mgmt`, 192.168.4.21 (`ssh ubuntu@`), built by the idempotent
  `ansible/mgmt.yml` (re-run must be `changed=0`); admin kubeconfig `~/.kube/lab-mgmt.kubeconfig`.
 - Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`,
@@ -74,6 +77,7 @@ Summaries only, never raw transcripts; mask secrets and check the staged diff be
 | Decisions | `docs/decisions.md` (one table) | a direction chosen, rejected or reversed |
 | Prompts | `docs/prompts/YYYY-MM-DD-HHMM-<slug>.md` | long crafted prompts/specs that kick off work (verbatim) |
 | Reviews | `docs/reviews/YYYY-MM-DD-HHMM-<slug>.md` | review findings, most severe first, `- [ ]` ticked as processed |
+| Future tasks | `docs/future-tasks.md`, newest on top | work the owner defers: problem, options, cost, changes, owner-run commands |
 | Changelog | `CHANGELOG.md`, newest on top | lab milestone, version change, design direction, new runbook/config — after the owner confirms it works |

 - **Plans:** header `# Plan: <title>` + `**Created:** <timestamp>`. Plan mode can only write to
@@ -26,7 +26,7 @@ growing, in later phases, into a "poor man's Kubernetes-as-a-Service".

 | Hypervisor type | Management API | CAPI infrastructure provider |
 | --- | --- | --- |
-| Proxmox VE (e.g. `pv1-1`, 192.168.0.8) | PVE REST API :8006 | CAPMOX (phase 1) |
+| Proxmox VE cluster `homelab`: `pv1-1` 192.168.0.8, `pv1-2` 192.168.0.10 | PVE REST API :8006 | CAPMOX (phase 1) |
 | Ubuntu 24.04 + KVM/libvirt | libvirt RPC (`qemu+ssh`/`qemu+tls`) | none mature upstream → phase 2 |

 VM network: `vmbr0` (VLAN-aware), VLAN 4, build host 192.168.4.181.
@@ -82,6 +82,7 @@ Phases 2 and 3 are independent and can be done in either order.
 │   └── cluster-inspector/        # RBAC, kubeconfig minting script, gitignored kubeconfigs/
 ├── docs/
 │   ├── decisions.md              # decision log: alternatives, why, status
+│   ├── future-tasks.md           # deferred work with context, newest first
 │   ├── plans/                    # plans (plan mode), one file per plan
 │   ├── plans-executions/         # per-plan execution logs with exact commands
 │   ├── sessions/                 # research/design conversation summaries
@@ -117,3 +118,6 @@ Short summary; the full log with alternatives and status is [docs/decisions.md](
   `status.capacity` on `ProxmoxMachineTemplate`.
 6. **Management cluster from one idempotent Ansible playbook**, CAPI providers installed by
   **cluster-api-operator** instead of `clusterctl init` — re-runs are no-ops, upgrades are version bumps.
+7. **One template copy and one worker MachineDeployment per PVE node** (no shared storage):
+   `local-lvm` templates can't be cloned across nodes. cluster-autoscaler balances the groups on a
+   shared `node.cluster.x-k8s.io/pool` label.
@@ -29,6 +29,11 @@ mgmt_memory_mib: 4096
 mgmt_disk_size: 30G
 mgmt_ssh_public_key: "{{ lookup('file', lookup('env', 'HOME') + '/.ssh/id_ed25519.pub') }}"

+# Workload clusters that get a cluster-inspector read-only kubeconfig (--tags inspector).
+# Skipped while the cluster's <name>-kubeconfig Secret doesn't exist.
+inspector_workload_clusters:
+  - { name: lab1, namespace: default }
+
 # Installed by the cluster-api-operator; CR namespace = install namespace (clusterctl defaults).
 capi_providers:
  - { kind: CoreProvider, name: cluster-api, namespace: capi-system, version: "{{ capi_version }}" }
@@ -259,3 +259,32 @@
      become: false
      changed_when: false
      tags: [inspector]
+
+    - name: Workload clusters that have a CAPI kubeconfig Secret
+      ansible.builtin.command: >-
+        kubectl --kubeconfig {{ lookup('env', 'HOME') }}/.kube/lab-mgmt.kubeconfig
+        -n {{ item.namespace }} get secret {{ item.name }}-kubeconfig -o name
+      loop: "{{ inspector_workload_clusters }}"
+      loop_control:
+        label: "{{ item.namespace }}/{{ item.name }}"
+      register: workload_kubeconfig_secrets
+      delegate_to: localhost
+      become: false
+      changed_when: false
+      failed_when: false
+      tags: [inspector]
+
+    - name: Read-only kubeconfigs for workload clusters (skips clusters that don't exist yet)
+      ansible.builtin.command: >-
+        {{ playbook_dir }}/../.claude/cluster-inspector/make-kubeconfig.sh {{ item.item.name }}
+        --admin-kubeconfig {{ lookup('env', 'HOME') }}/.kube/lab-mgmt.kubeconfig
+        --capi-cluster {{ item.item.namespace }}/{{ item.item.name }}
+        --apply-rbac
+      loop: "{{ workload_kubeconfig_secrets.results }}"
+      loop_control:
+        label: "{{ item.item.namespace }}/{{ item.item.name }}"
+      when: item.rc == 0
+      delegate_to: localhost
+      become: false
+      changed_when: false
+      tags: [inspector]
@@ -4,8 +4,9 @@

 ## workload defaults
 PROXMOX_SOURCENODE: "pv1-1"               # node holding the template
-TEMPLATE_VMID: "<vmid of ubuntu-2404-kube-v1.35.9>"
-ALLOWED_NODES: "[pv1-1]"                  # more nodes only with shared storage
+TEMPLATE_VMID: "<vmid of ubuntu-2404-kube-v1.35.9>"   # 101 on pv1-1 (copy 9101 on pv1-2)
+ALLOWED_NODES: "[pv1-1]"                  # local-lvm: only the template's node. More PVE nodes = one
+                                          # worker MD + PMT per node by hand (lab1.yaml, decision #18)
 VM_SSH_KEYS: "ssh-ed25519 AAAA..."

 ## network (VLAN 4 on vmbr0 — add `vlan: 4` to the network device in ProxmoxMachineTemplates)
@@ -0,0 +1,420 @@
+# Workload cluster lab1 (phase-1 Step 3). Generated 2026-09-27 with clusterctl v1.12.11:
+#
+#   PROXMOX_SOURCENODE=pv1-1 TEMPLATE_VMID=101 ALLOWED_NODES="[pv1-1]" \
+#   VM_SSH_KEYS="$(cat ~/.ssh/id_ed25519.pub)" \
+#   CONTROL_PLANE_ENDPOINT_IP=192.168.4.30 NODE_IP_RANGES="[192.168.4.31-192.168.4.39]" \
+#   GATEWAY=192.168.4.8 IP_PREFIX=24 DNS_SERVERS="[8.8.8.8, 1.1.1.1]" BRIDGE=vmbr0 \
+#   BOOT_VOLUME_DEVICE=scsi0 BOOT_VOLUME_SIZE=40 NUM_SOCKETS=1 NUM_CORES=2 MEMORY_MIB=4096 \
+#   clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --kubernetes-version v1.35.9 \
+#     --control-plane-machine-count 1 --worker-machine-count 1 --target-namespace default
+#
+# Hand edits (docs/phase-1-proxmox-capi.md Step 3):
+#   1. pod CIDR 192.168.0.0/16 -> 10.244.0.0/16 (overlaps the lab LAN)
+#   2. kube-vip v0.7.1 -> v1.2.4 (kube-vip-prepare.sh kept)
+#   3. `format: qcow2` removed from both ProxmoxMachineTemplates (local-lvm is LVM-thin, raw only)
+#   4. `vlan: 4` on net0 of both ProxmoxMachineTemplates
+#   5. worker template: 1 socket x 4 cores, 2048 MiB, named lab1-worker-2g (owner's choice, decision #16;
+#      first 4096 MiB as lab1-worker). Templates are never edited in place: a new size = a new
+#      template name + MD infrastructureRef, which rolls the workers.
+#   6. MachineDeployment lab1-workers: `replicas` removed (the autoscaler owns it)
+#   7. MachineDeployment lab1-workers: autoscaler min/max + capacity annotations (runbook Step 5)
+#   8. MachineDeployment lab1-workers: rollout maxSurge 0 / maxUnavailable 1 (memory-tight host)
+#   9. ProxmoxCluster lab1: schedulerHints.memoryAdjustment 160 (CAPMOX memory overcommit, decision #17)
+#  10. Second worker group on PVE node pv1-2 (decision #18): MachineDeployment lab1-workers-pv1-2 +
+#      ProxmoxMachineTemplate lab1-worker-2g-pv1-2 (template copy 9101, allowedNodes [pv1-2]).
+#      local-lvm is node-local, so every node needs its own template VMID and its own MD.
+#  11. ProxmoxCluster lab1: node IP pool .31-.39 -> .31-.50 (20 addresses). Max sizes 6 (pv1-1) +
+#      12 (pv1-2) + 1 CP = 19; one address stays free for the KCP rollout surge (CAPMOX pools also
+#      hold a freed address for 60 s before reuse).
+#  12. Both worker MDs: label node.cluster.x-k8s.io/pool=workers (template + capacity labels
+#      annotation) for CA --balancing-label; capacity key maximum-pods -> maxPods.
+#  13. Symmetric per-node names (owner's request): MachineDeployment lab1-workers -> lab1-workers-pv1-1,
+#      ProxmoxMachineTemplate lab1-worker-2g -> lab1-worker-2g-pv1-1 (+ explicit allowedNodes [pv1-1]).
+#      A rename is a new object: the old MD and PMT are deleted by hand once the old MD is at 0.
+apiVersion: cluster.x-k8s.io/v1beta2
+kind: Cluster
+metadata:
+  labels:
+    cluster.x-k8s.io/cluster-name: lab1
+  name: lab1
+  namespace: default
+spec:
+  clusterNetwork:
+    pods:
+      cidrBlocks:
+      - 10.244.0.0/16
+  controlPlaneRef:
+    apiGroup: controlplane.cluster.x-k8s.io
+    kind: KubeadmControlPlane
+    name: lab1-control-plane
+  infrastructureRef:
+    apiGroup: infrastructure.cluster.x-k8s.io
+    kind: ProxmoxCluster
+    name: lab1
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
+kind: ProxmoxCluster
+metadata:
+  labels:
+    cluster.x-k8s.io/cluster-name: lab1
+  name: lab1
+  namespace: default
+spec:
+  allowedNodes:
+  - pv1-1
+  controlPlaneEndpoint:
+    host: 192.168.4.30
+    port: 6443
+  dnsServers:
+  - 8.8.8.8
+  - 1.1.1.1
+  ipv4Config:
+    addresses:
+    - 192.168.4.31-192.168.4.50
+    gateway: 192.168.4.8
+    prefix: 24
+  # CAPMOX reserves the configured memory of every non-template VM, stopped ones included, against
+  # host total × memoryAdjustment%. 160% of 29.16 GiB = 46.7 GiB (~17.7 GiB headroom today).
+  # Real overcommit protection on pv1-1: KSM (ksmtuned) + 8 GiB swap.
+  schedulerHints:
+    memoryAdjustment: 160
+---
+apiVersion: controlplane.cluster.x-k8s.io/v1beta2
+kind: KubeadmControlPlane
+metadata:
+  name: lab1-control-plane
+  namespace: default
+spec:
+  kubeadmConfigSpec:
+    files:
+    - content: |
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          creationTimestamp: null
+          name: kube-vip
+          namespace: kube-system
+        spec:
+          containers:
+          - args:
+            - manager
+            env:
+            - name: cp_enable
+              value: "true"
+            - name: vip_interface
+              value: ""
+            - name: address
+              value: 192.168.4.30
+            - name: port
+              value: "6443"
+            - name: vip_arp
+              value: "true"
+            - name: vip_leaderelection
+              value: "true"
+            - name: vip_leaseduration
+              value: "15"
+            - name: vip_renewdeadline
+              value: "10"
+            - name: vip_retryperiod
+              value: "2"
+            image: ghcr.io/kube-vip/kube-vip:v1.2.4
+            imagePullPolicy: IfNotPresent
+            name: kube-vip
+            resources: {}
+            securityContext:
+              capabilities:
+                add:
+                - NET_ADMIN
+                - NET_RAW
+            volumeMounts:
+            - mountPath: /etc/kubernetes/admin.conf
+              name: kubeconfig
+          hostAliases:
+          - hostnames:
+            - localhost
+            - kubernetes
+            ip: 127.0.0.1
+          hostNetwork: true
+          volumes:
+          - hostPath:
+              path: /etc/kubernetes/admin.conf
+              type: FileOrCreate
+            name: kubeconfig
+        status: {}
+      owner: root:root
+      path: /etc/kubernetes/manifests/kube-vip.yaml
+    - content: |
+        #!/bin/bash
+
+        # Copyright 2020 The Kubernetes Authors.
+        #
+        # Licensed under the Apache License, Version 2.0 (the "License");
+        # you may not use this file except in compliance with the License.
+        # You may obtain a copy of the License at
+        #
+        #     http://www.apache.org/licenses/LICENSE-2.0
+        #
+        # Unless required by applicable law or agreed to in writing, software
+        # distributed under the License is distributed on an "AS IS" BASIS,
+        # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+        # See the License for the specific language governing permissions and
+        # limitations under the License.
+
+        set -e
+
+        # Configure the workaround required for kubeadm init with kube-vip:
+        # xref: https://github.com/kube-vip/kube-vip/issues/684
+
+        # Nothing to do for kubernetes < v1.29
+        KUBEADM_MINOR="$(kubeadm version -o short | cut -d '.' -f 2)"
+        if [[ "$KUBEADM_MINOR" -lt "29" ]]; then
+          exit 0
+        fi
+
+        IS_KUBEADM_INIT="false"
+
+        # cloud-init kubeadm init
+        if [[ -f /run/kubeadm/kubeadm.yaml ]]; then
+          IS_KUBEADM_INIT="true"
+        fi
+
+        # ignition kubeadm init
+        if [[ -f /etc/kubeadm.sh ]] && grep -q -e "kubeadm init" /etc/kubeadm.sh; then
+          IS_KUBEADM_INIT="true"
+        fi
+
+        if [[ "$IS_KUBEADM_INIT" == "true" ]]; then
+          sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' \
+            /etc/kubernetes/manifests/kube-vip.yaml
+        fi
+      owner: root:root
+      path: /etc/kube-vip-prepare.sh
+      permissions: "0700"
+    initConfiguration:
+      nodeRegistration:
+        kubeletExtraArgs:
+        - name: provider-id
+          value: proxmox://'{{ ds.meta_data.instance_id }}'
+    joinConfiguration:
+      nodeRegistration:
+        kubeletExtraArgs:
+        - name: provider-id
+          value: proxmox://'{{ ds.meta_data.instance_id }}'
+    preKubeadmCommands:
+    - /etc/kube-vip-prepare.sh
+    users:
+    - name: root
+      sshAuthorizedKeys:
+      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINQxxkqmvtVI+8c5BkTaJ5c2HfBFRXJWMmEcevvfP9tV
+        jan.novak@Jans-MacBook-Air.local
+  machineTemplate:
+    spec:
+      infrastructureRef:
+        apiGroup: infrastructure.cluster.x-k8s.io
+        kind: ProxmoxMachineTemplate
+        name: lab1-control-plane
+  replicas: 1
+  version: v1.35.9
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
+kind: ProxmoxMachineTemplate
+metadata:
+  name: lab1-control-plane
+  namespace: default
+spec:
+  template:
+    spec:
+      disks:
+        bootVolume:
+          disk: scsi0
+          sizeGb: 40
+      full: true
+      memoryMiB: 4096
+      network:
+        networkDevices:
+        - bridge: vmbr0
+          model: virtio
+          name: net0
+          vlan: 4
+      numCores: 2
+      numSockets: 1
+      sourceNode: pv1-1
+      templateID: 101
+---
+apiVersion: cluster.x-k8s.io/v1beta2
+kind: MachineDeployment
+metadata:
+  annotations:
+    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
+    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "6"
+    # scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g-pv1-1
+    capacity.cluster-autoscaler.kubernetes.io/cpu: "4"                # numSockets 1 × numCores 4
+    capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi"           # memoryMiB 2048
+    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"  # bootVolume sizeGb 40
+    capacity.cluster-autoscaler.kubernetes.io/maxPods: "110"          # CA key; "maximum-pods" is ignored
+    # Labels for the from-zero template node. CA 1.35.2 reads MD labels from a path v1beta2 doesn't
+    # have, and CAPMOX publishes no nodeInfo, so without this os/arch are "" on the template.
+    capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
+  labels:
+    cluster.x-k8s.io/cluster-name: lab1
+  name: lab1-workers-pv1-1
+  namespace: default
+spec:
+  clusterName: lab1
+  # Delete an old worker before creating its replacement. The default surge (1/0) deadlocks on a full
+  # host: CAPMOX can't reserve memory for the surge VM, and CA won't scale down the old workers while
+  # the MD is rolling out (clusterapi ScaleDownNodeUpgradeProcessor).
+  rollout:
+    strategy:
+      type: RollingUpdate
+      rollingUpdate:
+        maxSurge: 0
+        maxUnavailable: 1
+  selector:
+    matchLabels: null
+  template:
+    metadata:
+      labels:
+        cluster.x-k8s.io/cluster-name: lab1
+        node-role.kubernetes.io/node: ""
+        # CA --balancing-label; node.cluster.x-k8s.io/* is synced to the Node by CAPI
+        node.cluster.x-k8s.io/pool: workers
+    spec:
+      bootstrap:
+        configRef:
+          apiGroup: bootstrap.cluster.x-k8s.io
+          kind: KubeadmConfigTemplate
+          name: lab1-worker
+      clusterName: lab1
+      infrastructureRef:
+        apiGroup: infrastructure.cluster.x-k8s.io
+        kind: ProxmoxMachineTemplate
+        name: lab1-worker-2g-pv1-1
+      version: v1.35.9
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
+kind: ProxmoxMachineTemplate
+metadata:
+  name: lab1-worker-2g-pv1-1
+  namespace: default
+spec:
+  template:
+    spec:
+      # Pinned like the pv1-2 template, independent of ProxmoxCluster.allowedNodes: template 101
+      # lives on pv1-1's local-lvm.
+      allowedNodes:
+      - pv1-1
+      disks:
+        bootVolume:
+          disk: scsi0
+          sizeGb: 40
+      full: true
+      memoryMiB: 2048
+      network:
+        networkDevices:
+        - bridge: vmbr0
+          model: virtio
+          name: net0
+          vlan: 4
+      numCores: 4
+      numSockets: 1
+      sourceNode: pv1-1
+      templateID: 101
+---
+# Worker group on pv1-2. Keep in sync with lab1-workers-pv1-1 / lab1-worker-2g-pv1-1 except node,
+# template and max-size: CA --balancing-label skips resource checks, so capacity must stay identical.
+apiVersion: cluster.x-k8s.io/v1beta2
+kind: MachineDeployment
+metadata:
+  annotations:
+    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
+    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12"
+    # scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g-pv1-2
+    capacity.cluster-autoscaler.kubernetes.io/cpu: "4"                # numSockets 1 × numCores 4
+    capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi"           # memoryMiB 2048
+    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"  # bootVolume sizeGb 40
+    capacity.cluster-autoscaler.kubernetes.io/maxPods: "110"          # CA key; "maximum-pods" is ignored
+    # Labels for the from-zero template node. CA 1.35.2 reads MD labels from a path v1beta2 doesn't
+    # have, and CAPMOX publishes no nodeInfo, so without this os/arch are "" on the template.
+    capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
+  labels:
+    cluster.x-k8s.io/cluster-name: lab1
+  name: lab1-workers-pv1-2
+  namespace: default
+spec:
+  clusterName: lab1
+  rollout:
+    strategy:
+      type: RollingUpdate
+      rollingUpdate:
+        maxSurge: 0
+        maxUnavailable: 1
+  selector:
+    matchLabels: null
+  template:
+    metadata:
+      labels:
+        cluster.x-k8s.io/cluster-name: lab1
+        node-role.kubernetes.io/node: ""
+        # CA --balancing-label; node.cluster.x-k8s.io/* is synced to the Node by CAPI
+        node.cluster.x-k8s.io/pool: workers
+    spec:
+      bootstrap:
+        configRef:
+          apiGroup: bootstrap.cluster.x-k8s.io
+          kind: KubeadmConfigTemplate
+          name: lab1-worker
+      clusterName: lab1
+      infrastructureRef:
+        apiGroup: infrastructure.cluster.x-k8s.io
+        kind: ProxmoxMachineTemplate
+        name: lab1-worker-2g-pv1-2
+      version: v1.35.9
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
+kind: ProxmoxMachineTemplate
+metadata:
+  name: lab1-worker-2g-pv1-2
+  namespace: default
+spec:
+  template:
+    spec:
+      # Overrides ProxmoxCluster.allowedNodes ([pv1-1], kept for the control plane): template 9101
+      # lives on pv1-2's local-lvm, and PVE refuses cross-node clones from local storage.
+      allowedNodes:
+      - pv1-2
+      disks:
+        bootVolume:
+          disk: scsi0
+          sizeGb: 40
+      full: true
+      memoryMiB: 2048
+      network:
+        networkDevices:
+        - bridge: vmbr0
+          model: virtio
+          name: net0
+          vlan: 4
+      numCores: 4
+      numSockets: 1
+      sourceNode: pv1-2
+      templateID: 9101
+---
+apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
+kind: KubeadmConfigTemplate
+metadata:
+  name: lab1-worker
+  namespace: default
+spec:
+  template:
+    spec:
+      joinConfiguration:
+        nodeRegistration:
+          kubeletExtraArgs:
+          - name: provider-id
+            value: proxmox://'{{ ds.meta_data.instance_id }}'
+      users:
+      - name: root
+        sshAuthorizedKeys:
+        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINQxxkqmvtVI+8c5BkTaJ5c2HfBFRXJWMmEcevvfP9tV
+          jan.novak@Jans-MacBook-Air.local
@@ -0,0 +1,8 @@
+# helm upgrade --install cilium cilium/cilium --version 1.20.2 -n kube-system -f values.yaml
+# into the workload cluster (lab1), not the management cluster
+ipam:
+  mode: kubernetes          # use the node podCIDRs from Cluster.spec.clusterNetwork (10.244.0.0/16), not 10.0.0.0/8
+operator:
+  replicas: 1               # default 2 with node anti-affinity would pin one to a worker and block scale-to-zero
+  nodeSelector:             # replicas: 1 alone still let the scheduler pick the worker; chart tolerates the CP taint
+    node-role.kubernetes.io/control-plane: ""
@@ -1,4 +1,4 @@
-# helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default -f values.yaml
+# helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler --version 9.59.0 -n default -f values.yaml
 # must be installed in the same namespace as the <cluster>-kubeconfig secret
 cloudProvider: clusterapi
 clusterAPIMode: kubeconfig-incluster      # workload via kubeconfig, management in-cluster
@@ -8,10 +8,35 @@ autoDiscovery:
  namespace: default
 image:
  tag: v1.35.2                            # chart default appVersion is 1.35.0
+rbac:
+  # CA reads the infrastructure machine template (capacity for scale-from-zero). The chart's
+  # ClusterRole only covers cluster.x-k8s.io; without this the first main loop blocks forever on
+  # "waiting for cache sync on infrastructure resource" and CA never scales.
+  additionalRules:
+    - apiGroups: [infrastructure.cluster.x-k8s.io]
+      resources: [proxmoxmachinetemplates]
+      verbs: [get, list, watch]
 extraArgs:
  expander: least-waste
-  scale-down-unneeded-time: 5m
-  scale-down-delay-after-add: 5m
-  scale-down-utilization-threshold: "0.5"
+  # Aggressive lab scale-down profile (CA 1.35 defaults in brackets): idle worker gone in ~3-4 min
+  # (scan-interval 1m adds up to a minute to the 1m unneeded time).
+  # Keep delay-after-add >= node startup (~2-3 min here) or bursts flap up/down.
+  scale-down-unneeded-time: 1m                # [10m]
+  scale-down-delay-after-add: 2m              # [10m] cluster-wide cooldown after any scale-up
+  scale-down-delay-after-failure: 1m          # [3m]
+  unremovable-node-recheck-timeout: 1m        # [5m]
+  scale-down-utilization-threshold: "0.6"     # [0.5]
+  max-graceful-termination-sec: "60"          # [600] drain wait per pod
  balance-similar-node-groups: true
-  v: 4
+  # One worker MD per PVE node. Compare groups only on this label: the generic comparator fails when
+  # one group has a real-node template (cached, 2Gi VM = ~1.78Gi allocatable, os/arch set) and the
+  # other a from-zero one. Disables the resource checks, so keep the MDs' capacity identical.
+  balancing-label: node.cluster.x-k8s.io/pool
+  # Fewer API requests instead of a higher kube-client-qps: every MD at 0 replicas costs ~3 uncached
+  # discovery GETs per NodeGroups() call (~15x per loop). Unschedulable pods and scale-ups/downs still
+  # trigger a loop immediately; only idle loops (scale-down evaluation) are spaced out.
+  scan-interval: 1m                           # [10s]
+  # Cilium taints new nodes NoSchedule until its agent is up. Without this, CA simulates the freshly
+  # registered (still tainted) node, finds the pending pods don't fit and adds one node too many.
+  startup-taint: node.cilium.io/agent-not-ready
+  v: 2                                        # [chart 4] 4 logs every throttled request and MD lookup
@@ -1,4 +1,6 @@
-# apply to the workload cluster; scale replicas to trigger scale-up (≈2 pods per 4 vCPU / 8 GiB worker)
+# apply to the workload cluster; scale replicas to trigger scale-up
+# 2 pods per 4 vCPU / 2 GiB worker: CPU-bound (3 × 1500m > 4000m); 512Mi keeps memory from binding
+# first (a 2 GiB VM has ~1.78 GiB allocatable; measured 3.73 GiB on a 4 GiB VM)
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -15,4 +17,4 @@ spec:
        - name: pause
          image: registry.k8s.io/pause:3.10
          resources:
-            requests: { cpu: "1500m", memory: "2Gi" }
+            requests: { cpu: "1500m", memory: "512Mi" }
@@ -15,7 +15,7 @@ image-builder fixes log; their dates are when the decision was recorded, not nec
 | 4 | 2026-09-24 | image-builder overrides go in a `PACKER_VAR_FILES` JSON | `PACKER_FLAGS` | flags were silently lost when `PACKER_FLAGS` was re-exported ([fixes log #4](image-builder-proxmox.md#fixes-log)) | active |
 | 5 | 2026-09-24 | Cilium installed by Helm with `ipam.mode=kubernetes`, pod CIDR `10.244.0.0/16` | ClusterResourceSet CNI flavors; pod CIDR `192.168.0.0/16` | keeps the CNI version under our control; `192.168.0.0/16` overlaps the lab LAN | active |
 | 6 | 2026-09-24 | Scale-from-zero via `capacity.cluster-autoscaler.kubernetes.io/*` annotations on the MachineDeployment | infrastructure provider publishing `status.capacity` | CAPMOX does not publish `status.capacity` on `ProxmoxMachineTemplate` | active |
-| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | active |
+| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | superseded by #18 |
 | 8 | 2026-09-26 | Phase 2: spike CAPLV first; own provider if insufficient; KubeVirt as fallback ([phase-2](phase-2-kvm-provider.md#suggested-approach)) | CAPK + KubeVirt, Metal3, own provider from the start | CAPLV shares CAPI 1.12 with CAPMOX and fits plain libvirt hosts directly | proposed |
 | 9 | 2026-09-26 | Phases 2 and 3 are independent and can be done in either order | strict sequence | no technical dependency between the KVM provider and hosted control planes | active |
 | 10 | 2026-09-26 | Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog ([session](sessions/2026-09-26-2355-history-conventions.md)) | chat history only; raw transcripts committed | research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets | active |
@@ -23,3 +23,8 @@ image-builder fixes log; their dates are when the decision was recorded, not nec
 | 12 | 2026-09-27 | Management cluster built by one idempotent Ansible playbook, `ansible/mgmt.yml`, run from the laptop ([plan](plans/2026-09-27-0058-mgmt-cluster-ansible.md)) | cloud-init + k3s auto-deploy manifests; bash scripts with guards; OpenTofu + Helm | owner's choice; idempotent modules, one tool; only `ansible.builtin`, charts through k3s's helm-controller so nothing extra on the VM | active |
 | 13 | 2026-09-27 | CAPI providers installed by cluster-api-operator 0.28.0 (Provider CRs), cert-manager v1.21.2 pre-installed | guarded `clusterctl init` | declarative; re-apply is a no-op; upgrade = `spec.version` bump; 0.28.0 built against CAPI 1.12.10. Consequence: no `clusterctl init/upgrade/move/delete` | active |
 | 14 | 2026-09-27 | Management VM created by the playbook from the pinned Ubuntu 24.04 cloud image (VM 201, 192.168.4.21) | hand-made VM; the image-builder node template | reproducible from the repo; the image-builder template is a kubeadm node image, not meant for k3s | active |
+| 15 | 2026-09-27 | `lab1` workers 4 vCPU / **4 GiB** (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker | 8 GiB workers with 2Gi inflate pods | owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape ([configs/capi/lab1.yaml](../configs/capi/lab1.yaml)) | superseded by #16 |
+| 16 | 2026-09-28 | `lab1` workers 4 vCPU / **2 GiB** (new `ProxmoxMachineTemplate/lab1-worker-2g`, rolled in via the MD); inflate pods 1500m / **512Mi**, still CPU-bound at 2 per worker | keep 4 GiB and delete stopped VMs; CAPMOX `memoryAdjustment` overcommit | CAPMOX reserves the configured memory of every VM, stopped ones included, so 4 GiB workers ran out after the first scale-up ([troubleshooting](phase-1-proxmox-capi.md#troubleshooting)); 2 GiB (~1.78 GiB allocatable) still leaves room for kubelet/containerd plus real pods, and max 5 workers now fits the host | proposed |
+| 17 | 2026-09-28 | CAPMOX memory overcommit on `lab1`: `ProxmoxCluster.spec.schedulerHints.memoryAdjustment: 160` (46.7 GiB reservable on the 29.16 GiB `pv1-1`); MD rollout `maxSurge 0 / maxUnavailable 1` | 100% (default); 120% (minimum sizing for max 5 × 2 GiB workers); 0 (memory check off); deleting stopped VMs | CAPMOX counts the configured memory of stopped VMs (12 GiB here), and no release can use real free memory; 160% keeps a bound, and host-side protection is KSM + swap, not ballooning ([session](sessions/2026-09-28-1007-capmox-memory-overcommit.md)) | proposed |
+| 18 | 2026-09-30 | PVE is now a 2-node cluster (`pv1-1` + `pv1-2`), no shared storage: a **template copy per node** (101 on `pv1-1`, 9101 on `pv1-2`, copied by clone + offline migrate), **one worker MD per node** (`lab1-workers-pv1-1` max 6, `lab1-workers-pv1-2` max 12, PMT `allowedNodes: [<node>]`), control plane stays on `pv1-1`, node IP pool `.31–.50` | shared storage (NFS); a second image-builder run on `pv1-2`; `templateSelector` with the same tags on both nodes; keep `lab1-workers` and only add a second MD | PVE refuses cross-node clones from local storage and CAPMOX v0.9.1 always passes the scheduled node as target; `templateSelector` needs exactly one match cluster-wide; a copy is byte-identical; owner wants no shared storage for now and symmetric names ([plan](plans/2026-09-29-2315-pv1-2-workers.md), [session](sessions/2026-09-29-2311-pve-two-node-cluster.md)) | active |
+| 19 | 2026-09-30 | cluster-autoscaler with two node groups: `balancing-label: node.cluster.x-k8s.io/pool` (label on both MDs + `capacity…/labels` annotation), `scan-interval: 1m`, `v: 2`; client QPS left at the default 5/10 | generic similarity comparator; `balancing-ignore-label` for `deployment-name`; `kube-client-qps 50 / burst 100`; `min-size 1` per group; `CAPI_VERSION=v1beta1` | the comparator fails between a cached real-node template and a from-zero template (6+2 instead of 4+4); CA 1.35.2 runs uncached discovery per zero-replica MD ~15× per loop, and the owner prefers fewer requests over a higher rate — pending pods still trigger loops immediately ([execution log](plans-executions/2026-09-29-2315-pv1-2-workers.md)) | active |
@@ -0,0 +1,79 @@
+# Future tasks
+
+Deferred work, newest first. When a task is picked up it gets a plan in `docs/plans/`; then tick
+it here and link the plan.
+
+## - [ ] Report three cluster-autoscaler clusterapi-provider bugs upstream
+
+**Added:** 2026-09-30 09:28 CEST · **Why deferred:** worked around in config (decision #19);
+reporting is not blocking.
+
+**Problem.** Three bugs in CA 1.35.2 `cloudprovider/clusterapi`, all still on master and 1.36.1 as
+of 2026-09-29, found in [the pv1-2 execution log](plans-executions/2026-09-29-2315-pv1-2-workers.md):
+1. **Uncached discovery.**
+   - `clusterapi_provider.go:195` builds a `memory.NewMemCacheClient`, but passes the raw
+     discovery client to the controller (`:209`).
+   - `getKindPreferredVersion` therefore does 3 GETs per MD at 0 replicas on every `NodeGroups()`
+     call, about 15× per loop. The loop took 18 s at 5 QPS.
+2. **`Labels()` reads the wrong path.** It reads `spec.template.spec.metadata.labels`
+   (`clusterapi_unstructured.go:207`), which doesn't exist in CAPI v1beta2, so MD template labels
+   never reach the from-zero template node. The unit tests use the same wrong path.
+3. **Empty os/arch labels.** For an infra template without `status`, as with CAPMOX,
+   `InstanceSystemInfo()` returns an empty struct. The template node then gets
+   `kubernetes.io/os=""` and `arch=""`, which override the generic `linux`/`amd64`.
+
+**Options:**
+- Issues only.
+- Issues plus small PRs: pass the cached client, fix the path, skip empty nodeInfo fields.
+- Carry a patched CA image: rejected, versions must stay upstream and pinned.
+
+**Cost.** About 1–2 h for issues with source references. PRs would add the CLA and a test for each.
+
+**Changes when done:**
+- Link the issues/PRs in `docs/phase-1-proxmox-capi.md` troubleshooting.
+- Once fixed in a 1.35.x patch release: bump CA in `docs/versions.md`, then revisit
+  `scan-interval` and the `capacity…/labels` annotation.
+
+**Run by the owner:** nothing on the lab. Filing on GitHub is the owner's call, since it goes out
+under their account.
+
+## - [ ] Namespace per workload cluster (`lab1` out of `default`)
+
+**Added:** 2026-09-28 09:29 CEST · **Why deferred:** keep phase 1 moving; `default` works for now.
+
+**Problem.** cluster-autoscaler has to run in the same namespace as the Cluster. The chart mounts
+`clusterAPIKubeconfigSecret` as a Secret volume, which only works within the pod's namespace, and
+`lab1-kubeconfig` lives in the Cluster's namespace. `lab1` is in `default`, so CA would be installed
+into `default` too. That mixes things in `default`, and with more clusters (phase 3) you'd get one
+CA per cluster sitting side by side in `default`.
+
+**Options compared:**
+
+| Option | Verdict |
+| --- | --- |
+| CA in `default` (current Step 6) | works; poor hygiene, doesn't scale to more clusters |
+| CA in its own namespace with a copy of the kubeconfig Secret | rejected: CAPI rewrites `lab1-kubeconfig` when it rotates the client cert, so the copy goes stale and CA silently loses the workload cluster |
+| **One namespace per cluster** (`lab1` holds the CAPI objects, `InClusterIPPool`, kubeconfig Secret and CA) | preferred: the usual CAPI multi-cluster layout, one CA per workload cluster, RBAC and cleanup per namespace, matches phase 3 |
+
+**Cost.** Namespace is immutable on CAPI objects, so `lab1` has to be deleted and recreated. That
+means about 15 minutes of provisioning plus reinstalling Cilium. `clusterctl move` is not an option,
+since it's forbidden with cluster-api-operator. CAPMOX credentials come from the controller's
+global Secret, so a new namespace needs no extra credentials. The cost grows once workloads and
+autoscaler state exist on `lab1`, so do it before the Step 7 results matter, or when creating the
+next cluster.
+
+**Changes:**
+
+- `configs/capi/lab1.yaml`: a `Namespace lab1` object and `metadata.namespace: lab1` on every
+  object, including `InClusterIPPool`.
+- `configs/cluster-autoscaler/values.yaml`: `autoDiscovery.namespace: lab1`; install with `-n lab1`
+  (`helm uninstall ca -n default` first if it was installed in `default`).
+- `ansible/group_vars/all.yml`: `inspector_workload_clusters` entry `{ name: lab1, namespace: lab1 }`.
+- `docs/phase-1-proxmox-capi.md`: Steps 3, 5 and 6 (namespace in commands and log expectations,
+  `MachineDeployment/lab1/lab1-workers-pv1-{1,2}`).
+- `.claude/agents/cluster-inspector.md`: the "Where things live" table says CA runs in `default`.
+- `docs/decisions.md`: new row (namespace per workload cluster).
+
+**Run by the owner:** `kubectl delete -f configs/capi/lab1.yaml` (old namespace), wait until the
+VMs are gone, `kubectl apply` the new manifest, Cilium `helm upgrade --install`,
+`ansible-playbook mgmt.yml --tags inspector`, then the CA install into `lab1`.
@@ -2,7 +2,8 @@

 Result: Proxmox template `ubuntu-2404-kube-v1.35.9` on node `pv1-1` — Ubuntu 24.04.4, containerd,
 kubeadm/kubelet/kubectl v1.35.9, pre-pulled control-plane images, cloud-init, qemu-guest-agent.
-Its VMID is `TEMPLATE_VMID` for CAPMOX.
+Its VMID (101) is `TEMPLATE_VMID` for CAPMOX. A byte-identical copy, 9101, sits on `pv1-2`
+([Template on other PVE nodes](#template-on-other-pve-nodes)).

 ## Build host prerequisites (Ubuntu 24.04)

@@ -71,8 +72,13 @@ Notes carried into CAPMOX (phase 1): same ACL set, `vlan: 4` on the network devi
 The template has no usable login (`builder` is locked) — access only via cloud-init (NoCloud,
 which the PVE Cloud-Init drive provides):

+Run it on the node that holds the template. VMIDs are cluster-wide, so pick a free `ID`: 9001 is
+now `debian-13-template`, and the last line destroys `$ID`. The key file must exist on that node
+(`scp ~/.ssh/id_ed25519.pub root@<node-ip>:/root/jan.pub`).
+
 ```bash
-TPL=<template vmid>; ID=9001
+TPL=<template vmid>; ID=9901
+qm status $ID 2>/dev/null && echo "VM $ID exists, pick another ID"   # must print nothing
 qm clone $TPL $ID --name smoke-k8s --full 1 --storage local-lvm
 qm set $ID --ide2 local-lvm:cloudinit
 qm set $ID --ciuser jan --sshkeys /root/jan.pub --ipconfig0 ip=dhcp
@@ -83,3 +89,28 @@ qm stop $ID && qm destroy $ID --purge
 ```

 kubelet crash-looping before `kubeadm init/join` is expected.
+
+## Template on other PVE nodes
+
+Without shared storage, a template on `local-lvm` can only be cloned on its own node. That holds
+for PVE and for CAPMOX: `can't clone VM to node '<target>' (VM uses local storage)`. Each additional
+PVE node therefore gets its own copy under its own VMID, since VMIDs are cluster-wide.
+
+Copy rather than rebuild: the copy is byte-identical, whereas a second Packer run would pick up
+newer apt packages. The source must have no disk or CD-ROM on non-shared storage other than
+`local-lvm`; check with `qm config 101`.
+
+```bash
+# on the node holding the template (pv1-1); 9101 = copy for pv1-2
+qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm  # clone stays on pv1-1
+qm migrate 9101 pv1-2 --with-local-disks                                        # offline: copies the LVs
+ssh root@192.168.0.10 qm template 9101
+```
+
+Then run the smoke test on the target node with `TPL=9101`. It also proves that VLAN 4 works on
+that node's `vmbr0`. The copy is referenced by `templateID: 9101` + `sourceNode: pv1-2` in that
+node's `ProxmoxMachineTemplate`
+([phase 1, Step 0](phase-1-proxmox-capi.md#step-0--proxmox-prep)).
+
+A new template build (a new Kubernetes version) means building on `pv1-1` and then repeating the
+copy for every other node, with a new VMID pair per version.
@@ -1,7 +1,8 @@
 # Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler

 Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload
-cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
+cluster `lab1` as Proxmox VMs on a two-node PVE cluster; cluster-autoscaler scales its workers 0–18
+in one node group per PVE node, balanced, including scale-from-zero.
 No custom code in this phase — only assembling upstream components.

 Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md).
@@ -10,18 +11,20 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro

 | Role | Where | Size |
 | --- | --- | --- |
+| PVE cluster `homelab` | `pv1-1` 192.168.0.8 (API endpoint for CAPMOX), `pv1-2` 192.168.0.10; `local-lvm` on each, no shared storage | 16 / 12 cores, 29.2 / 40.9 GiB |
+| Node template | `ubuntu-2404-kube-v1.35.9`: VMID 101 on `pv1-1`, copy 9101 on `pv1-2` | 20 GB |
 | Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
-| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
-| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 8 GB RAM |
+| `lab1` control plane | 1 VM on `pv1-1`, VIP `.30` via kube-vip | 2 vCPU, 4 GB RAM |
+| `lab1` workers | MachineDeployments `lab1-workers-pv1-1` (min 0 / max 6) and `lab1-workers-pv1-2` (min 0 / max 12); node IP pool `.31–.50` | 4 vCPU, 2 GB RAM |

 ## Checklist

 - [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
 - [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
 - [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers)
- [ ] Step 3 — workload cluster `lab1`
- [ ] Step 4 — CNI (Cilium)
- [ ] Step 5 — MachineDeployment autoscaling annotations
+- [x] Step 3 — workload cluster `lab1` (VMs 102/103, `.31`/`.32`, VIP `.30`)
+- [x] Step 4 — CNI (Cilium 1.20.2, `configs/cilium/values.yaml`)
+- [x] Step 5 — MachineDeployment autoscaling annotations
 - [ ] Step 6 — cluster-autoscaler
 - [ ] Step 7 — scale tests

@@ -39,11 +42,24 @@ pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser        #

 Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`.

-Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP
-for the control plane endpoint in the same subnet.
+The token and ACLs are cluster-wide in a PVE cluster, so a node that joins later needs nothing here.

-Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node
-requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template.
+Network: `vmbr0` is VLAN-aware on **every** PVE node, and VMs live in VLAN 4. On a new node, add
+`bridge-vlan-aware yes` + `bridge-vids 2-10` (or wider) to `iface vmbr0`, run `ifreload -a`, and
+trunk VLAN 4 on its switch port. CAPMOX writes `bridge`/`vlan` into the VM without checking them.
+Reserve a node IP range outside DHCP and one VIP for the control plane endpoint in the same subnet.
+
+Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2.
+- PVE refuses to clone a VM to another node from non-shared storage, and CAPMOX v0.9.1 always
+  passes the scheduled node as the clone target.
+- `templateSelector` needs exactly one match in the whole PVE cluster, so it can't pick a
+  per-node copy.
+- Without shared storage, therefore, every node gets its own template VMID (VMIDs are
+  cluster-wide) and its own worker MachineDeployment. Each MD's `ProxmoxMachineTemplate` sets
+  `sourceNode`, `templateID` and `allowedNodes: [<node>]`.
+- `ProxmoxCluster.allowedNodes` stays `[pv1-1]` for the control plane. The copy procedure is in
+  [image-builder-proxmox.md](image-builder-proxmox.md#template-on-other-pve-nodes), and the
+  reasoning in [decision #18](decisions.md).

 ## Step 2 — management cluster

@@ -71,7 +87,8 @@ ansible-playbook mgmt.yml --tags vm           # VM 201 on pv1-1, waits for SSH
 ansible-playbook mgmt.yml --tags k3s          # qemu-guest-agent, k3s, node Ready
 ansible-playbook mgmt.yml --tags addons       # cert-manager v1.21.2, cluster-api-operator 0.28.0
 ansible-playbook mgmt.yml --tags capi         # capmox-variables Secret + 5 Provider CRs, waits for controllers
-ansible-playbook mgmt.yml --tags kubeconfig   # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfig
+ansible-playbook mgmt.yml --tags kubeconfig   # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfigs
+ansible-playbook mgmt.yml --tags inspector    # only the read-only kubeconfigs (lab-mgmt + inspector_workload_clusters); weekly token refresh
 ansible-playbook mgmt.yml                     # everything; must be changed=0 on a built cluster
 ```

@@ -94,15 +111,11 @@ cluster-api-operator replaces `clusterctl init`:

 ## Step 3 — workload cluster `lab1`

-```bash
-clusterctl generate cluster lab1 \
-  --infrastructure proxmox:v0.9.1 \
-  --kubernetes-version v1.35.9 \
-  --control-plane-machine-count 1 \
-  --worker-machine-count 1 > lab1.yaml
-```
-
-Edit `lab1.yaml` before applying:
+The generated and hand-edited manifest is committed as
+[configs/capi/lab1.yaml](../configs/capi/lab1.yaml): template VMID 101, control-plane VIP
+`192.168.4.30`, node IPs `192.168.4.31–39`. Its header holds the exact `clusterctl generate`
+command (all variables as env vars, so no `~/.cluster-api/clusterctl.yaml` is needed). Regenerate
+only when a version changes, then redo these edits:

 1. **Pod CIDR** — `Cluster.spec.clusterNetwork.pods.cidrBlocks` defaults to `192.168.0.0/16`, which
   overlaps the lab network → use e.g. `10.244.0.0/16`.
@@ -110,47 +123,93 @@ Edit `lab1.yaml` before applying:
   (handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
 3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
 4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
-5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 8192`.
+5. **Worker size** — rename the worker template to `ProxmoxMachineTemplate/lab1-worker-2g` (and the MD
+   `infrastructureRef`): `numSockets: 1`, `numCores: 4`, `memoryMiB: 2048`. To resize later, add a
+   template with a new name and point the MD at it; CAPI then rolls the workers. Don't edit a
+   template in place.
 6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.

 ```bash
-kubectl apply -f lab1.yaml
+export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
+kubectl apply -f configs/capi/lab1.yaml
 clusterctl describe cluster lab1 --show-conditions all
 kubectl get proxmoxmachines,machines -w
 clusterctl get kubeconfig lab1 > lab1.kubeconfig
 ```

-Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses`). providerID is set by
+Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io`;
+plain `ipaddresses` resolves to the built-in `networking.k8s.io` type on k8s ≥ 1.33). providerID is set by
 the CAPMOX template as `proxmox://{{ ds.meta_data.instance_id }}` — don't touch it, the autoscaler
 matches Nodes to Machines by it.

 ## Step 4 — Cilium

+Values in [configs/cilium/values.yaml](../configs/cilium/values.yaml); `upgrade --install` is
+safe to re-run.
+
 ```bash
-export KUBECONFIG=$PWD/lab1.kubeconfig
-helm repo add cilium https://helm.cilium.io && helm repo update
-helm install cilium cilium/cilium --version 1.20.2 -n kube-system --set ipam.mode=kubernetes
+clusterctl get kubeconfig lab1 --kubeconfig ~/.kube/lab-mgmt.kubeconfig > ~/.kube/lab1.kubeconfig
+chmod 600 ~/.kube/lab1.kubeconfig
+export KUBECONFIG=~/.kube/lab1.kubeconfig
+
+helm repo add cilium https://helm.cilium.io && helm repo update cilium
+helm upgrade --install cilium cilium/cilium --version 1.20.2 -n kube-system \
+  -f configs/cilium/values.yaml --wait --timeout 10m
+
+kubectl get nodes                                          # both Ready
+kubectl -n kube-system get pods -o wide                    # cilium DaemonSet 2/2, operator 1/1, coredns Running
 ```

-`ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default.
+- `ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default.
+- `operator.replicas: 1` + `operator.nodeSelector` on `node-role.kubernetes.io/control-plane`: the
+  default 2 replicas with node anti-affinity put one on a worker, and a kube-system pod without a
+  PDB blocks scale-down to zero. `replicas: 1` alone isn't enough: the single replica was scheduled
+  onto the worker (the chart tolerates the control-plane taint, so the selector pins it there).
+- kube-proxy stays (kubeadm default); no kube-proxy replacement in this phase.

 ## Step 5 — autoscaling annotations

+Both worker MDs carry the same block. Only the name and max-size differ (6 on `pv1-1`, 12 on
+`pv1-2`):
+
 ```yaml
 metadata:
-  name: lab1-workers
+  name: lab1-workers-pv1-2
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
-    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
+    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "4"            # numSockets × numCores
-    capacity.cluster-autoscaler.kubernetes.io/memory: "8Gi"
+    capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi"       # memoryMiB 2048
    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
-    capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
+    capacity.cluster-autoscaler.kubernetes.io/maxPods: "110"      # not "maximum-pods" (ignored)
+    capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
+spec:
+  template:
+    metadata:
+      labels:
+        node.cluster.x-k8s.io/pool: workers    # synced to the Node by CAPI; CA --balancing-label
 ```

-Capacity annotations are required for scale-from-zero because CAPMOX does not publish
-`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints`
-annotations if pods target the group by selector/toleration.
+They are part of both MDs in [configs/capi/lab1.yaml](../configs/capi/lab1.yaml). Re-apply the file.
+`replicas` isn't in the manifest, so it stays as it is, and a new MD starts at the min-size:
+
+```bash
+export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
+kubectl diff -f configs/capi/lab1.yaml
+kubectl apply -f configs/capi/lab1.yaml
+```
+
+Scale-from-zero needs capacity annotations because CAPMOX does not publish `status.capacity`, and
+they must match the `ProxmoxMachineTemplate`. The `.../labels` annotation is needed as well, for two
+reasons in CA 1.35.2:
+- It reads MD template labels from `spec.template.spec.metadata.labels`, a path that doesn't exist in
+  CAPI v1beta2.
+- CAPMOX publishes no `status.nodeInfo`, so the from-zero template node would otherwise get
+  `kubernetes.io/os=""` and `arch=""`. A pod with an `os: linux` selector then can't trigger a
+  scale-up.
+
+Add a `.../taints` annotation if pods target the group by toleration. None of this has any effect
+until cluster-autoscaler runs (Step 6).

 ## Step 6 — cluster-autoscaler

@@ -158,19 +217,47 @@ Install into the **same namespace as the `lab1-kubeconfig` secret** (here `defau
 [configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):

 ```bash
-helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update
-helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \
-  -f configs/cluster-autoscaler/values.yaml
+export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
+helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler \
+  --version 9.59.0 -n default -f configs/cluster-autoscaler/values.yaml
 ```

-The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5.
+The values add `rbac.additionalRules` to read `proxmoxmachinetemplates`. The chart's ClusterRole
+doesn't cover infrastructure templates, and without them the first main loop hangs (see
+troubleshooting). Health check: the log shows `Starting main loop` followed by regular
+`Autoscaler loop triggered` lines, and `cluster-autoscaler-status` on lab1 leaves `Initializing`.
+
+CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`.
+
+**Settings in the values that matter with two node groups:**
+- `balancing-label: node.cluster.x-k8s.io/pool`. CA compares groups only on this label, so a
+  scale-up is split across both PVE nodes.
+- `scan-interval: 1m`. Unschedulable pods and finished scale-ups/downs still start a loop
+  immediately; only the idle scale-down evaluation runs less often.
+- `v: 2`.
+
+**Checking the node groups:** `discovered node group` is only logged at `v: 4`
+(`clusterapi_controller.go:800`). At `v: 2`, check the node groups in the status ConfigMap (below)
+instead. A balanced scale-up logs:
+
+```text
+Splitting scale-up between 2 similar node groups: {MachineDeployment/default/lab1-workers-pv1-2, MachineDeployment/default/lab1-workers-pv1-1}
+Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}]
+```
+
+The status ConfigMap and the leader-election lock live on the workload cluster, not on the management
+cluster: `kubectl get cm cluster-autoscaler-status -n kube-system --kubeconfig ~/.kube/lab1.kubeconfig`
+(chart arg `--namespace=kube-system`, because `clusterAPIConfigMapsNamespace` is empty).

 ## Step 7 — scale tests

-Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 2Gi each).
+Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 512Mi each →
+CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocatable).

 - [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear.
+- [ ] Scale-down: `--replicas=0` → after `scale-down-delay-after-add` (2m) + `scale-down-unneeded-time`
+  (1m) Machines and VMs disappear. That's the aggressive lab profile in the CA values; per-MD overrides are
+  `cluster.x-k8s.io/autoscaling-options-{scaledownunneededtime,scaledownutilizationthreshold,…}` annotations.
 - [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
 - [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.

@@ -181,10 +268,21 @@ Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods,
 | Symptom | Cause | Fix |
 | --- | --- | --- |
 | CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
+| CA runs and discovers the node group but never scales; log stops at `waiting for cache sync on infrastructure resource`, `proxmoxmachinetemplates … forbidden` every ~40 s; status ConfigMap stuck on `Initializing` | chart 9.59.0 ClusterRole has no rule for `infrastructure.cluster.x-k8s.io`; CA's first main loop waits for a template informer that can never sync | `rbac.additionalRules` with get/list/watch on `proxmoxmachinetemplates` in the CA values, `helm upgrade` |
 | CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
 | MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
+| scale-up adds one node too many (6 pods at 2 per node → 4 workers, second plan `3->4` about 70 s after `1->3`) | a new node registers with Cilium's `node.cilium.io/agent-not-ready:NoSchedule`; once registered it's no longer an "upcoming" template node, and CA sees pending pods that don't fit on it | `extraArgs.startup-taint: node.cilium.io/agent-not-ready` in the CA values (CA treats the node as still starting) |
+| CA scales the MD up, new Machines stay without a VM; ProxmoxMachine `VMProvisionFailed: cannot reserve 4294967296B of memory on node pv1-1: …B available memory left` although `free` shows GBs available | CAPMOX v0.9.1 scheduler (`GetReservableMemoryBytes`): node total × `schedulerHints.memoryAdjustment` (default 100%) minus the **configured** memory of every non-template VM, **running or stopped**, and every container. Stopped VMs 100/1001/2001 hold 12 GiB | free the reservation (delete unused stopped VMs, or convert them to templates), or set `ProxmoxCluster.spec.schedulerHints.memoryAdjustment` > 100 (overcommit); keep the MD max-size within what the host can actually run |
+| MD rollout never finishes and CA never scales down: CA logs `Node … will be skipped as it is currently under rollout`, new ProxmoxMachine `cannot reserve …B of memory` | deadlock. Default rollout surge (`maxSurge 1 / maxUnavailable 0`) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (`ScaleDownNodeUpgradeProcessor`) | `spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1}` on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory |
 | last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
-| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
+| `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` |
+| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maxPods (`maximum-pods` is not a CA key and is silently ignored) |
+| a pod with `nodeSelector: kubernetes.io/os: linux` doesn't trigger scale-from-zero | CA 1.35.2 builds the template node's os/arch labels from the infra template's `status.nodeInfo`. CAPMOX has none, so they become `""`. MD template labels aren't read either (CA reads the non-existent v1beta2 path `spec.template.spec.metadata.labels`) | `capacity.cluster-autoscaler.kubernetes.io/labels: "kubernetes.io/os=linux,kubernetes.io/arch=amd64,…"` on the MD |
+| scale-up goes into one group only: `No similar node groups found`, next loop the second group (6 + 2 instead of 4 + 4) | the generic comparator fails when one group's template comes from a real node (cached for `--node-info-cache-expire-time`, default 10 years: ~1.78 Gi allocatable, os/arch set) and the other from the synthetic from-zero template (2 Gi, os/arch empty). The mismatch reason is only logged at V(5). `cluster.x-k8s.io/deployment-name` is **not** the cause, because CA never reads it | same label on both MDs (`node.cluster.x-k8s.io/pool`: template label + capacity labels annotation) and `balancing-label: node.cluster.x-k8s.io/pool` in the CA values. This turns off the resource comparison, so keep the MDs' capacity identical |
+| CA loop takes 8–26 s; the log is full of `Waited before sending request … client-side throttling` on `/api`, `/apis`, `/apis/infrastructure.cluster.x-k8s.io/v1alpha2` | the CA 1.35.2 clusterapi provider runs **uncached** discovery (`getKindPreferredVersion`) for every MD at 0 replicas, on every `NodeGroups()` call (about 15× per loop). The client-go default is 5 QPS / burst 10. There is no upstream fix up to 1.36.1 | fewer loops: `scan-interval: 1m` (scale-up is still triggered by pending pods at once). Alternatively `kube-client-qps`/`-burst` (more requests, not chosen). `v: 2` hides the noise. Upstream fix: [future tasks](future-tasks.md) |
+| `W clusterstate.go:510 Failed to find acceptable ranges for MachineDeployment/…` | the MD was created during a loop: acceptable ranges are snapshotted at the start of the loop, and the group showed up in a later `NodeGroups()` call | none if it happens once. If it repeats every loop, the group is dropping out of `NodeGroups()` (scale-from-zero check failing: infra template unreadable, capacity annotations missing) |
+| ProxmoxMachine `VMProvisionFailed` … `can't clone VM to node '<node>' (VM uses local storage)` | the scheduled node (`allowedNodes`) isn't the node holding the template, and the template is on non-shared storage. A cross-node clone is then impossible, whatever `full`/`storage` are set to | per-node template copy + per-node MD with `allowedNodes: [<node>]` (see Step 0); or shared storage |
+| VMs on a new PVE node get no IP (no DHCP/IPAM connectivity on VLAN 4) | `vmbr0` on that node isn't VLAN-aware, or its switch port doesn't trunk VLAN 4 | `bridge-vlan-aware yes` + `bridge-vids` in `/etc/network/interfaces`, `ifreload -a`; check via `pvesh get /nodes/<node>/network/vmbr0` |
 | clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
 | ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
 | `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
@@ -0,0 +1,222 @@
+# Execution log: lab1 workers on both PVE nodes (template copy per node)
+
+Pairs with [docs/plans/2026-09-29-2315-pv1-2-workers.md](../plans/2026-09-29-2315-pv1-2-workers.md).
+
+## Status
+
+- [x] Step 0 — Record the plan
+- [x] Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner)
+- [x] Step 2 — Template 9101 on `pv1-2` (owner)
+- [x] Step 3 — Second worker MachineDeployment `lab1-workers-pv1-2`
+- [x] Step 4 — Scale test across both nodes
+- [x] Step 5 — Docs and history
+
+## Step 0 — Record the plan (2026-09-29 23:15 CEST)
+
+Plan and this log committed on their own, together with the
+[session note](../sessions/2026-09-29-2311-pve-two-node-cluster.md) that motivated them.
+
+Starting state (read-only PVE API):
+- cluster `homelab`: `pv1-1` 192.168.0.8, `pv1-2` 192.168.0.10
+- `vmbr0` on `pv1-2` has no `bridge_vlan_aware`
+- template 101 is only on `pv1-1`, with disks `scsi0` + `efidisk0`, both on `local-lvm`
+- `nextid` is 103
+
+## Steps 1–2 — bridge and template on `pv1-2` (owner, verified 2026-09-30 08:38 CEST)
+
+The owner made `vmbr0` VLAN-aware on `pv1-2` (`/etc/network/interfaces`, then `ifreload -a`):
+
+```
+bridge-vlan-aware yes
+bridge-vids 2-10
+```
+
+This deviates from the plan: `bridge-vids` is `2-10`, not `2-4094`. That's enough for VLAN 4, and
+any VLAN above 10 on `pv1-2` would need the range widened first.
+
+Template copy, run on `pv1-1`. A cross-node clone is refused from `local-lvm`, so the copy is a
+clone on the same node followed by an offline migration:
+
+```bash
+qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm
+qm migrate 9101 pv1-2 --with-local-disks
+ssh root@192.168.0.10 qm template 9101
+```
+
+The PVE API shows 9101 on `pv1-2` with `template: 1`, and `scsi0`/`efidisk0` as
+`local-lvm:base-9101-disk-*`.
+
+Smoke test ([image-builder-proxmox.md](../image-builder-proxmox.md#smoke-test-of-the-template)) on
+`pv1-2`, with `TPL=9101 ID=9901`: the VM got a VLAN 4 DHCP address through the guest agent, and was
+destroyed afterwards. Before running it:
+- `/root/jan.pub` had to be copied to `pv1-2` first: `scp ~/.ssh/id_ed25519.pub root@192.168.0.10:/root/jan.pub`.
+- The doc's `ID=9001` now collides with `debian-13-template`, and the snippet ends with
+  `qm destroy $ID`. The snippet now uses 9901 and checks that the ID is free (`a9f5879`).
+
+Worth noting:
+- VMIDs are cluster-wide (pmxcfs), which is why the copy needs its own ID. Names don't have to be
+  unique.
+- The copied `efidisk0` reports `size=528K` (the source says 4M). This doesn't matter: the template
+  boots SeaBIOS.
+
+## Step 3 — second worker MachineDeployment (prepared 2026-09-30, apply pending)
+
+Changes in `configs/capi/lab1.yaml`:
+- new `MachineDeployment/lab1-workers-pv1-2` and `ProxmoxMachineTemplate/lab1-worker-2g-pv1-2`
+  (`sourceNode: pv1-2`, `templateID: 9101`, `allowedNodes: [pv1-2]`)
+- `ProxmoxCluster.allowedNodes` stays `[pv1-1]` for the control plane
+- header hand edits #10 and #11
+
+Deviation from the plan: the IP pool is enlarged and the max sizes are raised (owner's request,
+"20 nodes"):
+- `ipv4Config.addresses` `192.168.4.31-192.168.4.39` → `192.168.4.31-192.168.4.50`
+- `lab1-workers` max 5 → **6**, `lab1-workers-pv1-2` max **12**
+- 6 + 12 + 1 CP = 19 of 20 addresses. The spare covers the KCP rollout surge, and CAPMOX pools
+  hold a freed address for 60 s (`AddressReuseGracePeriodSeconds`, `pkg/kubernetes/ipam/ipam.go:318`).
+- An ARP probe from `lab-mgmt` found `.32–.60` silent; `.30` (VIP) and `.31` (CP) share the CP VM's
+  MAC.
+- The owner confirmed `.31–.50` is outside the VLAN 4 DHCP range.
+- Changing the pool is safe: the ProxmoxCluster webhook only validates the addresses, and the
+  controller copies them into the InClusterIPPool on every reconcile (`ipam.go:323-324`).
+
+Sizing:
+- `pv1-1`: 6 × 2 GiB within ~17.7 GiB of reservable headroom (160 %).
+- `pv1-2`: 12 × 2 GiB of 65.4 GiB reservable (40.9 GiB physical); 48 vCPU on 12 cores, which is
+  fine for the pause-container `inflate` tests but CPU-bound under real load.
+
+`yq` parse check: 9 documents; both worker PMTs point at the right node, template and `allowedNodes`.
+
+Applied by the owner (`kubectl diff` / `kubectl apply -f configs/capi/lab1.yaml`). The CA
+discovered `lab1-workers (max 6)` and `lab1-workers-pv1-2 (max 12)`.
+
+## Step 4 — scale test, first run (2026-09-30 ~06:45 UTC)
+
+The `inflate` Deployment had 16 pods (1500m / 512Mi each, 2 per worker).
+- **Result:** 8 workers Ready by 06:47Z.
+  - `lab1-workers` 0→6: VMIDs 103–108 on `pv1-1`, IPs .34–.39.
+  - `lab1-workers-pv1-2` 0→2: VMIDs 109–110 on `pv1-2`, IPs .32–.33.
+  - No `VMProvisionFailed`. `status.proxmoxNode` and providerIDs are consistent. The pool had 9 of
+    20 addresses in use.
+- **Checked with:** the `cluster-inspector` agent (read-only). Root causes came from the source
+  (`upstream-researcher`, CA tag `cluster-autoscaler-1.35.2`, CAPI v1.12.11, CAPMOX v0.9.1).
+
+Two problems.
+
+**1. Not balanced.** The log shows `Best option to resize: lab1-workers` → `No similar node groups
+found` → `0->6 (max: 6)`. Only in the next loop did `lab1-workers-pv1-2` get `0->2`.
+- The inspector guessed the `cluster.x-k8s.io/deployment-name` label. The source rules that out:
+  CA never reads it.
+- What the source does show in CA 1.35.2 `cloudprovider/clusterapi`:
+  - `Labels()` reads `spec.template.spec.metadata.labels`, which doesn't exist in v1beta2
+    (`clusterapi_unstructured.go:207`). MD template labels therefore never reach the from-zero
+    template node.
+  - CAPMOX `ProxmoxMachineTemplate` has no `status`, so `kubernetes.io/os` and `arch` become `""` on
+    the template (`clusterapi_nodegroup.go:393-405`). A pod with `nodeSelector: kubernetes.io/os:
+    linux` would not trigger a scale-up from zero.
+  - The capacity key is `maxPods`; our `maximum-pods` was ignored, and 110 only applied because it
+    is the default.
+  - `MixedTemplateNodeInfoProvider` caches a template built from a real node for
+    `--node-info-cache-expire-time` (default 87600h). A group that has had nodes keeps that
+    template: ~1.78Gi allocatable, os/arch set, `node-role` label. A from-zero group gets the
+    synthetic template: 2Gi, empty os/arch. The generic comparator rejects that pair.
+  - That this cache caused this particular decision is inferred. The reason for a mismatch is only
+    logged at V(5).
+- **Fix (`lab1.yaml` hand edit #12):**
+  - both MDs carry the template label `node.cluster.x-k8s.io/pool: workers`, which CAPI syncs to the
+    Node;
+  - the annotation `capacity.cluster-autoscaler.kubernetes.io/labels:
+    node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64`;
+  - `maximum-pods` → `maxPods`;
+  - CA `balancing-label: node.cluster.x-k8s.io/pool`, which compares only that label and turns off
+    the resource checks.
+
+**2. Loop 8–26 s, throttled.**
+- The log is full of `Waited before sending request … client-side throttling` on `/api`, `/apis`
+  and `/apis/infrastructure.cluster.x-k8s.io/v1alpha2` against the lab-mgmt API.
+- Cause:
+  - For every MD at 0 replicas, `CanScaleFromZero` → `readInfrastructureReferenceResource` →
+    `getKindPreferredVersion` runs through the provider's **uncached** discovery client. The cached
+    one is built at `clusterapi_provider.go:195` but not passed to the controller.
+  - That is 3 GETs per MD per `NodeGroups()` call, and `NodeGroups()` runs about 15× per loop.
+  - The client uses QPS 5 / burst 10 (`--kube-client-qps` / `--kube-client-burst` defaults).
+- There is no upstream fix up to 1.36.1 or master (2026-09-29).
+- The owner rejected raising QPS (50/100) and wants fewer requests instead. Fix in
+  `configs/cluster-autoscaler/values.yaml`:
+  - `scan-interval: 1m` (was 10s). `loop/trigger.go:74-100`: unschedulable pods and finished
+    scale-ups/downs still start a loop immediately. Only idle loops are spaced out, so scale-down
+    takes up to about +1 min.
+  - `v: 2` (chart default 4). The throttling and "discovered node group" lines are V(3)/V(4).
+- Considered and not taken:
+  - `min-size: 1`: no MD at 0 means no discovery, but two idle VMs.
+  - `CAPI_VERSION=v1beta1`: v1beta1 refs carry `apiVersion`, so no discovery. But v1beta1 is
+    deprecated and the path is untested with CAPMOX.
+  - An upstream patch to pass the cached discovery client (a future task).
+
+## Rename of the pv1-1 group (owner's request, 2026-09-30)
+
+This deviates from the plan: the plan kept `lab1-workers`. The owner wants symmetric per-node
+names now that node names show the MD name (`lab1-workers-zldvn-*` next to
+`lab1-workers-pv1-2-nlv6s-*`).
+
+Changes in `configs/capi/lab1.yaml` (hand edit #13):
+- `MachineDeployment/lab1-workers` → `lab1-workers-pv1-1`.
+- `ProxmoxMachineTemplate/lab1-worker-2g` → `lab1-worker-2g-pv1-1`, now with explicit
+  `allowedNodes: [pv1-1]`.
+
+A rename creates new objects. `kubectl apply` doesn't prune, so the owner deletes the old MD and PMT
+once the old MD has scaled to 0. The timing is right because the CA was already scaling every worker
+down (`inflate` at 0). The new MD comes up with 0 replicas: `replicas` is unset, and the CAPI webhook
+defaults it to the autoscaler min-size annotation.
+
+## Step 4 — re-test after the fixes (2026-09-30 07:23 UTC)
+
+The owner applied the changes in this order:
+1. `kubectl apply -f configs/capi/lab1.yaml`: new `lab1-workers-pv1-1` + `lab1-worker-2g-pv1-1`, the
+   pool label, `maxPods`, os/arch.
+2. Deleted the old `MachineDeployment/lab1-workers` and `ProxmoxMachineTemplate/lab1-worker-2g`.
+3. `helm upgrade --install ca … --version 9.59.0 -f configs/cluster-autoscaler/values.yaml`:
+   `scan-interval 1m`, `v 2`, `balancing-label`.
+4. Scaled `inflate` 0 → 16.
+
+The CA log (`--since=15m`):
+
+```
+orchestrator.go:752] Splitting scale-up between 2 similar node groups: {…/lab1-workers-pv1-2, …/lab1-workers-pv1-1}
+orchestrator.go:263] Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}]
+```
+
+- **Placement:** ProxmoxMachines `lab1-workers-pv1-1-*` on `pv1-1` (VMIDs 103, 104, 107, 110) and
+  `lab1-workers-pv1-2-*` on `pv1-2` (105, 106, 108, 109).
+- **Nodes:** all 8 Ready about 1–1.5 min after creation, with labels
+  `node.cluster.x-k8s.io/pool=workers` and `kubernetes.io/os=linux`.
+- **Loop timing:** idle loops now run on the `1m0s timer`.
+- **One transient warning:** `W clusterstate.go:510 Failed to find acceptable ranges for
+  MachineDeployment/default/lab1-workers-pv1-1`, once, in the loop during which the new MD was
+  created. Acceptable ranges are snapshotted at the start of the loop, and the group appeared in a
+  later `NodeGroups()` call. Harmless; it did not repeat.
+
+Worth noting:
+- `ProxmoxMachineTemplate/lab1-worker` (4 GiB, decision #15) is still in the cluster, and no MD
+  references it. Deleting it is left to the owner.
+
+## Step 5 — docs and history (2026-09-30 09:28 CEST)
+
+- `docs/decisions.md`: #7 → `superseded by #18`. #18 (template copy + MD per PVE node, pool
+  `.31–.50`, symmetric names) and #19 (CA `balancing-label`, `scan-interval 1m`, `v 2`, QPS left at
+  the default) are both `active`, since the lab confirmed them in Step 4.
+- `docs/phase-1-proxmox-capi.md`:
+  - intro and topology (PVE cluster, templates, both MDs, pool);
+  - Step 0: bridge on every node, per-node template;
+  - Step 5: annotations incl. `maxPods`, `labels`, the pool label;
+  - Step 6: the settings for two groups, and what a balanced scale-up logs;
+  - troubleshooting: 7 rows (maxPods, empty os/arch, no balancing, throttling, the
+    acceptable-ranges warning, cross-node clone, VLAN on a new node).
+- `docs/image-builder-proxmox.md`: section "Template on other PVE nodes"; the result line names
+  101/9101.
+- `configs/capi/clusterctl.yaml.example`: comments on `TEMPLATE_VMID` / `ALLOWED_NODES`.
+- `docs/future-tasks.md`: report the three CA clusterapi bugs upstream. The namespace task now uses
+  the new MD names.
+- `README.md` (lab inventory, key decision 7), `CLAUDE.md` / `AGENTS.md` (lab facts):
+  PVE cluster, `pv1-2`, template 9101.
+- `.claude/settings.json`: two inspector allowlist rules `md lab1-workers` → `md lab1-workers-pv1-1`.
+- `CHANGELOG.md`: entry added after the owner confirmed the setup works (2026-09-30 11:05 CEST). The owner's pending #16/#17 work went in first as its own commit (`5e5b6c5`).
@@ -0,0 +1,119 @@
+# Plan: lab1 workers on both PVE nodes (template copy per node)
+
+**Created:** 2026-09-29 23:15 CEST
+
+## Context
+
+`pv1-1` is now the PVE cluster `homelab`, and `pv1-2` (192.168.0.10, 12 cores, 40.9 GiB) has joined
+it. There is no shared storage, and the owner doesn't want any for now. CAPMOX v0.9.1 can't clone
+across nodes from `local-lvm`, and `templateSelector` needs exactly one match in the whole cluster
+([session note](../sessions/2026-09-29-2311-pve-two-node-cluster.md)). So each node needs its own
+template VMID, and workers get one MachineDeployment per PVE node.
+
+Two blockers found through the PVE API:
+- `vmbr0` on `pv1-2` is not VLAN-aware.
+- Template 101 exists only on `pv1-1`. Its config is two `local-lvm` disks (`scsi0`, `efidisk0`),
+  no ISO or `local` references, so an offline migration is possible.
+
+Owner's choices:
+- Copy template 101 rather than rebuilding it.
+- Keep `lab1-workers` as it is and add `lab1-workers-pv1-2`.
+
+## Steps
+
+### Step 0 — Record the plan
+Copy this plan to `docs/plans/<date +%Y-%m-%d-%H%M>-pv1-2-workers.md`, create the execution log
+`docs/plans-executions/<same name>` with the checklist, and commit both on their own to `main`.
+
+### Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner runs)
+In `/etc/network/interfaces` on `pv1-2`, add to the `iface vmbr0` stanza:
+```
+bridge-vlan-aware yes
+bridge-vids 2-4094
+```
+Then run `ifreload -a`. The switch port for `pv1-2` must carry VLAN 4 tagged, the same as `pv1-1`.
+
+Verify (read-only, by me): `GET /nodes/pv1-2/network/vmbr0` → `bridge_vlan_aware: 1`.
+
+### Step 2 — Template 9101 on `pv1-2` (owner runs on `pv1-1`)
+```bash
+qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm  # the clone stays on pv1-1 (local storage)
+qm migrate 9101 pv1-2 --with-local-disks                                        # offline: copies both LVs to pv1-2's local-lvm
+ssh root@192.168.0.10 qm template 9101                                          # on pv1-2
+```
+- VMID 9101 is outside CAPMOX's `nextid` range (103+), next to the existing 9000/9001 templates.
+- The name stays the same on purpose. CAPMOX addresses the template by `templateID`, not by name.
+
+Verify:
+- `/cluster/resources`: 9101 is on `pv1-2` with `template: 1`.
+- Smoke test from [image-builder-proxmox.md](../image-builder-proxmox.md#smoke-test-of-the-template),
+  run on `pv1-2` with `TPL=9101`. It also proves that VLAN 4 works on `pv1-2` (DHCP lease, guest
+  agent answers).
+
+### Step 3 — Second worker MachineDeployment (`configs/capi/lab1.yaml`)
+- New `ProxmoxMachineTemplate/lab1-worker-2g-pv1-2`: a copy of `lab1-worker-2g` with:
+  - `sourceNode: pv1-2`
+  - `templateID: 9101`
+  - `allowedNodes: [pv1-2]` (the machine-level list overrides the cluster's,
+    `vmscheduler.go:52-62`)
+- New `MachineDeployment/lab1-workers-pv1-2`: a copy of `lab1-workers` with:
+  - the same KubeadmConfigTemplate `lab1-worker` and the same capacity annotations
+  - the same rollout `maxSurge 0 / maxUnavailable 1`
+  - min 0 / max **4**
+- `lab1-workers`: max-size annotation 5 → **3**. Only an annotation changes, so no rollout.
+  `lab1-worker-2g` is not edited; templates are never edited in place.
+- `ProxmoxCluster.allowedNodes` stays `[pv1-1]`. The control plane (template 101) must not be
+  scheduled onto `pv1-2`.
+- IP pool check: .31–.39 has 9 addresses; 1 CP + 3 + 4 workers = 8, so one address is spare.
+- Update the header comment: generate command and hand edit #10 (second MD, per-node template),
+  plus #7 for the new max-size.
+
+Owner runs:
+```bash
+kubectl diff -f configs/capi/lab1.yaml   # expect: new PMT + MD, max-size 5→3 on lab1-workers
+kubectl apply -f configs/capi/lab1.yaml
+```
+- No change to cluster-autoscaler values: `autoDiscovery` picks up the new MD, and
+  `balance-similar-node-groups: true` plus the `least-waste` expander spread scale-ups across the
+  two groups.
+- The RBAC rule for `proxmoxmachinetemplates` already covers the new template.
+
+### Step 4 — Scale test
+1. Scale `inflate` so it needs about 4 workers (`configs/test/inflate.yaml`, 2 pods per worker).
+2. Expect:
+   - the CA log shows `discovered node group: MachineDeployment/default/lab1-workers-pv1-2 (min: 0, max: 4…)`
+   - the scale-up is balanced across both MDs
+   - the ProxmoxMachines land on `pv1-1` / `pv1-2` according to `status.proxmoxNode`
+   - the nodes go Ready (Cilium)
+3. Then scale down to 0 and confirm both groups shrink.
+Diagnosis goes through the `cluster-inspector` agent.
+
+### Step 5 — Docs and history
+- `docs/decisions.md`:
+  - new row #18: "Template copy per PVE node (101 on `pv1-1`, 9101 on `pv1-2`); one worker MD per
+    node". Alternatives: shared storage, a rebuild per node, `templateSelector`. Status
+    `proposed` until Step 4 passes, then `active`.
+  - #7 → `superseded by #18`
+- `docs/phase-1-proxmox-capi.md`:
+  - topology table (two MDs, nodes)
+  - Step 0: the storage/network paragraph (per-node template, VLAN-aware bridge on every node)
+  - Step 5 annotations
+  - troubleshooting rows:
+    - clone to another node → "VM uses local storage"
+    - VLAN not reachable on the new node
+- `docs/image-builder-proxmox.md`: a new section "Template on other PVE nodes" with the Step 2
+  commands. The result line then names both VMIDs.
+- `configs/capi/clusterctl.yaml.example`: fix the comments on `ALLOWED_NODES` / `SOURCENODE`.
+- CLAUDE.md, AGENTS.md "Lab facts" and README line 29: PVE cluster `homelab`, `pv1-2`
+  192.168.0.10, templates 101/9101. These files have uncommitted owner edits, so I stage only my
+  hunks (`git add -p`) or ask first.
+- Session note: add the outcome and a link to the plan.
+- `CHANGELOG.md` entry, only after the owner confirms Step 4 works.
+
+## Verification
+- PVE API (read-only): `vmbr0` VLAN-aware on both nodes; template 9101 on `pv1-2`.
+- Smoke-test VM on `pv1-2` gets a VLAN 4 address.
+- `clusterctl describe cluster lab1`: both MDs are healthy.
+- The Step 4 scale test puts workers on both PVE nodes, and they join and become Ready.
+- Rollback: the owner deletes `MachineDeployment/lab1-workers-pv1-2` and restores max 5 on
+  `lab1-workers`. Template 9101 can stay.
@@ -0,0 +1,74 @@
+# Session: lab1 created — first review by the cluster-inspector agent
+
+**Created:** 2026-09-27 11:12 CEST
+
+## Question
+
+> lab cluster looks good, can you look at it with inspector agent ?
+
+`lab1` had just been applied from `configs/capi/lab1.yaml` (phase-1 Step 3). Only the
+management-side read-only kubeconfig existed, so the review covered what the management cluster sees.
+
+## What was checked
+
+`cluster-inspector` agent with `lab-mgmt-ro`: Cluster / KCP / MD / MS / Machines / ProxmoxMachines /
+ProxmoxCluster conditions, IPAM claims, MD field managers, logs of all five CAPI controllers since
+creation, Warning events in `default`. `clusterctl describe` failed (403 on CRDs).
+
+## Findings
+
+- CAPI / CAPMOX side healthy: VMs 102 (CP, 192.168.4.31) and 103 (worker, .32) cloned from
+  template 101 with VLAN 4, raw disk, planned sizes; control plane initialised and Available,
+  etcd/certs healthy, VIP 192.168.4.30 reachable; providerIDs match across Machine /
+  ProxmoxMachine / Node. First real use of the `capmox@pve!capi` token: no Proxmox API errors.
+- Controller errors only in the first ~60 s after creation (no route to VIP, `lab1-kubeconfig`
+  not found, `vm not initialized`) — normal bootstrap noise, nothing after 09:01 UTC.
+- MachineDeployment `replicas` defaulted to 1 and is owned by no field manager, so re-applying
+  `lab1.yaml` won't fight cluster-autoscaler.
+- Both nodes NotReady (`cni plugin not initialized`) as of ~09:09 UTC → Cilium (Step 4) not
+  installed or not starting; the owner's "looks good" didn't hold for node readiness.
+- IPAM pool 2/9 used; 1 CP + 5 workers + 1 surge = 7 fits.
+- On k8s 1.35, `kubectl get ipaddresses` hits the built-in `networking.k8s.io` IPAddress, not
+  CAPI's — runbook and agent now use `ipaddresses.ipam.cluster.x-k8s.io`.
+
+## Open questions
+
+- Why the CNI wasn't up (not installed yet vs. image pull / egress from VLAN 4) — needs the lab1
+  read-only kubeconfig or the owner's check.
+
+## Follow-ups
+
+- `.claude/cluster-inspector/rbac.yaml`: CRD read added (for `clusterctl describe`); applied by the
+  next `ansible-playbook mgmt.yml --tags inspector`.
+- Phase-1 checklist: Step 3 ticked; next Step 4 (Cilium), then Step 5 annotations in `lab1.yaml`.
+
+## 2026-09-28 00:55 CEST — re-check after Steps 4–5
+
+**What was checked:** the `cluster-inspector` agent with `lab-mgmt-ro` covered the MD annotations,
+field managers, Cluster/KCP/MD conditions, IPAM and controller logs. For the workload side, Claude
+ran read-only `kubectl get` with the owner's admin `~/.kube/lab1.kubeconfig`, because no `lab1-ro`
+kubeconfig exists yet. The agent refuses admin kubeconfigs when permission is relayed through
+another agent.
+
+**Findings:**
+
+- `lab1` has been stable for 13h. Cluster, KCP and MD are Available, and the controllers have
+  logged no errors since the ~09:00 UTC bootstrap noise.
+- All six Step 5 annotations are on `lab1-workers` and copied to its MachineSet.
+- Nothing owns `spec.replicas`: `kubectl-client-side-apply` owns only
+  clusterName/selector/template.
+- Both nodes are Ready on v1.35.9, and Node providerIDs match the Machines.
+- Cilium 2/2, envoy 2/2, operator 1/1 and CoreDNS 2/2 (both replicas on the CP) are healthy. No
+  pods are outside Running, and there are no PDBs anywhere.
+- **`cilium-operator` runs on the worker** despite `operator.replicas: 1`. With CA's default
+  `skip-nodes-with-system-pods=true` and no PDB, it would block Step 7's scale-to-zero. Fixed in
+  `configs/cilium/values.yaml` with a `nodeSelector` on the control-plane role; the owner still
+  has to apply it with `helm upgrade`.
+
+**Follow-ups:**
+
+- Owner: `helm upgrade` Cilium with the new values, then confirm the operator runs on
+  `lab1-control-plane-*`.
+- Owner: mint a `lab1-ro` kubeconfig with `.claude/cluster-inspector/make-kubeconfig.sh`, so that
+  workload checks can go through the agent.
+- After Step 6: check that only CA / the scale subresource owns `spec.replicas`.
@@ -0,0 +1,64 @@
+# Session: CAPMOX memory accounting and overcommit on pv1-1
+
+**Created:** 2026-09-28 10:07 CEST
+
+## Question
+
+> can we configure capmox so that it works with real available memory instead of calculation what
+> stopped vms would use if somebody would start them? also can we configure memory overprovisioning
+> on proxmox node?
+
+Context: during Step 7, new workers failed with `cannot reserve …B of memory on node pv1-1`. The
+host had about 12.8 GiB actually available, but CAPMOX counted 168 MB. Later this deadlocked the
+2 GiB worker rollout (see the phase-1 troubleshooting table).
+
+## What was checked (2026-09-28)
+
+- CAPMOX source for v0.9.1 (the latest tag) and `main`:
+  - `pkg/proxmox/goproxmox/api_client.go` `GetReservableMemoryBytes`;
+  - `internal/service/scheduler/vmscheduler.go`;
+  - `api/v1alpha2/proxmoxcluster_types.go` `SchedulerHints`;
+  - `api/v1alpha2/proxmoxmachine_types.go`;
+  - `internal/webhook/proxmoxcluster_webhook.go`.
+- `pv1-1` (read-only):
+  - `qm list`, node memory status;
+  - KSM: `ksmtuned` active, `/sys/kernel/mm/ksm/run` = 0, `pages_sharing` = 0;
+  - 8 GiB swap;
+  - VM 103 has no `balloon:` line.
+
+## Findings
+
+- **No "real available memory" mode in any CAPMOX release.** `main` is identical to v0.9.1. The
+  reservable memory is `node total × memoryAdjustment/100` minus the configured `maxmem` of every
+  non-template VM (running or stopped) and every container. That's deliberate: real usage is
+  racy, because VMs grow into their configured memory later. Using free memory would need an
+  upstream change.
+- **The only knob is `ProxmoxCluster.spec.schedulerHints.memoryAdjustment`** (percent, default 100):
+  - `> 100` allows bounded overcommit;
+  - `0` turns the memory check off completely (it then only checks request ≤ host total).
+  - The ProxmoxCluster update webhook only validates the control-plane endpoint, so the hint can
+    be changed on a live cluster. The scheduler reads it whenever it places a VM, and the stuck
+    ProxmoxMachine retries.
+- **Sizing for `lab1`:** everything except the workers is 21 GiB configured (VMs 100, 102, 201,
+  1001, 2001, 160549). 5 workers × 2 GiB brings that to 31 GiB, and the host has 29.16 GiB, so
+  `memoryAdjustment ≥ 107`. At 120 there's 35 GiB reservable, which fits max 5 workers with about
+  4 GiB to spare.
+- **Proxmox itself enforces no memory overcommit limit.** VMs can be started past physical RAM; CAPMOX
+  was the only admission control. On the PVE side, "overprovisioning" is about how the host copes:
+  - **KSM:** `ksmtuned` is active but only merges below 20% free (`KSM_THRES_COEF`). Identical
+    Ubuntu clones deduplicate well. It's currently idle because the host has about 44% free.
+  - **Host swap (8 GiB):** a safety net, but slow.
+  - **Ballooning** (`balloon: <min>` below `memory`, auto-balloon above 80% host use): not
+    exposed by CAPMOX v0.9.1 (`ProxmoxMachineSpec` has no balloon field). It's also a poor fit
+    for Kubernetes nodes: the guest keeps reporting full capacity while the balloon takes memory
+    away, so the kubelet schedules against memory that isn't there.
+
+## Open questions
+
+- ~~120 or 0?~~ The owner chose **160** (decision #17): 46.7 GiB reservable, about 17.7 GiB headroom with today's VMs.
+- Is an upstream feature request worth filing (a free-memory mode, or excluding stopped VMs)?
+
+## Follow-ups
+
+- Done: `schedulerHints.memoryAdjustment: 160` on `ProxmoxCluster/lab1` in `configs/capi/lab1.yaml`,
+  decisions #17. Watch real host memory and KSM `pages_sharing` during scale tests.
@@ -0,0 +1,100 @@
+# Session: CAPMOX against a two-node PVE cluster
+
+**Date:** 2026-09-29 23:11 CEST
+
+## Question
+
+`pv1-1` became the PVE cluster `homelab`, and a second node `pv1-2` joined it. Does the CAPI/CAPMOX
+configuration need changes to work against the cluster?
+
+## What was checked
+
+- PVE API (read-only GETs with the `capmox@pve!capi` token, 2026-09-29): `/cluster/status`,
+  `/nodes`, `/storage`, `/nodes/<n>/network`, `/nodes/<n>/storage`, `/cluster/resources`,
+  `/access/acl`.
+- CAPMOX v0.9.1 source (Go proxy zip): scheduler, clone path, template lookup,
+  `templates/cluster-template.yaml`.
+- PVE `qemu-server` `PVE/API2/Qemu.pm` (HEAD, not pinned to a PVE release): the clone target check.
+
+## Findings
+
+PVE state:
+
+| Item | pv1-1 | pv1-2 |
+| --- | --- | --- |
+| IP | 192.168.0.8 | 192.168.0.10 |
+| CPU / RAM | 16 / 29.2 GiB | 12 / 40.9 GiB |
+| `vmbr0` VLAN-aware | yes | **no** |
+| `local-lvm` | local LVM-thin | local LVM-thin (not shared) |
+| Templates | 101 `ubuntu-2404-kube-v1.35.9`, 9000, 9001 | none |
+
+- Storage definitions (`local`, `local-lvm`, `proxy-seeds`) are cluster-wide without a `nodes`
+  restriction, but none of them is shared.
+- The `capmox@pve` ACLs (`/`, `/storage`, `/sdn/zones/localnetwork`) are cluster-wide, so they
+  cover `pv1-2` as well. There are no SDN zones.
+
+CAPMOX v0.9.1:
+
+- **Node choice.** If the ProxmoxMachine has its own `allowedNodes`, that list replaces the
+  ProxmoxCluster's (`internal/service/scheduler/vmscheduler.go:52-62`).
+  - The scheduler balances by how many machines of this cluster are already on each node
+    (`status.nodeLocations`), and only picks a node where the requested memory fits
+    (`vmscheduler.go:75-119`).
+  - `memoryAdjustment` is applied per node, against that node's own total memory.
+  - With empty `allowedNodes` the scheduler isn't called and the VM clones on the source node
+    (`internal/service/vmservice/vm.go:477`).
+- **Clone.** `target` goes to the PVE clone call as-is (`pkg/proxmox/goproxmox/api_client.go:84-95`).
+  - PVE refuses a cross-node clone when any source disk is on non-shared storage:
+    `can't clone VM to node '<target>' (VM uses local storage)`.
+  - Setting `full: true` or `storage` does not change that.
+  - v0.9.1 has no `localStorage` option.
+- **`templateSelector`** searches the whole PVE cluster and needs exactly one match
+  (`api_client.go:149-212`). "One template per node with the same tags" fails with
+  `found 2 VM templates`.
+- **API URL and credentials.** Everything goes through one API URL; nothing in the credentials
+  names a node.
+  - Cross-node requests should be forwarded by PVE (standard PVE behaviour, not checked in code).
+  - `192.168.0.8` is `pv1-1`, so a down `pv1-1` also means no API for CAPMOX.
+- **Bridge.** `bridge`/`vlan` are written into `netX` without any check (`vm.go:338`).
+
+Conclusion:
+
+- Nothing has to change for the current setup to keep working. `lab1` pins `allowedNodes: [pv1-1]`
+  and `sourceNode: pv1-1`, which is decision #7.
+- Using `pv1-2` requires two things:
+  1. Make `vmbr0` on `pv1-2` VLAN-aware and trunk VLAN 4 on its switch port.
+  2. Either put a template copy on `pv1-2`, or add shared storage.
+- With a template copy per node:
+  - Workers get one ProxmoxMachineTemplate and one MachineDeployment per PVE node, each with
+    `sourceNode`, `templateID` (the node's own VMID) and `allowedNodes: [<node>]`. For
+    cluster-autoscaler that is one node group per PVE node.
+  - KubeadmControlPlane takes a single infrastructureRef, so the control plane stays on one node.
+
+## Open questions
+
+- How to get the template onto `pv1-2`: a second image-builder run (`PROXMOX_NODE=pv1-2`), or
+  `qm clone` + `qm migrate` / vzdump-restore? The image-builder run is reproducible; the copy is
+  faster.
+- Is shared storage (NFS) worth adding instead? It gives one MachineDeployment spread over both
+  nodes, and the control plane can use both nodes too.
+- Behaviour when a cross-node clone fails: the scheduler may keep picking the same node. This is
+  inferred from the code, not tested.
+
+## Follow-ups
+
+- The owner chose: no shared storage for now; copy template 101 to `pv1-2` as 9101; keep
+  `lab1-workers` and add `lab1-workers-pv1-2`. Plan:
+  [2026-09-29-2315-pv1-2-workers](../plans/2026-09-29-2315-pv1-2-workers.md). It covers the bridge
+  fix, the template copy, the second MD, the CA limits and a decision row superseding #7.
+- Update the "Lab facts" in CLAUDE.md/AGENTS.md: PVE cluster `homelab`, `pv1-2` at 192.168.0.10.
+
+## Outcome (2026-09-30)
+
+Done as planned, and confirmed on the lab:
+- template 9101 on `pv1-2`;
+- MDs `lab1-workers-pv1-1` (max 6) + `lab1-workers-pv1-2` (max 12);
+- a balanced scale-up of 4 + 4 across both PVE nodes.
+
+Two cluster-autoscaler findings from the scale test (no balancing, and throttling from uncached
+discovery) led to decision #19. Details are in the
+[execution log](../plans-executions/2026-09-29-2315-pv1-2-workers.md).
Author	SHA1	Message	Date
kacerrandClaude Opus 5.5	6ef55778ff	lab1 workers on both PVE nodes: per-node template + MD, balanced CA PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage: - template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50; lab1-workers renamed for symmetry (decision #18, #7 superseded) - CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19). Scale test: 16 pods -> 4 + 4 across both nodes. - runbook, image-builder copy procedure, fixes tables, future task to report three CA clusterapi bugs upstream, lab facts, changelog, execution log Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-30 11:07:17 +02:00
kacerrandClaude Opus 5.5	5e5b6c506a	lab1: 2 GiB workers, CAPMOX memory overcommit, CA install (Step 6), future-tasks Owner's pending work, committed ahead of the pv1-2 change it underlies: - lab1.yaml: worker template lab1-worker-2g (2048 MiB, decision #16), MD rollout maxSurge 0 / maxUnavailable 1, ProxmoxCluster schedulerHints.memoryAdjustment 160 (#17) - cluster-autoscaler values: RBAC for proxmoxmachinetemplates, aggressive scale-down profile, Cilium startup-taint; runbook Steps 4-7 and troubleshooting rows - inflate test pods 512Mi; decisions #16/#17; session note on the memory overcommit - docs/future-tasks.md convention (CLAUDE.md, README) with the namespace-per-cluster task - AGENTS.md; allowlist entries in .claude/settings.json Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-30 11:05:39 +02:00
kacerrandClaude Opus 5.5	f549d4a004	pv1-2: VLAN-aware vmbr0, template 9101 (Steps 1-2 done) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-30 08:39:12 +02:00
kacerrandClaude Opus 5.5	a9f5879f09	image-builder smoke test: free VMID 9901, key file per node ID 9001 is now debian-13-template and the snippet ends with qm destroy $ID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-30 08:36:03 +02:00
kacerrandClaude Opus 5.5	b88f721e3b	Plan: lab1 workers on pv1-2 via per-node template copy Session note on CAPMOX v0.9.1 against the 2-node PVE cluster, the plan and its execution log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-29 23:16:49 +02:00
kacerrandClaude Opus 5.5	81462e0e0f	lab1: pin cilium-operator to the control plane; Step 5 verified Re-check of lab1 after Steps 4-5 (inspector on lab-mgmt, read-only gets on lab1): - Cluster/KCP/MD Available for 13h; no controller errors since bootstrap. - All six autoscaler annotations on lab1-workers (copied to the MachineSet). spec.replicas has no owner: kubectl-client-side-apply owns only clusterName/selector/template, so re-applying lab1.yaml won't fight cluster-autoscaler. - Nodes Ready on v1.35.9; Node/Machine providerIDs match. - Cilium, envoy and CoreDNS healthy. No PDBs in the cluster. Found: cilium-operator (replicas: 1) was scheduled onto the only worker. It is a non-DaemonSet kube-system pod without a PDB, so with CA's default skip-nodes-with-system-pods=true the last worker could never scale to zero (Step 7). The chart already tolerates the control-plane taint, so a nodeSelector on node-role.kubernetes.io/control-plane pins it there. - configs/cilium/values.yaml: operator.nodeSelector. - phase-1 doc: Step 5 ticked, Step 4 explains why replicas: 1 isn't enough, new troubleshooting row. - Session note extended with the re-check. Not applied yet: the owner runs helm upgrade with the new values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-28 09:23:41 +02:00
kacerrandClaude Opus 5.5	49c548e296	cluster-inspector: read-only kubeconfig for lab1, workload-aware triage The agent could only reach lab-mgmt, so it couldn't verify nodes, CNI or system pods on lab1. It correctly refused the admin ~/.kube/lab1.kubeconfig when the owner's permission was relayed through the main session. Workload kubeconfigs: - make-kubeconfig.sh: new --capi-cluster [NS/]NAME. --admin-kubeconfig is then the management cluster. The workload admin kubeconfig is read from Secret NS/NAME-kubeconfig into a mktemp file, removed on exit, so no admin file for the workload cluster lands on disk. - ansible/mgmt.yml --tags inspector now also mints kubeconfigs for every cluster in the new inspector_workload_clusters (group_vars: lab1). Clusters without a <name>-kubeconfig Secret are skipped. The tag is now the single weekly token refresh; tasks stay changed_when: false, so the playbook's changed=0 rule still holds. - Owner ran --tags inspector: kubeconfigs/lab1.kubeconfig (lab1-ro) minted, can-i checks as expected. Agent prompt: - Cluster map: what to inspect on lab-mgmt vs the workload cluster. Machine -> Node -> VM via nodeRef, providerID and ProxmoxMachine.spec.virtualMachineID (checked on live objects). - New triage step: scale-down blockers (non-DaemonSet kube-system pods without PDB, local storage, bare pods, PDBs at 0, safe-to-evict / scale-down-disabled). - Refuses admin kubeconfigs even when a caller relays permission; points the owner to --tags inspector on missing file or 401. - Verb before --kubeconfig so commands match the kubectl get/describe/logs allow rules; single-quote jsonpath/custom-columns for zsh. - Report format: abnormal first, then health summary, then unverified. settings.json: allow kubectl describe/logs/events and clusterctl describe. All four are read-only, and the RO kubeconfigs enforce read-only anyway. Checked: bash -n on the script; ansible-playbook --syntax-check and ansible-lint --profile production pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-28 09:23:41 +02:00
kacerrandClaude Opus 5.5	1d0b669d07	lab1: autoscaler min/max and capacity annotations on lab1-workers (Step 5) kubectl diff against the live cluster shows only the six annotations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-27 11:26:55 +02:00
kacerrandClaude Opus 5.5	9fb8a8dcc1	Phase-1 Step 4 done: Cilium up, lab1 Available Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-27 11:25:47 +02:00
kacerrandClaude Opus 5.5	2c793da4f6	Cilium values file for lab1; runbook Step 4 uses helm upgrade --install ipam.mode=kubernetes, operator.replicas=1 (default 2 would pin one on a worker and block scale-to-zero). Keys checked against chart 1.20.2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-27 11:16:04 +02:00
kacerrandClaude Opus 5.5	6d552ebf33	lab1 provisioned: fix IPAddress kubectl name, inspector CRD read, review notes - runbook + inspector agent: ipaddresses.ipam.cluster.x-k8s.io (plain `ipaddresses` is the built-in networking.k8s.io type on k8s 1.35) - inspector RBAC: get/list CRDs so clusterctl describe works read-only - phase-1 Step 3 ticked; session note with the first lab1 review Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-27 11:14:11 +02:00
kacerrandClaude Opus 5.5	57f58f1fc5	Add lab1 workload cluster manifest; workers sized 4 vCPU / 4 GiB Generated with clusterctl v1.12.11 + the six runbook edits (pod CIDR, kube-vip v1.2.4, no qcow2, VLAN 4, worker size, no MD replicas); passes server-side dry run. Inflate test pods lowered to 1Gi to stay CPU-bound, capacity annotation 4Gi, decision #15. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-27 10:54:44 +02:00