cluster-autoscaler-with-capi/docs/decisions.md

Decisions

Every direction chosen, rejected or reversed, newest at the bottom. Rows are never rewritten: a reversal is a new row, and the old row's status becomes superseded by #N. Status is active, proposed (not yet confirmed by lab work) or superseded by #N.

Rows 1–9 were back-filled on 2026-09-26 from the README, versions.md, the phase docs and the image-builder fixes log; their dates are when the decision was recorded, not necessarily made.

#	Date	Decision	Alternatives considered	Why	Status
1	2026-09-24	Pin CAPI core to 1.12.x (versions.md)	latest CAPI 1.14.x	CAPMOX v0.9.x is only tested against CAPI 1.11/1.12; caps workload Kubernetes and cluster-autoscaler at 1.35	active
2	2026-09-24	Management cluster is k3s on a Proxmox VM	kind on a workstation	CAPI controllers and the autoscaler must run permanently	active
3	2026-09-24	Node images built with kubernetes-sigs/image-builder (Ubuntu 24.04, Proxmox builder)	—	upstream-maintained CAPI images	active
4	2026-09-24	image-builder overrides go in a PACKER_VAR_FILES JSON	PACKER_FLAGS	flags were silently lost when PACKER_FLAGS was re-exported (fixes log #4)	active
5	2026-09-24	Cilium installed by Helm with ipam.mode=kubernetes, pod CIDR 10.244.0.0/16	ClusterResourceSet CNI flavors; pod CIDR 192.168.0.0/16	keeps the CNI version under our control; 192.168.0.0/16 overlaps the lab LAN	active
6	2026-09-24	Scale-from-zero via capacity.cluster-autoscaler.kubernetes.io/* annotations on the MachineDeployment	infrastructure provider publishing status.capacity	CAPMOX does not publish status.capacity on ProxmoxMachineTemplate	active
7	2026-09-24	Template only on pv1-1, ALLOWED_NODES=[pv1-1]	template on every node; shared storage	local-lvm is node-local and there is no shared storage yet	superseded by #18
8	2026-09-26	Phase 2: spike CAPLV first; own provider if insufficient; KubeVirt as fallback (phase-2)	CAPK + KubeVirt, Metal3, own provider from the start	CAPLV shares CAPI 1.12 with CAPMOX and fits plain libvirt hosts directly	proposed
9	2026-09-26	Phases 2 and 3 are independent and can be done in either order	strict sequence	no technical dependency between the KVM provider and hosted control planes	active
10	2026-09-26	Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog (session)	chat history only; raw transcripts committed	research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets	active
11	2026-09-27	Cluster inspection by agents only via read-only ServiceAccount kubeconfigs minted by .claude/cluster-inspector/make-kubeconfig.sh (short-lived TokenRequest tokens, gitignored)	admin kubeconfig + permission allowlist; long-lived SA token Secret; RBAC under configs/	RBAC (view, no Secrets) enforces read-only regardless of which kubectl verb an agent runs — also closes the kubectl get secret gap from the history-conventions session; short-lived tokens limit leak impact	active
12	2026-09-27	Management cluster built by one idempotent Ansible playbook, ansible/mgmt.yml, run from the laptop (plan)	cloud-init + k3s auto-deploy manifests; bash scripts with guards; OpenTofu + Helm	owner's choice; idempotent modules, one tool; only ansible.builtin, charts through k3s's helm-controller so nothing extra on the VM	active
13	2026-09-27	CAPI providers installed by cluster-api-operator 0.28.0 (Provider CRs), cert-manager v1.21.2 pre-installed	guarded clusterctl init	declarative; re-apply is a no-op; upgrade = spec.version bump; 0.28.0 built against CAPI 1.12.10. Consequence: no clusterctl init/upgrade/move/delete	active
14	2026-09-27	Management VM created by the playbook from the pinned Ubuntu 24.04 cloud image (VM 201, 192.168.4.21)	hand-made VM; the image-builder node template	reproducible from the repo; the image-builder template is a kubeadm node image, not meant for k3s	active
15	2026-09-27	lab1 workers 4 vCPU / 4 GiB (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker	8 GiB workers with 2Gi inflate pods	owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape (configs/capi/lab1.yaml)	superseded by #16
16	2026-09-28	lab1 workers 4 vCPU / 2 GiB (new ProxmoxMachineTemplate/lab1-worker-2g, rolled in via the MD); inflate pods 1500m / 512Mi, still CPU-bound at 2 per worker	keep 4 GiB and delete stopped VMs; CAPMOX memoryAdjustment overcommit	CAPMOX reserves the configured memory of every VM, stopped ones included, so 4 GiB workers ran out after the first scale-up (troubleshooting); 2 GiB (~1.78 GiB allocatable) still leaves room for kubelet/containerd plus real pods, and max 5 workers now fits the host	proposed
17	2026-09-28	CAPMOX memory overcommit on lab1: ProxmoxCluster.spec.schedulerHints.memoryAdjustment: 160 (46.7 GiB reservable on the 29.16 GiB pv1-1); MD rollout maxSurge 0 / maxUnavailable 1	100% (default); 120% (minimum sizing for max 5 × 2 GiB workers); 0 (memory check off); deleting stopped VMs	CAPMOX counts the configured memory of stopped VMs (12 GiB here), and no release can use real free memory; 160% keeps a bound, and host-side protection is KSM + swap, not ballooning (session)	proposed
18	2026-09-30	PVE is now a 2-node cluster (pv1-1 + pv1-2), no shared storage: a template copy per node (101 on pv1-1, 9101 on pv1-2, copied by clone + offline migrate), one worker MD per node (lab1-workers-pv1-1 max 6, lab1-workers-pv1-2 max 12, PMT allowedNodes: [<node>]), control plane stays on pv1-1, node IP pool .31–.50	shared storage (NFS); a second image-builder run on pv1-2; templateSelector with the same tags on both nodes; keep lab1-workers and only add a second MD	PVE refuses cross-node clones from local storage and CAPMOX v0.9.1 always passes the scheduled node as target; templateSelector needs exactly one match cluster-wide; a copy is byte-identical; owner wants no shared storage for now and symmetric names (plan, session)	active
19	2026-09-30	cluster-autoscaler with two node groups: balancing-label: node.cluster.x-k8s.io/pool (label on both MDs + capacity…/labels annotation), scan-interval: 1m, v: 2; client QPS left at the default 5/10	generic similarity comparator; balancing-ignore-label for deployment-name; kube-client-qps 50 / burst 100; min-size 1 per group; CAPI_VERSION=v1beta1	the comparator fails between a cached real-node template and a from-zero template (6+2 instead of 4+4); CA 1.35.2 runs uncached discovery per zero-replica MD ~15× per loop, and the owner prefers fewer requests over a higher rate — pending pods still trigger loops immediately (execution log)	active