lab1 workers on both PVE nodes: per-node template + MD, balanced CA

PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage:
- template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and
  lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50;
  lab1-workers renamed for symmetry (decision #18, #7 superseded)
- CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels
  with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19).
  Scale test: 16 pods -> 4 + 4 across both nodes.
- runbook, image-builder copy procedure, fixes tables, future task to report
  three CA clusterapi bugs upstream, lab facts, changelog, execution log

Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/decisions.md

@@ -15,7 +15,7 @@ image-builder fixes log; their dates are when the decision was recorded, not nec
 | 4 | 2026-09-24 | image-builder overrides go in a `PACKER_VAR_FILES` JSON | `PACKER_FLAGS` | flags were silently lost when `PACKER_FLAGS` was re-exported ([fixes log #4](image-builder-proxmox.md#fixes-log)) | active |
 | 5 | 2026-09-24 | Cilium installed by Helm with `ipam.mode=kubernetes`, pod CIDR `10.244.0.0/16` | ClusterResourceSet CNI flavors; pod CIDR `192.168.0.0/16` | keeps the CNI version under our control; `192.168.0.0/16` overlaps the lab LAN | active |
 | 6 | 2026-09-24 | Scale-from-zero via `capacity.cluster-autoscaler.kubernetes.io/*` annotations on the MachineDeployment | infrastructure provider publishing `status.capacity` | CAPMOX does not publish `status.capacity` on `ProxmoxMachineTemplate` | active |
-| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | active |
+| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | superseded by #18 |
 | 8 | 2026-09-26 | Phase 2: spike CAPLV first; own provider if insufficient; KubeVirt as fallback ([phase-2](phase-2-kvm-provider.md#suggested-approach)) | CAPK + KubeVirt, Metal3, own provider from the start | CAPLV shares CAPI 1.12 with CAPMOX and fits plain libvirt hosts directly | proposed |
 | 9 | 2026-09-26 | Phases 2 and 3 are independent and can be done in either order | strict sequence | no technical dependency between the KVM provider and hosted control planes | active |
 | 10 | 2026-09-26 | Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog ([session](sessions/2026-09-26-2355-history-conventions.md)) | chat history only; raw transcripts committed | research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets | active |
@@ -26,3 +26,5 @@ image-builder fixes log; their dates are when the decision was recorded, not nec
 | 15 | 2026-09-27 | `lab1` workers 4 vCPU / **4 GiB** (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker | 8 GiB workers with 2Gi inflate pods | owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape ([configs/capi/lab1.yaml](../configs/capi/lab1.yaml)) | superseded by #16 |
 | 16 | 2026-09-28 | `lab1` workers 4 vCPU / **2 GiB** (new `ProxmoxMachineTemplate/lab1-worker-2g`, rolled in via the MD); inflate pods 1500m / **512Mi**, still CPU-bound at 2 per worker | keep 4 GiB and delete stopped VMs; CAPMOX `memoryAdjustment` overcommit | CAPMOX reserves the configured memory of every VM, stopped ones included, so 4 GiB workers ran out after the first scale-up ([troubleshooting](phase-1-proxmox-capi.md#troubleshooting)); 2 GiB (~1.78 GiB allocatable) still leaves room for kubelet/containerd plus real pods, and max 5 workers now fits the host | proposed |
 | 17 | 2026-09-28 | CAPMOX memory overcommit on `lab1`: `ProxmoxCluster.spec.schedulerHints.memoryAdjustment: 160` (46.7 GiB reservable on the 29.16 GiB `pv1-1`); MD rollout `maxSurge 0 / maxUnavailable 1` | 100% (default); 120% (minimum sizing for max 5 × 2 GiB workers); 0 (memory check off); deleting stopped VMs | CAPMOX counts the configured memory of stopped VMs (12 GiB here), and no release can use real free memory; 160% keeps a bound, and host-side protection is KSM + swap, not ballooning ([session](sessions/2026-09-28-1007-capmox-memory-overcommit.md)) | proposed |
+| 18 | 2026-09-30 | PVE is now a 2-node cluster (`pv1-1` + `pv1-2`), no shared storage: a **template copy per node** (101 on `pv1-1`, 9101 on `pv1-2`, copied by clone + offline migrate), **one worker MD per node** (`lab1-workers-pv1-1` max 6, `lab1-workers-pv1-2` max 12, PMT `allowedNodes: [<node>]`), control plane stays on `pv1-1`, node IP pool `.31–.50` | shared storage (NFS); a second image-builder run on `pv1-2`; `templateSelector` with the same tags on both nodes; keep `lab1-workers` and only add a second MD | PVE refuses cross-node clones from local storage and CAPMOX v0.9.1 always passes the scheduled node as target; `templateSelector` needs exactly one match cluster-wide; a copy is byte-identical; owner wants no shared storage for now and symmetric names ([plan](plans/2026-09-29-2315-pv1-2-workers.md), [session](sessions/2026-09-29-2311-pve-two-node-cluster.md)) | active |
+| 19 | 2026-09-30 | cluster-autoscaler with two node groups: `balancing-label: node.cluster.x-k8s.io/pool` (label on both MDs + `capacity…/labels` annotation), `scan-interval: 1m`, `v: 2`; client QPS left at the default 5/10 | generic similarity comparator; `balancing-ignore-label` for `deployment-name`; `kube-client-qps 50 / burst 100`; `min-size 1` per group; `CAPI_VERSION=v1beta1` | the comparator fails between a cached real-node template and a from-zero template (6+2 instead of 4+4); CA 1.35.2 runs uncached discovery per zero-replica MD ~15× per loop, and the owner prefers fewer requests over a higher rate — pending pods still trigger loops immediately ([execution log](plans-executions/2026-09-29-2315-pv1-2-workers.md)) | active |