lab1 workers on both PVE nodes: per-node template + MD, balanced CA

PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage:
- template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and
  lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50;
  lab1-workers renamed for symmetry (decision #18, #7 superseded)
- CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels
  with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19).
  Scale test: 16 pods -> 4 + 4 across both nodes.
- runbook, image-builder copy procedure, fixes tables, future task to report
  three CA clusterapi bugs upstream, lab facts, changelog, execution log

Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plans-executions/2026-09-29-2315-pv1-2-workers.md

@@ -7,9 +7,9 @@ Pairs with [docs/plans/2026-09-29-2315-pv1-2-workers.md](../plans/2026-09-29-231
 - [x] Step 0 — Record the plan
 - [x] Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner)
 - [x] Step 2 — Template 9101 on `pv1-2` (owner)
-- [ ] Step 3 — Second worker MachineDeployment `lab1-workers-pv1-2`
-- [ ] Step 4 — Scale test across both nodes
-- [ ] Step 5 — Docs and history
+- [x] Step 3 — Second worker MachineDeployment `lab1-workers-pv1-2`
+- [x] Step 4 — Scale test across both nodes
+- [x] Step 5 — Docs and history
 
 ## Step 0 — Record the plan (2026-09-29 23:15 CEST)
 
@@ -58,3 +58,165 @@ Worth noting:
   unique.
 - The copied `efidisk0` reports `size=528K` (the source says 4M). This doesn't matter: the template
   boots SeaBIOS.
+
+## Step 3 — second worker MachineDeployment (prepared 2026-09-30, apply pending)
+
+Changes in `configs/capi/lab1.yaml`:
+- new `MachineDeployment/lab1-workers-pv1-2` and `ProxmoxMachineTemplate/lab1-worker-2g-pv1-2`
+  (`sourceNode: pv1-2`, `templateID: 9101`, `allowedNodes: [pv1-2]`)
+- `ProxmoxCluster.allowedNodes` stays `[pv1-1]` for the control plane
+- header hand edits #10 and #11
+
+Deviation from the plan: the IP pool is enlarged and the max sizes are raised (owner's request,
+"20 nodes"):
+- `ipv4Config.addresses` `192.168.4.31-192.168.4.39` → `192.168.4.31-192.168.4.50`
+- `lab1-workers` max 5 → **6**, `lab1-workers-pv1-2` max **12**
+- 6 + 12 + 1 CP = 19 of 20 addresses. The spare covers the KCP rollout surge, and CAPMOX pools
+  hold a freed address for 60 s (`AddressReuseGracePeriodSeconds`, `pkg/kubernetes/ipam/ipam.go:318`).
+- An ARP probe from `lab-mgmt` found `.32–.60` silent; `.30` (VIP) and `.31` (CP) share the CP VM's
+  MAC.
+- The owner confirmed `.31–.50` is outside the VLAN 4 DHCP range.
+- Changing the pool is safe: the ProxmoxCluster webhook only validates the addresses, and the
+  controller copies them into the InClusterIPPool on every reconcile (`ipam.go:323-324`).
+
+Sizing:
+- `pv1-1`: 6 × 2 GiB within ~17.7 GiB of reservable headroom (160 %).
+- `pv1-2`: 12 × 2 GiB of 65.4 GiB reservable (40.9 GiB physical); 48 vCPU on 12 cores, which is
+  fine for the pause-container `inflate` tests but CPU-bound under real load.
+
+`yq` parse check: 9 documents; both worker PMTs point at the right node, template and `allowedNodes`.
+
+Applied by the owner (`kubectl diff` / `kubectl apply -f configs/capi/lab1.yaml`). The CA
+discovered `lab1-workers (max 6)` and `lab1-workers-pv1-2 (max 12)`.
+
+## Step 4 — scale test, first run (2026-09-30 ~06:45 UTC)
+
+The `inflate` Deployment had 16 pods (1500m / 512Mi each, 2 per worker).
+- **Result:** 8 workers Ready by 06:47Z.
+  - `lab1-workers` 0→6: VMIDs 103–108 on `pv1-1`, IPs .34–.39.
+  - `lab1-workers-pv1-2` 0→2: VMIDs 109–110 on `pv1-2`, IPs .32–.33.
+  - No `VMProvisionFailed`. `status.proxmoxNode` and providerIDs are consistent. The pool had 9 of
+    20 addresses in use.
+- **Checked with:** the `cluster-inspector` agent (read-only). Root causes came from the source
+  (`upstream-researcher`, CA tag `cluster-autoscaler-1.35.2`, CAPI v1.12.11, CAPMOX v0.9.1).
+
+Two problems.
+
+**1. Not balanced.** The log shows `Best option to resize: lab1-workers` → `No similar node groups
+found` → `0->6 (max: 6)`. Only in the next loop did `lab1-workers-pv1-2` get `0->2`.
+- The inspector guessed the `cluster.x-k8s.io/deployment-name` label. The source rules that out:
+  CA never reads it.
+- What the source does show in CA 1.35.2 `cloudprovider/clusterapi`:
+  - `Labels()` reads `spec.template.spec.metadata.labels`, which doesn't exist in v1beta2
+    (`clusterapi_unstructured.go:207`). MD template labels therefore never reach the from-zero
+    template node.
+  - CAPMOX `ProxmoxMachineTemplate` has no `status`, so `kubernetes.io/os` and `arch` become `""` on
+    the template (`clusterapi_nodegroup.go:393-405`). A pod with `nodeSelector: kubernetes.io/os:
+    linux` would not trigger a scale-up from zero.
+  - The capacity key is `maxPods`; our `maximum-pods` was ignored, and 110 only applied because it
+    is the default.
+  - `MixedTemplateNodeInfoProvider` caches a template built from a real node for
+    `--node-info-cache-expire-time` (default 87600h). A group that has had nodes keeps that
+    template: ~1.78Gi allocatable, os/arch set, `node-role` label. A from-zero group gets the
+    synthetic template: 2Gi, empty os/arch. The generic comparator rejects that pair.
+  - That this cache caused this particular decision is inferred. The reason for a mismatch is only
+    logged at V(5).
+- **Fix (`lab1.yaml` hand edit #12):**
+  - both MDs carry the template label `node.cluster.x-k8s.io/pool: workers`, which CAPI syncs to the
+    Node;
+  - the annotation `capacity.cluster-autoscaler.kubernetes.io/labels:
+    node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64`;
+  - `maximum-pods` → `maxPods`;
+  - CA `balancing-label: node.cluster.x-k8s.io/pool`, which compares only that label and turns off
+    the resource checks.
+
+**2. Loop 8–26 s, throttled.**
+- The log is full of `Waited before sending request … client-side throttling` on `/api`, `/apis`
+  and `/apis/infrastructure.cluster.x-k8s.io/v1alpha2` against the lab-mgmt API.
+- Cause:
+  - For every MD at 0 replicas, `CanScaleFromZero` → `readInfrastructureReferenceResource` →
+    `getKindPreferredVersion` runs through the provider's **uncached** discovery client. The cached
+    one is built at `clusterapi_provider.go:195` but not passed to the controller.
+  - That is 3 GETs per MD per `NodeGroups()` call, and `NodeGroups()` runs about 15× per loop.
+  - The client uses QPS 5 / burst 10 (`--kube-client-qps` / `--kube-client-burst` defaults).
+- There is no upstream fix up to 1.36.1 or master (2026-09-29).
+- The owner rejected raising QPS (50/100) and wants fewer requests instead. Fix in
+  `configs/cluster-autoscaler/values.yaml`:
+  - `scan-interval: 1m` (was 10s). `loop/trigger.go:74-100`: unschedulable pods and finished
+    scale-ups/downs still start a loop immediately. Only idle loops are spaced out, so scale-down
+    takes up to about +1 min.
+  - `v: 2` (chart default 4). The throttling and "discovered node group" lines are V(3)/V(4).
+- Considered and not taken:
+  - `min-size: 1`: no MD at 0 means no discovery, but two idle VMs.
+  - `CAPI_VERSION=v1beta1`: v1beta1 refs carry `apiVersion`, so no discovery. But v1beta1 is
+    deprecated and the path is untested with CAPMOX.
+  - An upstream patch to pass the cached discovery client (a future task).
+
+## Rename of the pv1-1 group (owner's request, 2026-09-30)
+
+This deviates from the plan: the plan kept `lab1-workers`. The owner wants symmetric per-node
+names now that node names show the MD name (`lab1-workers-zldvn-*` next to
+`lab1-workers-pv1-2-nlv6s-*`).
+
+Changes in `configs/capi/lab1.yaml` (hand edit #13):
+- `MachineDeployment/lab1-workers` → `lab1-workers-pv1-1`.
+- `ProxmoxMachineTemplate/lab1-worker-2g` → `lab1-worker-2g-pv1-1`, now with explicit
+  `allowedNodes: [pv1-1]`.
+
+A rename creates new objects. `kubectl apply` doesn't prune, so the owner deletes the old MD and PMT
+once the old MD has scaled to 0. The timing is right because the CA was already scaling every worker
+down (`inflate` at 0). The new MD comes up with 0 replicas: `replicas` is unset, and the CAPI webhook
+defaults it to the autoscaler min-size annotation.
+
+## Step 4 — re-test after the fixes (2026-09-30 07:23 UTC)
+
+The owner applied the changes in this order:
+1. `kubectl apply -f configs/capi/lab1.yaml`: new `lab1-workers-pv1-1` + `lab1-worker-2g-pv1-1`, the
+   pool label, `maxPods`, os/arch.
+2. Deleted the old `MachineDeployment/lab1-workers` and `ProxmoxMachineTemplate/lab1-worker-2g`.
+3. `helm upgrade --install ca … --version 9.59.0 -f configs/cluster-autoscaler/values.yaml`:
+   `scan-interval 1m`, `v 2`, `balancing-label`.
+4. Scaled `inflate` 0 → 16.
+
+The CA log (`--since=15m`):
+
+```
+orchestrator.go:752] Splitting scale-up between 2 similar node groups: {…/lab1-workers-pv1-2, …/lab1-workers-pv1-1}
+orchestrator.go:263] Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}]
+```
+
+- **Placement:** ProxmoxMachines `lab1-workers-pv1-1-*` on `pv1-1` (VMIDs 103, 104, 107, 110) and
+  `lab1-workers-pv1-2-*` on `pv1-2` (105, 106, 108, 109).
+- **Nodes:** all 8 Ready about 1–1.5 min after creation, with labels
+  `node.cluster.x-k8s.io/pool=workers` and `kubernetes.io/os=linux`.
+- **Loop timing:** idle loops now run on the `1m0s timer`.
+- **One transient warning:** `W clusterstate.go:510 Failed to find acceptable ranges for
+  MachineDeployment/default/lab1-workers-pv1-1`, once, in the loop during which the new MD was
+  created. Acceptable ranges are snapshotted at the start of the loop, and the group appeared in a
+  later `NodeGroups()` call. Harmless; it did not repeat.
+
+Worth noting:
+- `ProxmoxMachineTemplate/lab1-worker` (4 GiB, decision #15) is still in the cluster, and no MD
+  references it. Deleting it is left to the owner.
+
+## Step 5 — docs and history (2026-09-30 09:28 CEST)
+
+- `docs/decisions.md`: #7 → `superseded by #18`. #18 (template copy + MD per PVE node, pool
+  `.31–.50`, symmetric names) and #19 (CA `balancing-label`, `scan-interval 1m`, `v 2`, QPS left at
+  the default) are both `active`, since the lab confirmed them in Step 4.
+- `docs/phase-1-proxmox-capi.md`:
+  - intro and topology (PVE cluster, templates, both MDs, pool);
+  - Step 0: bridge on every node, per-node template;
+  - Step 5: annotations incl. `maxPods`, `labels`, the pool label;
+  - Step 6: the settings for two groups, and what a balanced scale-up logs;
+  - troubleshooting: 7 rows (maxPods, empty os/arch, no balancing, throttling, the
+    acceptable-ranges warning, cross-node clone, VLAN on a new node).
+- `docs/image-builder-proxmox.md`: section "Template on other PVE nodes"; the result line names
+  101/9101.
+- `configs/capi/clusterctl.yaml.example`: comments on `TEMPLATE_VMID` / `ALLOWED_NODES`.
+- `docs/future-tasks.md`: report the three CA clusterapi bugs upstream. The namespace task now uses
+  the new MD names.
+- `README.md` (lab inventory, key decision 7), `CLAUDE.md` / `AGENTS.md` (lab facts):
+  PVE cluster, `pv1-2`, template 9101.
+- `.claude/settings.json`: two inspector allowlist rules `md lab1-workers` → `md lab1-workers-pv1-1`.
+- `CHANGELOG.md`: entry added after the owner confirmed the setup works (2026-09-30 11:05 CEST). The owner's pending #16/#17 work went in first as its own commit (`5e5b6c5`).