cluster-autoscaler-with-capi/docs/plans-executions/2026-09-29-2315-pv1-2-workers.md

Execution log: lab1 workers on both PVE nodes (template copy per node)

Pairs with docs/plans/2026-09-29-2315-pv1-2-workers.md.

Status

• Step 0 — Record the plan
• Step 1 — vmbr0 on pv1-2 VLAN-aware (owner)
• Step 2 — Template 9101 on pv1-2 (owner)
• Step 3 — Second worker MachineDeployment lab1-workers-pv1-2
• Step 4 — Scale test across both nodes
• Step 5 — Docs and history

Step 0 — Record the plan (2026-09-29 23:15 CEST)

Plan and this log committed on their own, together with the session note that motivated them.

Starting state (read-only PVE API):

• cluster homelab: pv1-1 192.168.0.8, pv1-2 192.168.0.10
• vmbr0 on pv1-2 has no bridge_vlan_aware
• template 101 is only on pv1-1, with disks scsi0 + efidisk0, both on local-lvm
• nextid is 103

Steps 1–2 — bridge and template on pv1-2 (owner, verified 2026-09-30 08:38 CEST)

The owner made vmbr0 VLAN-aware on pv1-2 (/etc/network/interfaces, then ifreload -a):

bridge-vlan-aware yes
bridge-vids 2-10

This deviates from the plan: bridge-vids is 2-10, not 2-4094. That's enough for VLAN 4, and any VLAN above 10 on pv1-2 would need the range widened first.

Template copy, run on pv1-1. A cross-node clone is refused from local-lvm, so the copy is a clone on the same node followed by an offline migration:

qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm
qm migrate 9101 pv1-2 --with-local-disks
ssh root@192.168.0.10 qm template 9101

The PVE API shows 9101 on pv1-2 with template: 1, and scsi0/efidisk0 as local-lvm:base-9101-disk-*.

Smoke test (image-builder-proxmox.md) on pv1-2, with TPL=9101 ID=9901: the VM got a VLAN 4 DHCP address through the guest agent, and was destroyed afterwards. Before running it:

• /root/jan.pub had to be copied to pv1-2 first: scp ~/.ssh/id_ed25519.pub root@192.168.0.10:/root/jan.pub.
• The doc's ID=9001 now collides with debian-13-template, and the snippet ends with qm destroy $ID. The snippet now uses 9901 and checks that the ID is free (a9f5879).

Worth noting:

• VMIDs are cluster-wide (pmxcfs), which is why the copy needs its own ID. Names don't have to be unique.
• The copied efidisk0 reports size=528K (the source says 4M). This doesn't matter: the template boots SeaBIOS.

Step 3 — second worker MachineDeployment (prepared 2026-09-30, apply pending)

Changes in configs/capi/lab1.yaml:

• new MachineDeployment/lab1-workers-pv1-2 and ProxmoxMachineTemplate/lab1-worker-2g-pv1-2 (sourceNode: pv1-2, templateID: 9101, allowedNodes: [pv1-2])
• ProxmoxCluster.allowedNodes stays [pv1-1] for the control plane
• header hand edits #10 and #11

Deviation from the plan: the IP pool is enlarged and the max sizes are raised (owner's request, "20 nodes"):

• ipv4Config.addresses 192.168.4.31-192.168.4.39 → 192.168.4.31-192.168.4.50
• lab1-workers max 5 → 6, lab1-workers-pv1-2 max 12
• 6 + 12 + 1 CP = 19 of 20 addresses. The spare covers the KCP rollout surge, and CAPMOX pools hold a freed address for 60 s (AddressReuseGracePeriodSeconds, pkg/kubernetes/ipam/ipam.go:318).
• An ARP probe from lab-mgmt found .32–.60 silent; .30 (VIP) and .31 (CP) share the CP VM's MAC.
• The owner confirmed .31–.50 is outside the VLAN 4 DHCP range.
• Changing the pool is safe: the ProxmoxCluster webhook only validates the addresses, and the controller copies them into the InClusterIPPool on every reconcile (ipam.go:323-324).

Sizing:

• pv1-1: 6 × 2 GiB within ~17.7 GiB of reservable headroom (160 %).
• pv1-2: 12 × 2 GiB of 65.4 GiB reservable (40.9 GiB physical); 48 vCPU on 12 cores, which is fine for the pause-container inflate tests but CPU-bound under real load.

yq parse check: 9 documents; both worker PMTs point at the right node, template and allowedNodes.

Applied by the owner (kubectl diff / kubectl apply -f configs/capi/lab1.yaml). The CA discovered lab1-workers (max 6) and lab1-workers-pv1-2 (max 12).

Step 4 — scale test, first run (2026-09-30 ~06:45 UTC)

The inflate Deployment had 16 pods (1500m / 512Mi each, 2 per worker).

• Result: 8 workers Ready by 06:47Z.
  • lab1-workers 0→6: VMIDs 103–108 on pv1-1, IPs .34–.39.
  • lab1-workers-pv1-2 0→2: VMIDs 109–110 on pv1-2, IPs .32–.33.
  • No VMProvisionFailed. status.proxmoxNode and providerIDs are consistent. The pool had 9 of 20 addresses in use.
• Checked with: the cluster-inspector agent (read-only). Root causes came from the source (upstream-researcher, CA tag cluster-autoscaler-1.35.2, CAPI v1.12.11, CAPMOX v0.9.1).

Two problems.

1. Not balanced. The log shows Best option to resize: lab1-workers → No similar node groups found → 0->6 (max: 6). Only in the next loop did lab1-workers-pv1-2 get 0->2.

• The inspector guessed the cluster.x-k8s.io/deployment-name label. The source rules that out: CA never reads it.
• What the source does show in CA 1.35.2 cloudprovider/clusterapi:
  • Labels() reads spec.template.spec.metadata.labels, which doesn't exist in v1beta2 (clusterapi_unstructured.go:207). MD template labels therefore never reach the from-zero template node.
  • CAPMOX ProxmoxMachineTemplate has no status, so kubernetes.io/os and arch become "" on the template (clusterapi_nodegroup.go:393-405). A pod with nodeSelector: kubernetes.io/os: linux would not trigger a scale-up from zero.
  • The capacity key is maxPods; our maximum-pods was ignored, and 110 only applied because it is the default.
  • MixedTemplateNodeInfoProvider caches a template built from a real node for --node-info-cache-expire-time (default 87600h). A group that has had nodes keeps that template: ~1.78Gi allocatable, os/arch set, node-role label. A from-zero group gets the synthetic template: 2Gi, empty os/arch. The generic comparator rejects that pair.
  • That this cache caused this particular decision is inferred. The reason for a mismatch is only logged at V(5).
• Fix (lab1.yaml hand edit #12):
  • both MDs carry the template label node.cluster.x-k8s.io/pool: workers, which CAPI syncs to the Node;
  • the annotation capacity.cluster-autoscaler.kubernetes.io/labels: node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64;
  • maximum-pods → maxPods;
  • CA balancing-label: node.cluster.x-k8s.io/pool, which compares only that label and turns off the resource checks.

2. Loop 8–26 s, throttled.

• The log is full of Waited before sending request … client-side throttling on /api, /apis and /apis/infrastructure.cluster.x-k8s.io/v1alpha2 against the lab-mgmt API.
• Cause:
  • For every MD at 0 replicas, CanScaleFromZero → readInfrastructureReferenceResource → getKindPreferredVersion runs through the provider's uncached discovery client. The cached one is built at clusterapi_provider.go:195 but not passed to the controller.
  • That is 3 GETs per MD per NodeGroups() call, and NodeGroups() runs about 15× per loop.
  • The client uses QPS 5 / burst 10 (--kube-client-qps / --kube-client-burst defaults).
• There is no upstream fix up to 1.36.1 or master (2026-09-29).
• The owner rejected raising QPS (50/100) and wants fewer requests instead. Fix in configs/cluster-autoscaler/values.yaml:
  • scan-interval: 1m (was 10s). loop/trigger.go:74-100: unschedulable pods and finished scale-ups/downs still start a loop immediately. Only idle loops are spaced out, so scale-down takes up to about +1 min.
  • v: 2 (chart default 4). The throttling and "discovered node group" lines are V(3)/V(4).
• Considered and not taken:
  • min-size: 1: no MD at 0 means no discovery, but two idle VMs.
  • CAPI_VERSION=v1beta1: v1beta1 refs carry apiVersion, so no discovery. But v1beta1 is deprecated and the path is untested with CAPMOX.
  • An upstream patch to pass the cached discovery client (a future task).

Rename of the pv1-1 group (owner's request, 2026-09-30)

This deviates from the plan: the plan kept lab1-workers. The owner wants symmetric per-node names now that node names show the MD name (lab1-workers-zldvn-* next to lab1-workers-pv1-2-nlv6s-*).

Changes in configs/capi/lab1.yaml (hand edit #13):

• MachineDeployment/lab1-workers → lab1-workers-pv1-1.
• ProxmoxMachineTemplate/lab1-worker-2g → lab1-worker-2g-pv1-1, now with explicit allowedNodes: [pv1-1].

A rename creates new objects. kubectl apply doesn't prune, so the owner deletes the old MD and PMT once the old MD has scaled to 0. The timing is right because the CA was already scaling every worker down (inflate at 0). The new MD comes up with 0 replicas: replicas is unset, and the CAPI webhook defaults it to the autoscaler min-size annotation.

Step 4 — re-test after the fixes (2026-09-30 07:23 UTC)

The owner applied the changes in this order:

1. kubectl apply -f configs/capi/lab1.yaml: new lab1-workers-pv1-1 + lab1-worker-2g-pv1-1, the pool label, maxPods, os/arch.
2. Deleted the old MachineDeployment/lab1-workers and ProxmoxMachineTemplate/lab1-worker-2g.
3. helm upgrade --install ca … --version 9.59.0 -f configs/cluster-autoscaler/values.yaml: scan-interval 1m, v 2, balancing-label.
4. Scaled inflate 0 → 16.

The CA log (--since=15m):

orchestrator.go:752] Splitting scale-up between 2 similar node groups: {…/lab1-workers-pv1-2, …/lab1-workers-pv1-1}
orchestrator.go:263] Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}]

• Placement: ProxmoxMachines lab1-workers-pv1-1-* on pv1-1 (VMIDs 103, 104, 107, 110) and lab1-workers-pv1-2-* on pv1-2 (105, 106, 108, 109).
• Nodes: all 8 Ready about 1–1.5 min after creation, with labels node.cluster.x-k8s.io/pool=workers and kubernetes.io/os=linux.
• Loop timing: idle loops now run on the 1m0s timer.
• One transient warning: W clusterstate.go:510 Failed to find acceptable ranges for MachineDeployment/default/lab1-workers-pv1-1, once, in the loop during which the new MD was created. Acceptable ranges are snapshotted at the start of the loop, and the group appeared in a later NodeGroups() call. Harmless; it did not repeat.

Worth noting:

• ProxmoxMachineTemplate/lab1-worker (4 GiB, decision #15) is still in the cluster, and no MD references it. Deleting it is left to the owner.

Step 5 — docs and history (2026-09-30 09:28 CEST)

• docs/decisions.md: #7 → superseded by #18. #18 (template copy + MD per PVE node, pool .31–.50, symmetric names) and #19 (CA balancing-label, scan-interval 1m, v 2, QPS left at the default) are both active, since the lab confirmed them in Step 4.
• docs/phase-1-proxmox-capi.md:
  • intro and topology (PVE cluster, templates, both MDs, pool);
  • Step 0: bridge on every node, per-node template;
  • Step 5: annotations incl. maxPods, labels, the pool label;
  • Step 6: the settings for two groups, and what a balanced scale-up logs;
  • troubleshooting: 7 rows (maxPods, empty os/arch, no balancing, throttling, the acceptable-ranges warning, cross-node clone, VLAN on a new node).
• docs/image-builder-proxmox.md: section "Template on other PVE nodes"; the result line names 101/9101.
• configs/capi/clusterctl.yaml.example: comments on TEMPLATE_VMID / ALLOWED_NODES.
• docs/future-tasks.md: report the three CA clusterapi bugs upstream. The namespace task now uses the new MD names.
• README.md (lab inventory, key decision 7), CLAUDE.md / AGENTS.md (lab facts): PVE cluster, pv1-2, template 9101.
• .claude/settings.json: two inspector allowlist rules md lab1-workers → md lab1-workers-pv1-1.
• CHANGELOG.md: entry added after the owner confirmed the setup works (2026-09-30 11:05 CEST). The owner's pending #16/#17 work went in first as its own commit (5e5b6c5).