cluster-autoscaler-with-capi/docs/plans/2026-09-29-2315-pv1-2-workers.md

Plan: lab1 workers on both PVE nodes (template copy per node)

Created: 2026-09-29 23:15 CEST

Context

pv1-1 is now the PVE cluster homelab, and pv1-2 (192.168.0.10, 12 cores, 40.9 GiB) has joined it. There is no shared storage, and the owner doesn't want any for now. CAPMOX v0.9.1 can't clone across nodes from local-lvm, and templateSelector needs exactly one match in the whole cluster (session note). So each node needs its own template VMID, and workers get one MachineDeployment per PVE node.

Two blockers found through the PVE API:

• vmbr0 on pv1-2 is not VLAN-aware.
• Template 101 exists only on pv1-1. Its config is two local-lvm disks (scsi0, efidisk0), no ISO or local references, so an offline migration is possible.

Owner's choices:

• Copy template 101 rather than rebuilding it.
• Keep lab1-workers as it is and add lab1-workers-pv1-2.

Steps

Step 0 — Record the plan

Copy this plan to docs/plans/<date +%Y-%m-%d-%H%M>-pv1-2-workers.md, create the execution log docs/plans-executions/<same name> with the checklist, and commit both on their own to main.

Step 1 — vmbr0 on pv1-2 VLAN-aware (owner runs)

In /etc/network/interfaces on pv1-2, add to the iface vmbr0 stanza:

bridge-vlan-aware yes
bridge-vids 2-4094

Then run ifreload -a. The switch port for pv1-2 must carry VLAN 4 tagged, the same as pv1-1.

Verify (read-only, by me): GET /nodes/pv1-2/network/vmbr0 → bridge_vlan_aware: 1.

Step 2 — Template 9101 on pv1-2 (owner runs on pv1-1)

qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm  # the clone stays on pv1-1 (local storage)
qm migrate 9101 pv1-2 --with-local-disks                                        # offline: copies both LVs to pv1-2's local-lvm
ssh root@192.168.0.10 qm template 9101                                          # on pv1-2

• VMID 9101 is outside CAPMOX's nextid range (103+), next to the existing 9000/9001 templates.
• The name stays the same on purpose. CAPMOX addresses the template by templateID, not by name.

Verify:

• /cluster/resources: 9101 is on pv1-2 with template: 1.
• Smoke test from image-builder-proxmox.md, run on pv1-2 with TPL=9101. It also proves that VLAN 4 works on pv1-2 (DHCP lease, guest agent answers).

Step 3 — Second worker MachineDeployment (configs/capi/lab1.yaml)

• New ProxmoxMachineTemplate/lab1-worker-2g-pv1-2: a copy of lab1-worker-2g with:
  • sourceNode: pv1-2
  • templateID: 9101
  • allowedNodes: [pv1-2] (the machine-level list overrides the cluster's, vmscheduler.go:52-62)
• New MachineDeployment/lab1-workers-pv1-2: a copy of lab1-workers with:
  • the same KubeadmConfigTemplate lab1-worker and the same capacity annotations
  • the same rollout maxSurge 0 / maxUnavailable 1
  • min 0 / max 4
• lab1-workers: max-size annotation 5 → 3. Only an annotation changes, so no rollout. lab1-worker-2g is not edited; templates are never edited in place.
• ProxmoxCluster.allowedNodes stays [pv1-1]. The control plane (template 101) must not be scheduled onto pv1-2.
• IP pool check: .31–.39 has 9 addresses; 1 CP + 3 + 4 workers = 8, so one address is spare.
• Update the header comment: generate command and hand edit #10 (second MD, per-node template), plus #7 for the new max-size.

Owner runs:

kubectl diff -f configs/capi/lab1.yaml   # expect: new PMT + MD, max-size 5→3 on lab1-workers
kubectl apply -f configs/capi/lab1.yaml

• No change to cluster-autoscaler values: autoDiscovery picks up the new MD, and balance-similar-node-groups: true plus the least-waste expander spread scale-ups across the two groups.
• The RBAC rule for proxmoxmachinetemplates already covers the new template.

Step 4 — Scale test

1. Scale inflate so it needs about 4 workers (configs/test/inflate.yaml, 2 pods per worker).
2. Expect:
   • the CA log shows discovered node group: MachineDeployment/default/lab1-workers-pv1-2 (min: 0, max: 4…)
   • the scale-up is balanced across both MDs
   • the ProxmoxMachines land on pv1-1 / pv1-2 according to status.proxmoxNode
   • the nodes go Ready (Cilium)
3. Then scale down to 0 and confirm both groups shrink. Diagnosis goes through the cluster-inspector agent.

Step 5 — Docs and history

• docs/decisions.md:
  • new row #18: "Template copy per PVE node (101 on pv1-1, 9101 on pv1-2); one worker MD per node". Alternatives: shared storage, a rebuild per node, templateSelector. Status proposed until Step 4 passes, then active.
  • #7 → superseded by #18
• docs/phase-1-proxmox-capi.md:
  • topology table (two MDs, nodes)
  • Step 0: the storage/network paragraph (per-node template, VLAN-aware bridge on every node)
  • Step 5 annotations
  • troubleshooting rows:
    • clone to another node → "VM uses local storage"
    • VLAN not reachable on the new node
• docs/image-builder-proxmox.md: a new section "Template on other PVE nodes" with the Step 2 commands. The result line then names both VMIDs.
• configs/capi/clusterctl.yaml.example: fix the comments on ALLOWED_NODES / SOURCENODE.
• CLAUDE.md, AGENTS.md "Lab facts" and README line 29: PVE cluster homelab, pv1-2 192.168.0.10, templates 101/9101. These files have uncommitted owner edits, so I stage only my hunks (git add -p) or ask first.
• Session note: add the outcome and a link to the plan.
• CHANGELOG.md entry, only after the owner confirms Step 4 works.

Verification

• PVE API (read-only): vmbr0 VLAN-aware on both nodes; template 9101 on pv1-2.
• Smoke-test VM on pv1-2 gets a VLAN 4 address.
• clusterctl describe cluster lab1: both MDs are healthy.
• The Step 4 scale test puts workers on both PVE nodes, and they join and become Ready.
• Rollback: the owner deletes MachineDeployment/lab1-workers-pv1-2 and restores max 5 on lab1-workers. Template 9101 can stay.