Files

T

kacerrandClaude Opus 5.5 6ef55778ff lab1 workers on both PVE nodes: per-node template + MD, balanced CA

PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage:
- template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and
  lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50;
  lab1-workers renamed for symmetry (decision #18, #7 superseded)
- CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels
  with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19).
  Scale test: 16 pods -> 4 + 4 across both nodes.
- runbook, image-builder copy procedure, fixes tables, future task to report
  three CA clusterapi bugs upstream, lab facts, changelog, execution log

Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-09-30 11:07:17 +02:00

4.8 KiB

Raw Blame History

Session: CAPMOX against a two-node PVE cluster

Date: 2026-09-29 23:11 CEST

Question

pv1-1 became the PVE cluster homelab, and a second node pv1-2 joined it. Does the CAPI/CAPMOX configuration need changes to work against the cluster?

What was checked

PVE API (read-only GETs with the capmox@pve!capi token, 2026-09-29): /cluster/status, /nodes, /storage, /nodes/<n>/network, /nodes/<n>/storage, /cluster/resources, /access/acl.
CAPMOX v0.9.1 source (Go proxy zip): scheduler, clone path, template lookup, templates/cluster-template.yaml.
PVE qemu-server PVE/API2/Qemu.pm (HEAD, not pinned to a PVE release): the clone target check.

Findings

PVE state:

Item	pv1-1	pv1-2
IP	192.168.0.8	192.168.0.10
CPU / RAM	16 / 29.2 GiB	12 / 40.9 GiB
`vmbr0` VLAN-aware	yes	no
`local-lvm`	local LVM-thin	local LVM-thin (not shared)
Templates	101 `ubuntu-2404-kube-v1.35.9`, 9000, 9001	none

Storage definitions (local, local-lvm, proxy-seeds) are cluster-wide without a nodes restriction, but none of them is shared.
The capmox@pve ACLs (/, /storage, /sdn/zones/localnetwork) are cluster-wide, so they cover pv1-2 as well. There are no SDN zones.

CAPMOX v0.9.1:

Node choice. If the ProxmoxMachine has its own allowedNodes, that list replaces the ProxmoxCluster's (internal/service/scheduler/vmscheduler.go:52-62).
- The scheduler balances by how many machines of this cluster are already on each node (status.nodeLocations), and only picks a node where the requested memory fits (vmscheduler.go:75-119).
- memoryAdjustment is applied per node, against that node's own total memory.
- With empty allowedNodes the scheduler isn't called and the VM clones on the source node (internal/service/vmservice/vm.go:477).
Clone. target goes to the PVE clone call as-is (pkg/proxmox/goproxmox/api_client.go:84-95).
- PVE refuses a cross-node clone when any source disk is on non-shared storage: can't clone VM to node '<target>' (VM uses local storage).
- Setting full: true or storage does not change that.
- v0.9.1 has no localStorage option.
templateSelector searches the whole PVE cluster and needs exactly one match (api_client.go:149-212). "One template per node with the same tags" fails with found 2 VM templates.
API URL and credentials. Everything goes through one API URL; nothing in the credentials names a node.
- Cross-node requests should be forwarded by PVE (standard PVE behaviour, not checked in code).
- 192.168.0.8 is pv1-1, so a down pv1-1 also means no API for CAPMOX.
Bridge. bridge/vlan are written into netX without any check (vm.go:338).

Conclusion:

Nothing has to change for the current setup to keep working. lab1 pins allowedNodes: [pv1-1] and sourceNode: pv1-1, which is decision #7.
Using pv1-2 requires two things:
1. Make vmbr0 on pv1-2 VLAN-aware and trunk VLAN 4 on its switch port.
2. Either put a template copy on pv1-2, or add shared storage.
With a template copy per node:
- Workers get one ProxmoxMachineTemplate and one MachineDeployment per PVE node, each with sourceNode, templateID (the node's own VMID) and allowedNodes: [<node>]. For cluster-autoscaler that is one node group per PVE node.
- KubeadmControlPlane takes a single infrastructureRef, so the control plane stays on one node.

Open questions

How to get the template onto pv1-2: a second image-builder run (PROXMOX_NODE=pv1-2), or qm clone + qm migrate / vzdump-restore? The image-builder run is reproducible; the copy is faster.
Is shared storage (NFS) worth adding instead? It gives one MachineDeployment spread over both nodes, and the control plane can use both nodes too.
Behaviour when a cross-node clone fails: the scheduler may keep picking the same node. This is inferred from the code, not tested.

Follow-ups

The owner chose: no shared storage for now; copy template 101 to pv1-2 as 9101; keep lab1-workers and add lab1-workers-pv1-2. Plan: 2026-09-29-2315-pv1-2-workers. It covers the bridge fix, the template copy, the second MD, the CA limits and a decision row superseding #7.
Update the "Lab facts" in CLAUDE.md/AGENTS.md: PVE cluster homelab, pv1-2 at 192.168.0.10.

Outcome (2026-09-30)

Done as planned, and confirmed on the lab:

template 9101 on pv1-2;
MDs lab1-workers-pv1-1 (max 6) + lab1-workers-pv1-2 (max 12);
a balanced scale-up of 4 + 4 across both PVE nodes.

Two cluster-autoscaler findings from the scale test (no balancing, and throttling from uncached discovery) led to decision #19. Details are in the execution log.

4.8 KiB Raw Blame History