cluster-autoscaler-with-capi/docs/phase-1-proxmox-capi.md

Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler

Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload cluster lab1 as Proxmox VMs on a two-node PVE cluster; cluster-autoscaler scales its workers 0–18 in one node group per PVE node, balanced, including scale-from-zero. No custom code in this phase — only assembling upstream components.

Versions: see versions.md. Template build: see image-builder-proxmox.md.

Topology

Role	Where	Size
PVE cluster homelab	pv1-1 192.168.0.8 (API endpoint for CAPMOX), pv1-2 192.168.0.10; local-lvm on each, no shared storage	16 / 12 cores, 29.2 / 40.9 GiB
Node template	ubuntu-2404-kube-v1.35.9: VMID 101 on pv1-1, copy 9101 on pv1-2	20 GB
Management cluster	VM 201 lab-mgmt (192.168.4.21) on pv1-1, k3s v1.35.8+k3s1	2 vCPU, 4 GB RAM, 30 GB
lab1 control plane	1 VM on pv1-1, VIP .30 via kube-vip	2 vCPU, 4 GB RAM
lab1 workers	MachineDeployments lab1-workers-pv1-1 (min 0 / max 6) and lab1-workers-pv1-2 (min 0 / max 12); node IP pool .31–.50	4 vCPU, 2 GB RAM

Checklist

• Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
• Step 1 — VM template ubuntu-2404-kube-v1.35.9 built and smoke-tested
• Step 2 — management cluster (ansible/mgmt.yml: k3s + cluster-api-operator + providers)
• Step 3 — workload cluster lab1 (VMs 102/103, .31/.32, VIP .30)
• Step 4 — CNI (Cilium 1.20.2, configs/cilium/values.yaml)
• Step 5 — MachineDeployment autoscaling annotations
• Step 6 — cluster-autoscaler
• Step 7 — scale tests

Step 0 — Proxmox prep

pveum user add capmox@pve
pveum user token add capmox@pve capi -privsep 0          # token inherits user ACLs

pveum aclmod /                       -user capmox@pve -role PVEVMAdmin
pveum aclmod /                       -user capmox@pve -role PVEAuditor        # Sys.Audit etc.
pveum aclmod /storage                -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks
pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser        # NIC on any local bridge (PVE 8+)

Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX docs/advanced-setups.md.

The token and ACLs are cluster-wide in a PVE cluster, so a node that joins later needs nothing here.

Network: vmbr0 is VLAN-aware on every PVE node, and VMs live in VLAN 4. On a new node, add bridge-vlan-aware yes + bridge-vids 2-10 (or wider) to iface vmbr0, run ifreload -a, and trunk VLAN 4 on its switch port. CAPMOX writes bridge/vlan into the VM without checking them. Reserve a node IP range outside DHCP and one VIP for the control plane endpoint in the same subnet.

Storage: local-lvm (LVM-thin) → disks must be raw, never qcow2.

• PVE refuses to clone a VM to another node from non-shared storage, and CAPMOX v0.9.1 always passes the scheduled node as the clone target.
• templateSelector needs exactly one match in the whole PVE cluster, so it can't pick a per-node copy.
• Without shared storage, therefore, every node gets its own template VMID (VMIDs are cluster-wide) and its own worker MachineDeployment. Each MD's ProxmoxMachineTemplate sets sourceNode, templateID and allowedNodes: [<node>].
• ProxmoxCluster.allowedNodes stays [pv1-1] for the control plane. The copy procedure is in image-builder-proxmox.md, and the reasoning in decision #18.

Step 2 — management cluster

One idempotent playbook, run from the laptop: ansible/mgmt.yml. It creates the VM on pv1-1 from the pinned Ubuntu 24.04 cloud image, installs k3s, then cert-manager and cluster-api-operator (as k3s HelmCharts), then the five CAPI providers as operator Provider CRs. Versions and VM parameters are in ansible/group_vars/all.yml. A re-run must report changed=0.

Laptop prerequisites:

• root SSH to pv1-1 with ~/.ssh/id_ed25519 (ssh-copy-id root@192.168.0.8);

• ansible-core 2.18 (brew install ansible, or prefix commands with uvx --from 'ansible-core>=2.18,<2.19');

• cp ansible/secrets.yml.example ansible/secrets.yml and fill in the capmox@pve!capi token secret;

• clusterctl v1.12.11 for Step 3 (not brew install clusterctl, which installs 1.14.x):

  curl -fL -o /tmp/clusterctl \
    https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-darwin-arm64
  install -m 0755 /tmp/clusterctl ~/bin/clusterctl && rm /tmp/clusterctl
  

cd ansible
ansible-playbook mgmt.yml --tags vm           # VM 201 on pv1-1, waits for SSH
ansible-playbook mgmt.yml --tags k3s          # qemu-guest-agent, k3s, node Ready
ansible-playbook mgmt.yml --tags addons       # cert-manager v1.21.2, cluster-api-operator 0.28.0
ansible-playbook mgmt.yml --tags capi         # capmox-variables Secret + 5 Provider CRs, waits for controllers
ansible-playbook mgmt.yml --tags kubeconfig   # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfigs
ansible-playbook mgmt.yml --tags inspector    # only the read-only kubeconfigs (lab-mgmt + inspector_workload_clusters); weekly token refresh
ansible-playbook mgmt.yml                     # everything; must be changed=0 on a built cluster

export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A
# all READY True, INSTALLEDVERSION v1.12.11 / v0.9.1 / v1.1.1

cluster-api-operator replaces clusterctl init:

• Upgrade a provider by bumping its version in group_vars/all.yml and re-running --tags capi, one provider per run.
• Use clusterctl only for generate cluster (always with --infrastructure proxmox:v0.9.1), describe and get kubeconfig. Never init, upgrade, move or delete: they need the clusterctl inventory, which the operator doesn't create.
• Proxmox credentials live only in ansible/secrets.yml; ~/.cluster-api/clusterctl.yaml (from configs/capi/clusterctl.yaml.example) holds just the workload variables.

Step 3 — workload cluster lab1

The generated and hand-edited manifest is committed as configs/capi/lab1.yaml: template VMID 101, control-plane VIP 192.168.4.30, node IPs 192.168.4.31–39. Its header holds the exact clusterctl generate command (all variables as env vars, so no ~/.cluster-api/clusterctl.yaml is needed). Regenerate only when a version changes, then redo these edits:

1. Pod CIDR — Cluster.spec.clusterNetwork.pods.cidrBlocks defaults to 192.168.0.0/16, which overlaps the lab network → use e.g. 10.244.0.0/16.
2. kube-vip — bump ghcr.io/kube-vip/kube-vip:v0.7.1 → v1.2.4; keep kube-vip-prepare.sh (handles super-admin.conf for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
3. Disk format — delete format: "qcow2" from both ProxmoxMachineTemplates (LVM-thin).
4. Network — bridge: vmbr0 + vlan: 4 on the network device.
5. Worker size — rename the worker template to ProxmoxMachineTemplate/lab1-worker-2g (and the MD infrastructureRef): numSockets: 1, numCores: 4, memoryMiB: 2048. To resize later, add a template with a new name and point the MD at it; CAPI then rolls the workers. Don't edit a template in place.
6. Replicas — remove spec.replicas from MachineDeployment/lab1-workers; the autoscaler owns it.

export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl apply -f configs/capi/lab1.yaml
clusterctl describe cluster lab1 --show-conditions all
kubectl get proxmoxmachines,machines -w
clusterctl get kubeconfig lab1 > lab1.kubeconfig

Node IPs come from the IPAM pool (kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io; plain ipaddresses resolves to the built-in networking.k8s.io type on k8s ≥ 1.33). providerID is set by the CAPMOX template as proxmox://{{ ds.meta_data.instance_id }} — don't touch it, the autoscaler matches Nodes to Machines by it.

Step 4 — Cilium

Values in configs/cilium/values.yaml; upgrade --install is safe to re-run.

clusterctl get kubeconfig lab1 --kubeconfig ~/.kube/lab-mgmt.kubeconfig > ~/.kube/lab1.kubeconfig
chmod 600 ~/.kube/lab1.kubeconfig
export KUBECONFIG=~/.kube/lab1.kubeconfig

helm repo add cilium https://helm.cilium.io && helm repo update cilium
helm upgrade --install cilium cilium/cilium --version 1.20.2 -n kube-system \
  -f configs/cilium/values.yaml --wait --timeout 10m

kubectl get nodes                                          # both Ready
kubectl -n kube-system get pods -o wide                    # cilium DaemonSet 2/2, operator 1/1, coredns Running

• ipam.mode=kubernetes uses the pod CIDR from clusterNetwork instead of Cilium's 10.0.0.0/8 default.
• operator.replicas: 1 + operator.nodeSelector on node-role.kubernetes.io/control-plane: the default 2 replicas with node anti-affinity put one on a worker, and a kube-system pod without a PDB blocks scale-down to zero. replicas: 1 alone isn't enough: the single replica was scheduled onto the worker (the chart tolerates the control-plane taint, so the selector pins it there).
• kube-proxy stays (kubeadm default); no kube-proxy replacement in this phase.

Step 5 — autoscaling annotations

Both worker MDs carry the same block. Only the name and max-size differ (6 on pv1-1, 12 on pv1-2):

metadata:
  name: lab1-workers-pv1-2
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "4"            # numSockets × numCores
    capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi"       # memoryMiB 2048
    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
    capacity.cluster-autoscaler.kubernetes.io/maxPods: "110"      # not "maximum-pods" (ignored)
    capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
spec:
  template:
    metadata:
      labels:
        node.cluster.x-k8s.io/pool: workers    # synced to the Node by CAPI; CA --balancing-label

They are part of both MDs in configs/capi/lab1.yaml. Re-apply the file. replicas isn't in the manifest, so it stays as it is, and a new MD starts at the min-size:

export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl diff -f configs/capi/lab1.yaml
kubectl apply -f configs/capi/lab1.yaml

Scale-from-zero needs capacity annotations because CAPMOX does not publish status.capacity, and they must match the ProxmoxMachineTemplate. The .../labels annotation is needed as well, for two reasons in CA 1.35.2:

• It reads MD template labels from spec.template.spec.metadata.labels, a path that doesn't exist in CAPI v1beta2.
• CAPMOX publishes no status.nodeInfo, so the from-zero template node would otherwise get kubernetes.io/os="" and arch="". A pod with an os: linux selector then can't trigger a scale-up.

Add a .../taints annotation if pods target the group by toleration. None of this has any effect until cluster-autoscaler runs (Step 6).

Step 6 — cluster-autoscaler

Install into the same namespace as the lab1-kubeconfig secret (here default), values in configs/cluster-autoscaler/values.yaml:

export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler \
  --version 9.59.0 -n default -f configs/cluster-autoscaler/values.yaml

The values add rbac.additionalRules to read proxmoxmachinetemplates. The chart's ClusterRole doesn't cover infrastructure templates, and without them the first main loop hangs (see troubleshooting). Health check: the log shows Starting main loop followed by regular Autoscaler loop triggered lines, and cluster-autoscaler-status on lab1 leaves Initializing.

CA runs on the management cluster as Deployment ca-clusterapi-cluster-autoscaler.

Settings in the values that matter with two node groups:

• balancing-label: node.cluster.x-k8s.io/pool. CA compares groups only on this label, so a scale-up is split across both PVE nodes.
• scan-interval: 1m. Unschedulable pods and finished scale-ups/downs still start a loop immediately; only the idle scale-down evaluation runs less often.
• v: 2.

Checking the node groups: discovered node group is only logged at v: 4 (clusterapi_controller.go:800). At v: 2, check the node groups in the status ConfigMap (below) instead. A balanced scale-up logs:

Splitting scale-up between 2 similar node groups: {MachineDeployment/default/lab1-workers-pv1-2, MachineDeployment/default/lab1-workers-pv1-1}
Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}]

The status ConfigMap and the leader-election lock live on the workload cluster, not on the management cluster: kubectl get cm cluster-autoscaler-status -n kube-system --kubeconfig ~/.kube/lab1.kubeconfig (chart arg --namespace=kube-system, because clusterAPIConfigMapsNamespace is empty).

Step 7 — scale tests

Workload: configs/test/inflate.yaml (pause pods, 1500m / 512Mi each → CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocatable).

• Scale-up: kubectl scale deploy/inflate --replicas=6 → workers 1 → 3; record pending → Running time.
• Scale-down: --replicas=0 → after scale-down-delay-after-add (2m) + scale-down-unneeded-time (1m) Machines and VMs disappear. That's the aggressive lab profile in the CA values; per-MD overrides are cluster.x-k8s.io/autoscaling-options-{scaledownunneededtime,scaledownutilizationthreshold,…} annotations.
• Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
• Scale-from-zero: --replicas=2 → CA provisions from capacity annotations.

kubectl describe pod <pending> shows TriggeredScaleUp / NotTriggerScaleUp events — fastest diagnostic.

Troubleshooting

Symptom	Cause	Fix
CA sees no node group	missing min/max annotations or wrong autoDiscovery	check MD annotations, clusterName/namespace in values
CA runs and discovers the node group but never scales; log stops at waiting for cache sync on infrastructure resource, proxmoxmachinetemplates … forbidden every ~40 s; status ConfigMap stuck on Initializing	chart 9.59.0 ClusterRole has no rule for infrastructure.cluster.x-k8s.io; CA's first main loop waits for a template informer that can never sync	rbac.additionalRules with get/list/watch on proxmoxmachinetemplates in the CA values, helm upgrade
CA never scales down	Node providerID ≠ Machine providerID	compare kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID with kubectl get machines
MD jumps back to old count	spec.replicas in manifest re-applied	remove replicas
scale-up adds one node too many (6 pods at 2 per node → 4 workers, second plan 3->4 about 70 s after 1->3)	a new node registers with Cilium's node.cilium.io/agent-not-ready:NoSchedule; once registered it's no longer an "upcoming" template node, and CA sees pending pods that don't fit on it	extraArgs.startup-taint: node.cilium.io/agent-not-ready in the CA values (CA treats the node as still starting)
CA scales the MD up, new Machines stay without a VM; ProxmoxMachine VMProvisionFailed: cannot reserve 4294967296B of memory on node pv1-1: …B available memory left although free shows GBs available	CAPMOX v0.9.1 scheduler (GetReservableMemoryBytes): node total × schedulerHints.memoryAdjustment (default 100%) minus the configured memory of every non-template VM, running or stopped, and every container. Stopped VMs 100/1001/2001 hold 12 GiB	free the reservation (delete unused stopped VMs, or convert them to templates), or set ProxmoxCluster.spec.schedulerHints.memoryAdjustment > 100 (overcommit); keep the MD max-size within what the host can actually run
MD rollout never finishes and CA never scales down: CA logs Node … will be skipped as it is currently under rollout, new ProxmoxMachine cannot reserve …B of memory	deadlock. Default rollout surge (maxSurge 1 / maxUnavailable 0) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (ScaleDownNodeUpgradeProcessor)	spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1} on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory
last worker won't go to 0	kube-system pods without PDB (skip-nodes-with-system-pods=true)	PDB for CoreDNS or skip-nodes-with-system-pods: false
cilium-operator runs on a worker despite operator.replicas: 1	chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker	operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""} in configs/cilium/values.yaml, helm upgrade
scale-from-zero does nothing	capacity annotations missing/wrong	fix cpu/memory/maxPods (maximum-pods is not a CA key and is silently ignored)
a pod with nodeSelector: kubernetes.io/os: linux doesn't trigger scale-from-zero	CA 1.35.2 builds the template node's os/arch labels from the infra template's status.nodeInfo. CAPMOX has none, so they become "". MD template labels aren't read either (CA reads the non-existent v1beta2 path spec.template.spec.metadata.labels)	capacity.cluster-autoscaler.kubernetes.io/labels: "kubernetes.io/os=linux,kubernetes.io/arch=amd64,…" on the MD
scale-up goes into one group only: No similar node groups found, next loop the second group (6 + 2 instead of 4 + 4)	the generic comparator fails when one group's template comes from a real node (cached for --node-info-cache-expire-time, default 10 years: ~1.78 Gi allocatable, os/arch set) and the other from the synthetic from-zero template (2 Gi, os/arch empty). The mismatch reason is only logged at V(5). cluster.x-k8s.io/deployment-name is not the cause, because CA never reads it	same label on both MDs (node.cluster.x-k8s.io/pool: template label + capacity labels annotation) and balancing-label: node.cluster.x-k8s.io/pool in the CA values. This turns off the resource comparison, so keep the MDs' capacity identical
CA loop takes 8–26 s; the log is full of Waited before sending request … client-side throttling on /api, /apis, /apis/infrastructure.cluster.x-k8s.io/v1alpha2	the CA 1.35.2 clusterapi provider runs uncached discovery (getKindPreferredVersion) for every MD at 0 replicas, on every NodeGroups() call (about 15× per loop). The client-go default is 5 QPS / burst 10. There is no upstream fix up to 1.36.1	fewer loops: scan-interval: 1m (scale-up is still triggered by pending pods at once). Alternatively kube-client-qps/-burst (more requests, not chosen). v: 2 hides the noise. Upstream fix: future tasks
W clusterstate.go:510 Failed to find acceptable ranges for MachineDeployment/…	the MD was created during a loop: acceptable ranges are snapshotted at the start of the loop, and the group showed up in a later NodeGroups() call	none if it happens once. If it repeats every loop, the group is dropping out of NodeGroups() (scale-from-zero check failing: infra template unreadable, capacity annotations missing)
ProxmoxMachine VMProvisionFailed … can't clone VM to node '<node>' (VM uses local storage)	the scheduled node (allowedNodes) isn't the node holding the template, and the template is on non-shared storage. A cross-node clone is then impossible, whatever full/storage are set to	per-node template copy + per-node MD with allowedNodes: [<node>] (see Step 0); or shared storage
VMs on a new PVE node get no IP (no DHCP/IPAM connectivity on VLAN 4)	vmbr0 on that node isn't VLAN-aware, or its switch port doesn't trunk VLAN 4	bridge-vlan-aware yes + bridge-vids in /etc/network/interfaces, ifreload -a; check via pvesh get /nodes/<node>/network/vmbr0
clone fails unsupported format 'qcow2'	qcow2 on LVM-thin	remove format
ProxmoxMachine stuck without IP	IPAM pool exhausted	kubectl get ipaddressclaims, widen range
SDN.Use 403	missing PVESDNUser	ACL on /sdn/zones/localnetwork
Provider CR not READY / capi play keeps retrying	operator preflight or component download failed	kubectl -n capi-operator-system logs deploy/cluster-api-operator, kubectl -n <ns> describe <kind> <name>
Provider keeps broken components after a failed fetch	stale per-version cache (cluster-api-operator #1075)	delete that provider's …-cache Secret, re-run --tags capi
CAPMOX still uses the old token after editing secrets.yml	operator doesn't watch configSecret	new Secret name in the template + configSecret reference
ansible-playbook mgmt.yml reports changed on a built cluster	a task isn't idempotent	-v for command output; on the VM k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml

kubectl -n capmox-system logs deploy/capmox-controller-manager -f
kubectl -n capi-system   logs deploy/capi-controller-manager -f
ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100'