cluster-autoscaler-with-capi/docs/phase-1-proxmox-capi.md

Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler

Goal: a manually installed k3s management cluster runs CAPI + CAPMOX and creates workload cluster lab1 as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero. No custom code in this phase — only assembling upstream components.

Versions: see versions.md. Template build: see image-builder-proxmox.md.

Topology

Role	Where	Size
Management cluster	1 VM on Proxmox, k3s v1.35.8+k3s1	2 vCPU, 4 GB RAM, 30 GB
lab1 control plane	1 VM, VIP via kube-vip	2 vCPU, 4 GB RAM
lab1 workers	MachineDeployment lab1-workers, min 0 / max 5	4 vCPU, 8 GB RAM

Checklist

• Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
• Step 1 — VM template ubuntu-2404-kube-v1.35.9 built and smoke-tested
• Step 2 — management cluster + clusterctl init
• Step 3 — workload cluster lab1
• Step 4 — CNI (Cilium)
• Step 5 — MachineDeployment autoscaling annotations
• Step 6 — cluster-autoscaler
• Step 7 — scale tests

Step 0 — Proxmox prep

pveum user add capmox@pve
pveum user token add capmox@pve capi -privsep 0          # token inherits user ACLs

pveum aclmod /                       -user capmox@pve -role PVEVMAdmin
pveum aclmod /                       -user capmox@pve -role PVEAuditor        # Sys.Audit etc.
pveum aclmod /storage                -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks
pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser        # NIC on any local bridge (PVE 8+)

Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX docs/advanced-setups.md.

Network: vmbr0 is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP for the control plane endpoint in the same subnet.

Storage: local-lvm (LVM-thin) → disks must be raw, never qcow2. Cloning to another PVE node requires shared storage; with local storage keep ALLOWED_NODES to the node holding the template.

Step 2 — management cluster

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.8+k3s1 sh -s - --disable traefik
mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config

curl -L -o clusterctl https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-linux-amd64
sudo install -m 0755 clusterctl /usr/local/bin/clusterctl

~/.cluster-api/clusterctl.yaml — start from configs/capi/clusterctl.yaml.example.

clusterctl init \
  --core cluster-api:v1.12.11 \
  --bootstrap kubeadm:v1.12.11 \
  --control-plane kubeadm:v1.12.11 \
  --infrastructure proxmox:v0.9.1 \
  --ipam in-cluster:v1.1.1

Always pin versions explicitly — without them clusterctl installs the newest core (1.14.x) and breaks the tested CAPMOX combination.

Step 3 — workload cluster lab1

clusterctl generate cluster lab1 \
  --infrastructure proxmox:v0.9.1 \
  --kubernetes-version v1.35.9 \
  --control-plane-machine-count 1 \
  --worker-machine-count 1 > lab1.yaml

Edit lab1.yaml before applying:

1. Pod CIDR — Cluster.spec.clusterNetwork.pods.cidrBlocks defaults to 192.168.0.0/16, which overlaps the lab network → use e.g. 10.244.0.0/16.
2. kube-vip — bump ghcr.io/kube-vip/kube-vip:v0.7.1 → v1.2.4; keep kube-vip-prepare.sh (handles super-admin.conf for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
3. Disk format — delete format: "qcow2" from both ProxmoxMachineTemplates (LVM-thin).
4. Network — bridge: vmbr0 + vlan: 4 on the network device.
5. Worker size — ProxmoxMachineTemplate/lab1-worker: numSockets: 1, numCores: 4, memoryMiB: 8192.
6. Replicas — remove spec.replicas from MachineDeployment/lab1-workers; the autoscaler owns it.

kubectl apply -f lab1.yaml
clusterctl describe cluster lab1 --show-conditions all
kubectl get proxmoxmachines,machines -w
clusterctl get kubeconfig lab1 > lab1.kubeconfig

Node IPs come from the IPAM pool (kubectl get ipaddressclaims,ipaddresses). providerID is set by the CAPMOX template as proxmox://{{ ds.meta_data.instance_id }} — don't touch it, the autoscaler matches Nodes to Machines by it.

Step 4 — Cilium

export KUBECONFIG=$PWD/lab1.kubeconfig
helm repo add cilium https://helm.cilium.io && helm repo update
helm install cilium cilium/cilium --version 1.20.2 -n kube-system --set ipam.mode=kubernetes

ipam.mode=kubernetes uses the pod CIDR from clusterNetwork instead of Cilium's 10.0.0.0/8 default.

Step 5 — autoscaling annotations

metadata:
  name: lab1-workers
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "4"            # numSockets × numCores
    capacity.cluster-autoscaler.kubernetes.io/memory: "8Gi"
    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
    capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"

Capacity annotations are required for scale-from-zero because CAPMOX does not publish status.capacity. They must match the ProxmoxMachineTemplate; add .../labels and .../taints annotations if pods target the group by selector/toleration.

Step 6 — cluster-autoscaler

Install into the same namespace as the lab1-kubeconfig secret (here default), values in configs/cluster-autoscaler/values.yaml:

helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update
helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \
  -f configs/cluster-autoscaler/values.yaml

The log must show node group MachineDeployment/default/lab1-workers with min 0 / max 5.

Step 7 — scale tests

Workload: configs/test/inflate.yaml (pause pods, 1500m / 2Gi each).

• Scale-up: kubectl scale deploy/inflate --replicas=6 → workers 1 → 3; record pending → Running time.
• Scale-down: --replicas=0 → after scale-down-unneeded-time (5m) Machines and VMs disappear.
• Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
• Scale-from-zero: --replicas=2 → CA provisions from capacity annotations.

kubectl describe pod <pending> shows TriggeredScaleUp / NotTriggerScaleUp events — fastest diagnostic.

Troubleshooting

Symptom	Cause	Fix
CA sees no node group	missing min/max annotations or wrong autoDiscovery	check MD annotations, clusterName/namespace in values
CA never scales down	Node providerID ≠ Machine providerID	compare kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID with kubectl get machines
MD jumps back to old count	spec.replicas in manifest re-applied	remove replicas
last worker won't go to 0	kube-system pods without PDB (skip-nodes-with-system-pods=true)	PDB for CoreDNS or skip-nodes-with-system-pods: false
scale-from-zero does nothing	capacity annotations missing/wrong	fix cpu/memory/maximum-pods
clone fails unsupported format 'qcow2'	qcow2 on LVM-thin	remove format
ProxmoxMachine stuck without IP	IPAM pool exhausted	kubectl get ipaddressclaims, widen range
SDN.Use 403	missing PVESDNUser	ACL on /sdn/zones/localnetwork

kubectl -n capmox-system logs deploy/capmox-controller-manager -f
kubectl -n capi-system   logs deploy/capi-controller-manager -f
ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100'