cluster-autoscaler-with-capi/docs/phase-1-proxmox-capi.md

Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler

Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload cluster lab1 as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero. No custom code in this phase — only assembling upstream components.

Versions: see versions.md. Template build: see image-builder-proxmox.md.

Topology

Role	Where	Size
Management cluster	VM 201 lab-mgmt (192.168.4.21) on pv1-1, k3s v1.35.8+k3s1	2 vCPU, 4 GB RAM, 30 GB
lab1 control plane	1 VM, VIP via kube-vip	2 vCPU, 4 GB RAM
lab1 workers	MachineDeployment lab1-workers, min 0 / max 5	4 vCPU, 2 GB RAM

Checklist

• Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
• Step 1 — VM template ubuntu-2404-kube-v1.35.9 built and smoke-tested
• Step 2 — management cluster (ansible/mgmt.yml: k3s + cluster-api-operator + providers)
• Step 3 — workload cluster lab1 (VMs 102/103, .31/.32, VIP .30)
• Step 4 — CNI (Cilium 1.20.2, configs/cilium/values.yaml)
• Step 5 — MachineDeployment autoscaling annotations
• Step 6 — cluster-autoscaler
• Step 7 — scale tests

Step 0 — Proxmox prep

pveum user add capmox@pve
pveum user token add capmox@pve capi -privsep 0          # token inherits user ACLs

pveum aclmod /                       -user capmox@pve -role PVEVMAdmin
pveum aclmod /                       -user capmox@pve -role PVEAuditor        # Sys.Audit etc.
pveum aclmod /storage                -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks
pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser        # NIC on any local bridge (PVE 8+)

Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX docs/advanced-setups.md.

Network: vmbr0 is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP for the control plane endpoint in the same subnet.

Storage: local-lvm (LVM-thin) → disks must be raw, never qcow2. Cloning to another PVE node requires shared storage; with local storage keep ALLOWED_NODES to the node holding the template.

Step 2 — management cluster

One idempotent playbook, run from the laptop: ansible/mgmt.yml. It creates the VM on pv1-1 from the pinned Ubuntu 24.04 cloud image, installs k3s, then cert-manager and cluster-api-operator (as k3s HelmCharts), then the five CAPI providers as operator Provider CRs. Versions and VM parameters are in ansible/group_vars/all.yml. A re-run must report changed=0.

Laptop prerequisites:

• root SSH to pv1-1 with ~/.ssh/id_ed25519 (ssh-copy-id root@192.168.0.8);

• ansible-core 2.18 (brew install ansible, or prefix commands with uvx --from 'ansible-core>=2.18,<2.19');

• cp ansible/secrets.yml.example ansible/secrets.yml and fill in the capmox@pve!capi token secret;

• clusterctl v1.12.11 for Step 3 (not brew install clusterctl, which installs 1.14.x):

  curl -fL -o /tmp/clusterctl \
    https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-darwin-arm64
  install -m 0755 /tmp/clusterctl ~/bin/clusterctl && rm /tmp/clusterctl
  

cd ansible
ansible-playbook mgmt.yml --tags vm           # VM 201 on pv1-1, waits for SSH
ansible-playbook mgmt.yml --tags k3s          # qemu-guest-agent, k3s, node Ready
ansible-playbook mgmt.yml --tags addons       # cert-manager v1.21.2, cluster-api-operator 0.28.0
ansible-playbook mgmt.yml --tags capi         # capmox-variables Secret + 5 Provider CRs, waits for controllers
ansible-playbook mgmt.yml --tags kubeconfig   # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfigs
ansible-playbook mgmt.yml --tags inspector    # only the read-only kubeconfigs (lab-mgmt + inspector_workload_clusters); weekly token refresh
ansible-playbook mgmt.yml                     # everything; must be changed=0 on a built cluster

export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A
# all READY True, INSTALLEDVERSION v1.12.11 / v0.9.1 / v1.1.1

cluster-api-operator replaces clusterctl init:

• Upgrade a provider by bumping its version in group_vars/all.yml and re-running --tags capi, one provider per run.
• Use clusterctl only for generate cluster (always with --infrastructure proxmox:v0.9.1), describe and get kubeconfig. Never init, upgrade, move or delete: they need the clusterctl inventory, which the operator doesn't create.
• Proxmox credentials live only in ansible/secrets.yml; ~/.cluster-api/clusterctl.yaml (from configs/capi/clusterctl.yaml.example) holds just the workload variables.

Step 3 — workload cluster lab1

The generated and hand-edited manifest is committed as configs/capi/lab1.yaml: template VMID 101, control-plane VIP 192.168.4.30, node IPs 192.168.4.31–39. Its header holds the exact clusterctl generate command (all variables as env vars, so no ~/.cluster-api/clusterctl.yaml is needed). Regenerate only when a version changes, then redo these edits:

1. Pod CIDR — Cluster.spec.clusterNetwork.pods.cidrBlocks defaults to 192.168.0.0/16, which overlaps the lab network → use e.g. 10.244.0.0/16.
2. kube-vip — bump ghcr.io/kube-vip/kube-vip:v0.7.1 → v1.2.4; keep kube-vip-prepare.sh (handles super-admin.conf for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
3. Disk format — delete format: "qcow2" from both ProxmoxMachineTemplates (LVM-thin).
4. Network — bridge: vmbr0 + vlan: 4 on the network device.
5. Worker size — rename the worker template to ProxmoxMachineTemplate/lab1-worker-2g (and the MD infrastructureRef): numSockets: 1, numCores: 4, memoryMiB: 2048. To resize later, add a template with a new name and point the MD at it; CAPI then rolls the workers. Don't edit a template in place.
6. Replicas — remove spec.replicas from MachineDeployment/lab1-workers; the autoscaler owns it.

export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl apply -f configs/capi/lab1.yaml
clusterctl describe cluster lab1 --show-conditions all
kubectl get proxmoxmachines,machines -w
clusterctl get kubeconfig lab1 > lab1.kubeconfig

Node IPs come from the IPAM pool (kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io; plain ipaddresses resolves to the built-in networking.k8s.io type on k8s ≥ 1.33). providerID is set by the CAPMOX template as proxmox://{{ ds.meta_data.instance_id }} — don't touch it, the autoscaler matches Nodes to Machines by it.

Step 4 — Cilium

Values in configs/cilium/values.yaml; upgrade --install is safe to re-run.

clusterctl get kubeconfig lab1 --kubeconfig ~/.kube/lab-mgmt.kubeconfig > ~/.kube/lab1.kubeconfig
chmod 600 ~/.kube/lab1.kubeconfig
export KUBECONFIG=~/.kube/lab1.kubeconfig

helm repo add cilium https://helm.cilium.io && helm repo update cilium
helm upgrade --install cilium cilium/cilium --version 1.20.2 -n kube-system \
  -f configs/cilium/values.yaml --wait --timeout 10m

kubectl get nodes                                          # both Ready
kubectl -n kube-system get pods -o wide                    # cilium DaemonSet 2/2, operator 1/1, coredns Running

• ipam.mode=kubernetes uses the pod CIDR from clusterNetwork instead of Cilium's 10.0.0.0/8 default.
• operator.replicas: 1 + operator.nodeSelector on node-role.kubernetes.io/control-plane: the default 2 replicas with node anti-affinity put one on a worker, and a kube-system pod without a PDB blocks scale-down to zero. replicas: 1 alone isn't enough: the single replica was scheduled onto the worker (the chart tolerates the control-plane taint, so the selector pins it there).
• kube-proxy stays (kubeadm default); no kube-proxy replacement in this phase.

Step 5 — autoscaling annotations

metadata:
  name: lab1-workers
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "4"            # numSockets × numCores
    capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi"       # memoryMiB 2048
    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
    capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"

They are part of MachineDeployment/lab1-workers in configs/capi/lab1.yaml; re-apply the file (the only diff is these annotations; replicas isn't in the manifest, so it stays as is):

export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl diff -f configs/capi/lab1.yaml       # expect only the six annotations on lab1-workers
kubectl apply -f configs/capi/lab1.yaml

Capacity annotations are required for scale-from-zero because CAPMOX does not publish status.capacity. They must match the ProxmoxMachineTemplate; add .../labels and .../taints annotations if pods target the group by selector/toleration. They have no effect until cluster-autoscaler runs (Step 6).

Step 6 — cluster-autoscaler

Install into the same namespace as the lab1-kubeconfig secret (here default), values in configs/cluster-autoscaler/values.yaml:

export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler \
  --version 9.59.0 -n default -f configs/cluster-autoscaler/values.yaml

The values add rbac.additionalRules to read proxmoxmachinetemplates. The chart's ClusterRole doesn't cover infrastructure templates, and without them the first main loop hangs (see troubleshooting). Health check: the log shows Starting main loop followed by regular Autoscaler loop triggered lines, and cluster-autoscaler-status on lab1 leaves Initializing.

CA runs on the management cluster as Deployment ca-clusterapi-cluster-autoscaler. Its log must show the node group; the line is logged at v: 4 on every loop (clusterapi_controller.go, CA 1.35.2):

kubectl logs deploy/ca-clusterapi-cluster-autoscaler -n default --kubeconfig ~/.kube/lab-mgmt.kubeconfig \
  | grep 'discovered node group'
# discovered node group: MachineDeployment/default/lab1-workers (min: 0, max: 5, replicas: 1)

The status ConfigMap and the leader-election lock live on the workload cluster, not on the management cluster: kubectl get cm cluster-autoscaler-status -n kube-system --kubeconfig ~/.kube/lab1.kubeconfig (chart arg --namespace=kube-system, because clusterAPIConfigMapsNamespace is empty).

Step 7 — scale tests

Workload: configs/test/inflate.yaml (pause pods, 1500m / 512Mi each → CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocatable).

• Scale-up: kubectl scale deploy/inflate --replicas=6 → workers 1 → 3; record pending → Running time.
• Scale-down: --replicas=0 → after scale-down-delay-after-add (2m) + scale-down-unneeded-time (1m) Machines and VMs disappear. That's the aggressive lab profile in the CA values; per-MD overrides are cluster.x-k8s.io/autoscaling-options-{scaledownunneededtime,scaledownutilizationthreshold,…} annotations.
• Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
• Scale-from-zero: --replicas=2 → CA provisions from capacity annotations.

kubectl describe pod <pending> shows TriggeredScaleUp / NotTriggerScaleUp events — fastest diagnostic.

Troubleshooting

Symptom	Cause	Fix
CA sees no node group	missing min/max annotations or wrong autoDiscovery	check MD annotations, clusterName/namespace in values
CA runs and discovers the node group but never scales; log stops at waiting for cache sync on infrastructure resource, proxmoxmachinetemplates … forbidden every ~40 s; status ConfigMap stuck on Initializing	chart 9.59.0 ClusterRole has no rule for infrastructure.cluster.x-k8s.io; CA's first main loop waits for a template informer that can never sync	rbac.additionalRules with get/list/watch on proxmoxmachinetemplates in the CA values, helm upgrade
CA never scales down	Node providerID ≠ Machine providerID	compare kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID with kubectl get machines
MD jumps back to old count	spec.replicas in manifest re-applied	remove replicas
scale-up adds one node too many (6 pods at 2 per node → 4 workers, second plan 3->4 about 70 s after 1->3)	a new node registers with Cilium's node.cilium.io/agent-not-ready:NoSchedule; once registered it's no longer an "upcoming" template node, and CA sees pending pods that don't fit on it	extraArgs.startup-taint: node.cilium.io/agent-not-ready in the CA values (CA treats the node as still starting)
CA scales the MD up, new Machines stay without a VM; ProxmoxMachine VMProvisionFailed: cannot reserve 4294967296B of memory on node pv1-1: …B available memory left although free shows GBs available	CAPMOX v0.9.1 scheduler (GetReservableMemoryBytes): node total × schedulerHints.memoryAdjustment (default 100%) minus the configured memory of every non-template VM, running or stopped, and every container. Stopped VMs 100/1001/2001 hold 12 GiB	free the reservation (delete unused stopped VMs, or convert them to templates), or set ProxmoxCluster.spec.schedulerHints.memoryAdjustment > 100 (overcommit); keep the MD max-size within what the host can actually run
MD rollout never finishes and CA never scales down: CA logs Node … will be skipped as it is currently under rollout, new ProxmoxMachine cannot reserve …B of memory	deadlock. Default rollout surge (maxSurge 1 / maxUnavailable 0) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (ScaleDownNodeUpgradeProcessor)	spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1} on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory
last worker won't go to 0	kube-system pods without PDB (skip-nodes-with-system-pods=true)	PDB for CoreDNS or skip-nodes-with-system-pods: false
cilium-operator runs on a worker despite operator.replicas: 1	chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker	operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""} in configs/cilium/values.yaml, helm upgrade
scale-from-zero does nothing	capacity annotations missing/wrong	fix cpu/memory/maximum-pods
clone fails unsupported format 'qcow2'	qcow2 on LVM-thin	remove format
ProxmoxMachine stuck without IP	IPAM pool exhausted	kubectl get ipaddressclaims, widen range
SDN.Use 403	missing PVESDNUser	ACL on /sdn/zones/localnetwork
Provider CR not READY / capi play keeps retrying	operator preflight or component download failed	kubectl -n capi-operator-system logs deploy/cluster-api-operator, kubectl -n <ns> describe <kind> <name>
Provider keeps broken components after a failed fetch	stale per-version cache (cluster-api-operator #1075)	delete that provider's …-cache Secret, re-run --tags capi
CAPMOX still uses the old token after editing secrets.yml	operator doesn't watch configSecret	new Secret name in the template + configSecret reference
ansible-playbook mgmt.yml reports changed on a built cluster	a task isn't idempotent	-v for command output; on the VM k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml

kubectl -n capmox-system logs deploy/capmox-controller-manager -f
kubectl -n capi-system   logs deploy/capi-controller-manager -f
ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100'