cluster-autoscaler-with-capi/docs/image-builder-proxmox.md

Node VM template with image-builder (Proxmox)

Result: Proxmox template ubuntu-2404-kube-v1.35.9 on node pv1-1 — Ubuntu 24.04.4, containerd, kubeadm/kubelet/kubectl v1.35.9, pre-pulled control-plane images, cloud-init, qemu-guest-agent. Its VMID (101) is TEMPLATE_VMID for CAPMOX. A byte-identical copy, 9101, sits on pv1-2 (Template on other PVE nodes).

Build host prerequisites (Ubuntu 24.04)

sudo apt install -y make git curl unzip ca-certificates openssl openssh-client \
                    python3 python3-venv python3-pip
python3 -m venv ~/ib-venv && source ~/ib-venv/bin/activate
pip install ansible-core==2.18.18          # avoids PEP 668 / --break-system-packages
git clone https://github.com/kubernetes-sigs/image-builder.git ~/image-builder
cd ~/image-builder/images/capi && git checkout v0.1.55
make deps-proxmox                          # python/ansible/packer 1.9.5 checks + packer init

Build

Overrides live in a var file (configs/image-builder/proxmox-lab.json.example), passed via PACKER_VAR_FILES (last on the packer command line → highest precedence).

export PROXMOX_URL="https://192.168.0.8:8006/api2/json"
export PROXMOX_USERNAME='capmox@pve!capi'
export PROXMOX_TOKEN="<secret>"
export PROXMOX_NODE="pv1-1"               # PVE node *name* (pvesh get /nodes), not IP; template lands here
export PROXMOX_ISO_POOL="local"           # storage with content 'iso'    (pvesm status --content iso)
export PROXMOX_STORAGE_POOL="local-lvm"   # storage with content 'images' (pvesm status --content images)
export PROXMOX_BRIDGE="vmbr0"
export PROXMOX_VLAN="4"
export PACKER_VAR_FILES=~/proxmox-lab.json
unset PACKER_FLAGS ISO_FILE

export SSH_PASSWORD=labpass               # optional: ssh builder@<vm> while provisioning runs
ON_ERROR_ASK=1 PACKER_LOG=1 PACKER_LOG_PATH=packer.log make build-proxmox-ubuntu-2404

The build VM (VLAN 4) needs DHCP and must reach the Packer HTTP server on the build host (ports 8000–9000) to fetch the autoinstall user-data.

What the build does

1. set-ssh-password renders the autoinstall user-data with a hashed password for user builder.
2. Packer creates VM packer-<uuid> (2 vCPU, 2 GB, 20 GB), attaches the ISO, serves http/24.04/.
3. After boot_wait it types the GRUB boot command over VNC: autoinstall ds=nocloud-net;s=http://<host>:<port>/24.04/.
4. Subiquity installs Ubuntu and reboots; Packer waits for SSH (up to 2h), IP via guest agent.
5. Provisioners: wait for cloud-init → ansible firstboot.yml → reboot → ansible node.yml (containerd, k8s binaries, images, sysctl, cloud-init config) → goss tests → lock builder, shutdown.
6. VM converted to template ubuntu-2404-kube-v1.35.9.

Observe it in the PVE noVNC console; DEBUG=1 steps through each Packer stage; make -n prints the exact packer build command. Typical duration 20–40 min.

Fixes log

#	Symptom	Root cause	Fix
1	POST .../storage/local/upload: write: broken pipe	pveproxy dropped the 3.17 GiB upload mid-stream — missing Datastore.AllocateTemplate (token had only PVEVMAdmin) or no spool space in /var/tmp (exact cause not pinned down)	PVEDatastoreAdmin on /storage, check df -h /var/tmp; then avoid re-uploads with iso_file
2	net0.bridge: invalid bridge ID 'vmbr0.4@vmbr0'	PROXMOX_BRIDGE set to the ip link name of a VLAN sub-interface; PVE wants a bridge name + separate tag	PROXMOX_BRIDGE=vmbr0, PROXMOX_VLAN=4 (→ bridge=vmbr0,tag=4)
3	403 Permission check failed (/sdn/zones/localnetwork/vmbr0.4, SDN.Use)	since PVE 8 every Linux bridge is an object in SDN zone localnetwork; PVEVMAdmin lacks SDN.Use	PVESDNUser on /sdn/zones/localnetwork
4	unsupported format 'qcow2' at .../LvmThinPlugin.pm line 102	image-builder defaults disk_format to qcow2; LVM-thin is raw-only. --var disk_format=raw in PACKER_FLAGS was lost when PACKER_FLAGS was re-exported	"disk_format": "raw" in the PACKER_VAR_FILES JSON
5	one of iso_file, iso_url, or ... cd_content must be specified for boot_iso	packer-plugin-proxmox v1.2.4 requires exactly one ISO source; ubuntu-2404.json sets iso_file from $ISO_FILE and a hardcoded iso_url → count 2. Misleading error; README tip is broken (worth an upstream issue)	set iso_file and "iso_url": "" in the var file, unset ISO_FILE

Notes carried into CAPMOX (phase 1): same ACL set, vlan: 4 on the network device, no format: qcow2 in ProxmoxMachineTemplates.

Smoke test of the template

The template has no usable login (builder is locked) — access only via cloud-init (NoCloud, which the PVE Cloud-Init drive provides):

Run it on the node that holds the template. VMIDs are cluster-wide, so pick a free ID: 9001 is now debian-13-template, and the last line destroys $ID. The key file must exist on that node (scp ~/.ssh/id_ed25519.pub root@<node-ip>:/root/jan.pub).

TPL=<template vmid>; ID=9901
qm status $ID 2>/dev/null && echo "VM $ID exists, pick another ID"   # must print nothing
qm clone $TPL $ID --name smoke-k8s --full 1 --storage local-lvm
qm set $ID --ide2 local-lvm:cloudinit
qm set $ID --ciuser jan --sshkeys /root/jan.pub --ipconfig0 ip=dhcp
qm set $ID --net0 virtio,bridge=vmbr0,tag=4
qm start $ID && qm guest cmd $ID network-get-interfaces
# inside: cloud-init status --long; kubeadm version -o short; crictl images
qm stop $ID && qm destroy $ID --purge

kubelet crash-looping before kubeadm init/join is expected.

Template on other PVE nodes

Without shared storage, a template on local-lvm can only be cloned on its own node. That holds for PVE and for CAPMOX: can't clone VM to node '<target>' (VM uses local storage). Each additional PVE node therefore gets its own copy under its own VMID, since VMIDs are cluster-wide.

Copy rather than rebuild: the copy is byte-identical, whereas a second Packer run would pick up newer apt packages. The source must have no disk or CD-ROM on non-shared storage other than local-lvm; check with qm config 101.

# on the node holding the template (pv1-1); 9101 = copy for pv1-2
qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm  # clone stays on pv1-1
qm migrate 9101 pv1-2 --with-local-disks                                        # offline: copies the LVs
ssh root@192.168.0.10 qm template 9101

Then run the smoke test on the target node with TPL=9101. It also proves that VLAN 4 works on that node's vmbr0. The copy is referenced by templateID: 9101 + sourceNode: pv1-2 in that node's ProxmoxMachineTemplate (phase 1, Step 0).

A new template build (a new Kubernetes version) means building on pv1-1 and then repeating the copy for every other node, with a new VMID pair per version.