kacerrandClaude Opus 5.5
6ef55778ff
PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage: - template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50; lab1-workers renamed for symmetry (decision #18, #7 superseded) - CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19). Scale test: 16 pods -> 4 + 4 across both nodes. - runbook, image-builder copy procedure, fixes tables, future task to report three CA clusterapi bugs upstream, lab facts, changelog, execution log Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
117 lines
6.8 KiB
Markdown
117 lines
6.8 KiB
Markdown
# Node VM template with image-builder (Proxmox)
|
||
|
||
Result: Proxmox template `ubuntu-2404-kube-v1.35.9` on node `pv1-1` — Ubuntu 24.04.4, containerd,
|
||
kubeadm/kubelet/kubectl v1.35.9, pre-pulled control-plane images, cloud-init, qemu-guest-agent.
|
||
Its VMID (101) is `TEMPLATE_VMID` for CAPMOX. A byte-identical copy, 9101, sits on `pv1-2`
|
||
([Template on other PVE nodes](#template-on-other-pve-nodes)).
|
||
|
||
## Build host prerequisites (Ubuntu 24.04)
|
||
|
||
```bash
|
||
sudo apt install -y make git curl unzip ca-certificates openssl openssh-client \
|
||
python3 python3-venv python3-pip
|
||
python3 -m venv ~/ib-venv && source ~/ib-venv/bin/activate
|
||
pip install ansible-core==2.18.18 # avoids PEP 668 / --break-system-packages
|
||
git clone https://github.com/kubernetes-sigs/image-builder.git ~/image-builder
|
||
cd ~/image-builder/images/capi && git checkout v0.1.55
|
||
make deps-proxmox # python/ansible/packer 1.9.5 checks + packer init
|
||
```
|
||
|
||
## Build
|
||
|
||
Overrides live in a var file ([configs/image-builder/proxmox-lab.json.example](../configs/image-builder/proxmox-lab.json.example)),
|
||
passed via `PACKER_VAR_FILES` (last on the packer command line → highest precedence).
|
||
|
||
```bash
|
||
export PROXMOX_URL="https://192.168.0.8:8006/api2/json"
|
||
export PROXMOX_USERNAME='capmox@pve!capi'
|
||
export PROXMOX_TOKEN="<secret>"
|
||
export PROXMOX_NODE="pv1-1" # PVE node *name* (pvesh get /nodes), not IP; template lands here
|
||
export PROXMOX_ISO_POOL="local" # storage with content 'iso' (pvesm status --content iso)
|
||
export PROXMOX_STORAGE_POOL="local-lvm" # storage with content 'images' (pvesm status --content images)
|
||
export PROXMOX_BRIDGE="vmbr0"
|
||
export PROXMOX_VLAN="4"
|
||
export PACKER_VAR_FILES=~/proxmox-lab.json
|
||
unset PACKER_FLAGS ISO_FILE
|
||
|
||
export SSH_PASSWORD=labpass # optional: ssh builder@<vm> while provisioning runs
|
||
ON_ERROR_ASK=1 PACKER_LOG=1 PACKER_LOG_PATH=packer.log make build-proxmox-ubuntu-2404
|
||
```
|
||
|
||
The build VM (VLAN 4) needs DHCP and must reach the Packer HTTP server on the build host
|
||
(ports 8000–9000) to fetch the autoinstall `user-data`.
|
||
|
||
## What the build does
|
||
|
||
1. `set-ssh-password` renders the autoinstall `user-data` with a hashed password for user `builder`.
|
||
2. Packer creates VM `packer-<uuid>` (2 vCPU, 2 GB, 20 GB), attaches the ISO, serves `http/24.04/`.
|
||
3. After `boot_wait` it types the GRUB boot command over VNC: `autoinstall ds=nocloud-net;s=http://<host>:<port>/24.04/`.
|
||
4. Subiquity installs Ubuntu and reboots; Packer waits for SSH (up to 2h), IP via guest agent.
|
||
5. Provisioners: wait for cloud-init → ansible `firstboot.yml` → reboot → ansible `node.yml`
|
||
(containerd, k8s binaries, images, sysctl, cloud-init config) → goss tests → lock `builder`, shutdown.
|
||
6. VM converted to template `ubuntu-2404-kube-v1.35.9`.
|
||
|
||
Observe it in the PVE noVNC console; `DEBUG=1` steps through each Packer stage; `make -n` prints the
|
||
exact `packer build` command. Typical duration 20–40 min.
|
||
|
||
## Fixes log
|
||
|
||
| # | Symptom | Root cause | Fix |
|
||
| --- | --- | --- | --- |
|
||
| 1 | `POST .../storage/local/upload: write: broken pipe` | pveproxy dropped the 3.17 GiB upload mid-stream — missing `Datastore.AllocateTemplate` (token had only `PVEVMAdmin`) or no spool space in `/var/tmp` (exact cause not pinned down) | `PVEDatastoreAdmin` on `/storage`, check `df -h /var/tmp`; then avoid re-uploads with `iso_file` |
|
||
| 2 | `net0.bridge: invalid bridge ID 'vmbr0.4@vmbr0'` | `PROXMOX_BRIDGE` set to the `ip link` name of a VLAN sub-interface; PVE wants a bridge name + separate tag | `PROXMOX_BRIDGE=vmbr0`, `PROXMOX_VLAN=4` (→ `bridge=vmbr0,tag=4`) |
|
||
| 3 | `403 Permission check failed (/sdn/zones/localnetwork/vmbr0.4, SDN.Use)` | since PVE 8 every Linux bridge is an object in SDN zone `localnetwork`; `PVEVMAdmin` lacks `SDN.Use` | `PVESDNUser` on `/sdn/zones/localnetwork` |
|
||
| 4 | `unsupported format 'qcow2' at .../LvmThinPlugin.pm line 102` | image-builder defaults `disk_format` to qcow2; LVM-thin is raw-only. `--var disk_format=raw` in `PACKER_FLAGS` was lost when `PACKER_FLAGS` was re-exported | `"disk_format": "raw"` in the `PACKER_VAR_FILES` JSON |
|
||
| 5 | `one of iso_file, iso_url, or ... cd_content must be specified for boot_iso` | packer-plugin-proxmox v1.2.4 requires **exactly one** ISO source; `ubuntu-2404.json` sets `iso_file` from `$ISO_FILE` *and* a hardcoded `iso_url` → count 2. Misleading error; README tip is broken (worth an upstream issue) | set `iso_file` and `"iso_url": ""` in the var file, unset `ISO_FILE` |
|
||
|
||
Notes carried into CAPMOX (phase 1): same ACL set, `vlan: 4` on the network device, no
|
||
`format: qcow2` in `ProxmoxMachineTemplate`s.
|
||
|
||
## Smoke test of the template
|
||
|
||
The template has no usable login (`builder` is locked) — access only via cloud-init (NoCloud,
|
||
which the PVE Cloud-Init drive provides):
|
||
|
||
Run it on the node that holds the template. VMIDs are cluster-wide, so pick a free `ID`: 9001 is
|
||
now `debian-13-template`, and the last line destroys `$ID`. The key file must exist on that node
|
||
(`scp ~/.ssh/id_ed25519.pub root@<node-ip>:/root/jan.pub`).
|
||
|
||
```bash
|
||
TPL=<template vmid>; ID=9901
|
||
qm status $ID 2>/dev/null && echo "VM $ID exists, pick another ID" # must print nothing
|
||
qm clone $TPL $ID --name smoke-k8s --full 1 --storage local-lvm
|
||
qm set $ID --ide2 local-lvm:cloudinit
|
||
qm set $ID --ciuser jan --sshkeys /root/jan.pub --ipconfig0 ip=dhcp
|
||
qm set $ID --net0 virtio,bridge=vmbr0,tag=4
|
||
qm start $ID && qm guest cmd $ID network-get-interfaces
|
||
# inside: cloud-init status --long; kubeadm version -o short; crictl images
|
||
qm stop $ID && qm destroy $ID --purge
|
||
```
|
||
|
||
kubelet crash-looping before `kubeadm init/join` is expected.
|
||
|
||
## Template on other PVE nodes
|
||
|
||
Without shared storage, a template on `local-lvm` can only be cloned on its own node. That holds
|
||
for PVE and for CAPMOX: `can't clone VM to node '<target>' (VM uses local storage)`. Each additional
|
||
PVE node therefore gets its own copy under its own VMID, since VMIDs are cluster-wide.
|
||
|
||
Copy rather than rebuild: the copy is byte-identical, whereas a second Packer run would pick up
|
||
newer apt packages. The source must have no disk or CD-ROM on non-shared storage other than
|
||
`local-lvm`; check with `qm config 101`.
|
||
|
||
```bash
|
||
# on the node holding the template (pv1-1); 9101 = copy for pv1-2
|
||
qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm # clone stays on pv1-1
|
||
qm migrate 9101 pv1-2 --with-local-disks # offline: copies the LVs
|
||
ssh root@192.168.0.10 qm template 9101
|
||
```
|
||
|
||
Then run the smoke test on the target node with `TPL=9101`. It also proves that VLAN 4 works on
|
||
that node's `vmbr0`. The copy is referenced by `templateID: 9101` + `sourceNode: pv1-2` in that
|
||
node's `ProxmoxMachineTemplate`
|
||
([phase 1, Step 0](phase-1-proxmox-capi.md#step-0--proxmox-prep)).
|
||
|
||
A new template build (a new Kubernetes version) means building on `pv1-1` and then repeating the
|
||
copy for every other node, with a new VMID pair per version.
|