kacerrandClaude Opus 5.5
5fd998c5f9
Phase 1-3 runbooks and design notes, version pins, config templates, project-history conventions, cluster-inspector read-only credentials setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
86 lines
5.1 KiB
Markdown
86 lines
5.1 KiB
Markdown
# Node VM template with image-builder (Proxmox)
|
||
|
||
Result: Proxmox template `ubuntu-2404-kube-v1.35.9` on node `pv1-1` — Ubuntu 24.04.4, containerd,
|
||
kubeadm/kubelet/kubectl v1.35.9, pre-pulled control-plane images, cloud-init, qemu-guest-agent.
|
||
Its VMID is `TEMPLATE_VMID` for CAPMOX.
|
||
|
||
## Build host prerequisites (Ubuntu 24.04)
|
||
|
||
```bash
|
||
sudo apt install -y make git curl unzip ca-certificates openssl openssh-client \
|
||
python3 python3-venv python3-pip
|
||
python3 -m venv ~/ib-venv && source ~/ib-venv/bin/activate
|
||
pip install ansible-core==2.18.18 # avoids PEP 668 / --break-system-packages
|
||
git clone https://github.com/kubernetes-sigs/image-builder.git ~/image-builder
|
||
cd ~/image-builder/images/capi && git checkout v0.1.55
|
||
make deps-proxmox # python/ansible/packer 1.9.5 checks + packer init
|
||
```
|
||
|
||
## Build
|
||
|
||
Overrides live in a var file ([configs/image-builder/proxmox-lab.json.example](../configs/image-builder/proxmox-lab.json.example)),
|
||
passed via `PACKER_VAR_FILES` (last on the packer command line → highest precedence).
|
||
|
||
```bash
|
||
export PROXMOX_URL="https://192.168.0.8:8006/api2/json"
|
||
export PROXMOX_USERNAME='capmox@pve!capi'
|
||
export PROXMOX_TOKEN="<secret>"
|
||
export PROXMOX_NODE="pv1-1" # PVE node *name* (pvesh get /nodes), not IP; template lands here
|
||
export PROXMOX_ISO_POOL="local" # storage with content 'iso' (pvesm status --content iso)
|
||
export PROXMOX_STORAGE_POOL="local-lvm" # storage with content 'images' (pvesm status --content images)
|
||
export PROXMOX_BRIDGE="vmbr0"
|
||
export PROXMOX_VLAN="4"
|
||
export PACKER_VAR_FILES=~/proxmox-lab.json
|
||
unset PACKER_FLAGS ISO_FILE
|
||
|
||
export SSH_PASSWORD=labpass # optional: ssh builder@<vm> while provisioning runs
|
||
ON_ERROR_ASK=1 PACKER_LOG=1 PACKER_LOG_PATH=packer.log make build-proxmox-ubuntu-2404
|
||
```
|
||
|
||
The build VM (VLAN 4) needs DHCP and must reach the Packer HTTP server on the build host
|
||
(ports 8000–9000) to fetch the autoinstall `user-data`.
|
||
|
||
## What the build does
|
||
|
||
1. `set-ssh-password` renders the autoinstall `user-data` with a hashed password for user `builder`.
|
||
2. Packer creates VM `packer-<uuid>` (2 vCPU, 2 GB, 20 GB), attaches the ISO, serves `http/24.04/`.
|
||
3. After `boot_wait` it types the GRUB boot command over VNC: `autoinstall ds=nocloud-net;s=http://<host>:<port>/24.04/`.
|
||
4. Subiquity installs Ubuntu and reboots; Packer waits for SSH (up to 2h), IP via guest agent.
|
||
5. Provisioners: wait for cloud-init → ansible `firstboot.yml` → reboot → ansible `node.yml`
|
||
(containerd, k8s binaries, images, sysctl, cloud-init config) → goss tests → lock `builder`, shutdown.
|
||
6. VM converted to template `ubuntu-2404-kube-v1.35.9`.
|
||
|
||
Observe it in the PVE noVNC console; `DEBUG=1` steps through each Packer stage; `make -n` prints the
|
||
exact `packer build` command. Typical duration 20–40 min.
|
||
|
||
## Fixes log
|
||
|
||
| # | Symptom | Root cause | Fix |
|
||
| --- | --- | --- | --- |
|
||
| 1 | `POST .../storage/local/upload: write: broken pipe` | pveproxy dropped the 3.17 GiB upload mid-stream — missing `Datastore.AllocateTemplate` (token had only `PVEVMAdmin`) or no spool space in `/var/tmp` (exact cause not pinned down) | `PVEDatastoreAdmin` on `/storage`, check `df -h /var/tmp`; then avoid re-uploads with `iso_file` |
|
||
| 2 | `net0.bridge: invalid bridge ID 'vmbr0.4@vmbr0'` | `PROXMOX_BRIDGE` set to the `ip link` name of a VLAN sub-interface; PVE wants a bridge name + separate tag | `PROXMOX_BRIDGE=vmbr0`, `PROXMOX_VLAN=4` (→ `bridge=vmbr0,tag=4`) |
|
||
| 3 | `403 Permission check failed (/sdn/zones/localnetwork/vmbr0.4, SDN.Use)` | since PVE 8 every Linux bridge is an object in SDN zone `localnetwork`; `PVEVMAdmin` lacks `SDN.Use` | `PVESDNUser` on `/sdn/zones/localnetwork` |
|
||
| 4 | `unsupported format 'qcow2' at .../LvmThinPlugin.pm line 102` | image-builder defaults `disk_format` to qcow2; LVM-thin is raw-only. `--var disk_format=raw` in `PACKER_FLAGS` was lost when `PACKER_FLAGS` was re-exported | `"disk_format": "raw"` in the `PACKER_VAR_FILES` JSON |
|
||
| 5 | `one of iso_file, iso_url, or ... cd_content must be specified for boot_iso` | packer-plugin-proxmox v1.2.4 requires **exactly one** ISO source; `ubuntu-2404.json` sets `iso_file` from `$ISO_FILE` *and* a hardcoded `iso_url` → count 2. Misleading error; README tip is broken (worth an upstream issue) | set `iso_file` and `"iso_url": ""` in the var file, unset `ISO_FILE` |
|
||
|
||
Notes carried into CAPMOX (phase 1): same ACL set, `vlan: 4` on the network device, no
|
||
`format: qcow2` in `ProxmoxMachineTemplate`s.
|
||
|
||
## Smoke test of the template
|
||
|
||
The template has no usable login (`builder` is locked) — access only via cloud-init (NoCloud,
|
||
which the PVE Cloud-Init drive provides):
|
||
|
||
```bash
|
||
TPL=<template vmid>; ID=9001
|
||
qm clone $TPL $ID --name smoke-k8s --full 1 --storage local-lvm
|
||
qm set $ID --ide2 local-lvm:cloudinit
|
||
qm set $ID --ciuser jan --sshkeys /root/jan.pub --ipconfig0 ip=dhcp
|
||
qm set $ID --net0 virtio,bridge=vmbr0,tag=4
|
||
qm start $ID && qm guest cmd $ID network-get-interfaces
|
||
# inside: cloud-init status --long; kubeadm version -o short; crictl images
|
||
qm stop $ID && qm destroy $ID --purge
|
||
```
|
||
|
||
kubelet crash-looping before `kubeadm init/join` is expected.
|