From b88f721e3b790832a7426312b98ecfd6a6fe6318 Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 29 Sep 2026 23:16:49 +0200 Subject: [PATCH] Plan: lab1 workers on pv1-2 via per-node template copy Session note on CAPMOX v0.9.1 against the 2-node PVE cluster, the plan and its execution log. Co-Authored-By: Claude Opus 5.5 --- .../2026-09-29-2315-pv1-2-workers.md | 23 ++++ docs/plans/2026-09-29-2315-pv1-2-workers.md | 119 ++++++++++++++++++ .../2026-09-29-2311-pve-two-node-cluster.md | 89 +++++++++++++ 3 files changed, 231 insertions(+) create mode 100644 docs/plans-executions/2026-09-29-2315-pv1-2-workers.md create mode 100644 docs/plans/2026-09-29-2315-pv1-2-workers.md create mode 100644 docs/sessions/2026-09-29-2311-pve-two-node-cluster.md diff --git a/docs/plans-executions/2026-09-29-2315-pv1-2-workers.md b/docs/plans-executions/2026-09-29-2315-pv1-2-workers.md new file mode 100644 index 0000000..37d129b --- /dev/null +++ b/docs/plans-executions/2026-09-29-2315-pv1-2-workers.md @@ -0,0 +1,23 @@ +# Execution log: lab1 workers on both PVE nodes (template copy per node) + +Pairs with [docs/plans/2026-09-29-2315-pv1-2-workers.md](../plans/2026-09-29-2315-pv1-2-workers.md). + +## Status + +- [x] Step 0 — Record the plan +- [ ] Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner) +- [ ] Step 2 — Template 9101 on `pv1-2` (owner) +- [ ] Step 3 — Second worker MachineDeployment `lab1-workers-pv1-2` +- [ ] Step 4 — Scale test across both nodes +- [ ] Step 5 — Docs and history + +## Step 0 — Record the plan (2026-09-29 23:15 CEST) + +Plan and this log committed on their own, together with the +[session note](../sessions/2026-09-29-2311-pve-two-node-cluster.md) that motivated them. + +Starting state (read-only PVE API): +- cluster `homelab`: `pv1-1` 192.168.0.8, `pv1-2` 192.168.0.10 +- `vmbr0` on `pv1-2` has no `bridge_vlan_aware` +- template 101 is only on `pv1-1`, with disks `scsi0` + `efidisk0`, both on `local-lvm` +- `nextid` is 103 diff --git a/docs/plans/2026-09-29-2315-pv1-2-workers.md b/docs/plans/2026-09-29-2315-pv1-2-workers.md new file mode 100644 index 0000000..48e684a --- /dev/null +++ b/docs/plans/2026-09-29-2315-pv1-2-workers.md @@ -0,0 +1,119 @@ +# Plan: lab1 workers on both PVE nodes (template copy per node) + +**Created:** 2026-09-29 23:15 CEST + +## Context + +`pv1-1` is now the PVE cluster `homelab`, and `pv1-2` (192.168.0.10, 12 cores, 40.9 GiB) has joined +it. There is no shared storage, and the owner doesn't want any for now. CAPMOX v0.9.1 can't clone +across nodes from `local-lvm`, and `templateSelector` needs exactly one match in the whole cluster +([session note](../sessions/2026-09-29-2311-pve-two-node-cluster.md)). So each node needs its own +template VMID, and workers get one MachineDeployment per PVE node. + +Two blockers found through the PVE API: +- `vmbr0` on `pv1-2` is not VLAN-aware. +- Template 101 exists only on `pv1-1`. Its config is two `local-lvm` disks (`scsi0`, `efidisk0`), + no ISO or `local` references, so an offline migration is possible. + +Owner's choices: +- Copy template 101 rather than rebuilding it. +- Keep `lab1-workers` as it is and add `lab1-workers-pv1-2`. + +## Steps + +### Step 0 — Record the plan +Copy this plan to `docs/plans/-pv1-2-workers.md`, create the execution log +`docs/plans-executions/` with the checklist, and commit both on their own to `main`. + +### Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner runs) +In `/etc/network/interfaces` on `pv1-2`, add to the `iface vmbr0` stanza: +``` +bridge-vlan-aware yes +bridge-vids 2-4094 +``` +Then run `ifreload -a`. The switch port for `pv1-2` must carry VLAN 4 tagged, the same as `pv1-1`. + +Verify (read-only, by me): `GET /nodes/pv1-2/network/vmbr0` → `bridge_vlan_aware: 1`. + +### Step 2 — Template 9101 on `pv1-2` (owner runs on `pv1-1`) +```bash +qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm # the clone stays on pv1-1 (local storage) +qm migrate 9101 pv1-2 --with-local-disks # offline: copies both LVs to pv1-2's local-lvm +ssh root@192.168.0.10 qm template 9101 # on pv1-2 +``` +- VMID 9101 is outside CAPMOX's `nextid` range (103+), next to the existing 9000/9001 templates. +- The name stays the same on purpose. CAPMOX addresses the template by `templateID`, not by name. + +Verify: +- `/cluster/resources`: 9101 is on `pv1-2` with `template: 1`. +- Smoke test from [image-builder-proxmox.md](../image-builder-proxmox.md#smoke-test-of-the-template), + run on `pv1-2` with `TPL=9101`. It also proves that VLAN 4 works on `pv1-2` (DHCP lease, guest + agent answers). + +### Step 3 — Second worker MachineDeployment (`configs/capi/lab1.yaml`) +- New `ProxmoxMachineTemplate/lab1-worker-2g-pv1-2`: a copy of `lab1-worker-2g` with: + - `sourceNode: pv1-2` + - `templateID: 9101` + - `allowedNodes: [pv1-2]` (the machine-level list overrides the cluster's, + `vmscheduler.go:52-62`) +- New `MachineDeployment/lab1-workers-pv1-2`: a copy of `lab1-workers` with: + - the same KubeadmConfigTemplate `lab1-worker` and the same capacity annotations + - the same rollout `maxSurge 0 / maxUnavailable 1` + - min 0 / max **4** +- `lab1-workers`: max-size annotation 5 → **3**. Only an annotation changes, so no rollout. + `lab1-worker-2g` is not edited; templates are never edited in place. +- `ProxmoxCluster.allowedNodes` stays `[pv1-1]`. The control plane (template 101) must not be + scheduled onto `pv1-2`. +- IP pool check: .31–.39 has 9 addresses; 1 CP + 3 + 4 workers = 8, so one address is spare. +- Update the header comment: generate command and hand edit #10 (second MD, per-node template), + plus #7 for the new max-size. + +Owner runs: +```bash +kubectl diff -f configs/capi/lab1.yaml # expect: new PMT + MD, max-size 5→3 on lab1-workers +kubectl apply -f configs/capi/lab1.yaml +``` +- No change to cluster-autoscaler values: `autoDiscovery` picks up the new MD, and + `balance-similar-node-groups: true` plus the `least-waste` expander spread scale-ups across the + two groups. +- The RBAC rule for `proxmoxmachinetemplates` already covers the new template. + +### Step 4 — Scale test +1. Scale `inflate` so it needs about 4 workers (`configs/test/inflate.yaml`, 2 pods per worker). +2. Expect: + - the CA log shows `discovered node group: MachineDeployment/default/lab1-workers-pv1-2 (min: 0, max: 4…)` + - the scale-up is balanced across both MDs + - the ProxmoxMachines land on `pv1-1` / `pv1-2` according to `status.proxmoxNode` + - the nodes go Ready (Cilium) +3. Then scale down to 0 and confirm both groups shrink. +Diagnosis goes through the `cluster-inspector` agent. + +### Step 5 — Docs and history +- `docs/decisions.md`: + - new row #18: "Template copy per PVE node (101 on `pv1-1`, 9101 on `pv1-2`); one worker MD per + node". Alternatives: shared storage, a rebuild per node, `templateSelector`. Status + `proposed` until Step 4 passes, then `active`. + - #7 → `superseded by #18` +- `docs/phase-1-proxmox-capi.md`: + - topology table (two MDs, nodes) + - Step 0: the storage/network paragraph (per-node template, VLAN-aware bridge on every node) + - Step 5 annotations + - troubleshooting rows: + - clone to another node → "VM uses local storage" + - VLAN not reachable on the new node +- `docs/image-builder-proxmox.md`: a new section "Template on other PVE nodes" with the Step 2 + commands. The result line then names both VMIDs. +- `configs/capi/clusterctl.yaml.example`: fix the comments on `ALLOWED_NODES` / `SOURCENODE`. +- CLAUDE.md, AGENTS.md "Lab facts" and README line 29: PVE cluster `homelab`, `pv1-2` + 192.168.0.10, templates 101/9101. These files have uncommitted owner edits, so I stage only my + hunks (`git add -p`) or ask first. +- Session note: add the outcome and a link to the plan. +- `CHANGELOG.md` entry, only after the owner confirms Step 4 works. + +## Verification +- PVE API (read-only): `vmbr0` VLAN-aware on both nodes; template 9101 on `pv1-2`. +- Smoke-test VM on `pv1-2` gets a VLAN 4 address. +- `clusterctl describe cluster lab1`: both MDs are healthy. +- The Step 4 scale test puts workers on both PVE nodes, and they join and become Ready. +- Rollback: the owner deletes `MachineDeployment/lab1-workers-pv1-2` and restores max 5 on + `lab1-workers`. Template 9101 can stay. diff --git a/docs/sessions/2026-09-29-2311-pve-two-node-cluster.md b/docs/sessions/2026-09-29-2311-pve-two-node-cluster.md new file mode 100644 index 0000000..e83a5a9 --- /dev/null +++ b/docs/sessions/2026-09-29-2311-pve-two-node-cluster.md @@ -0,0 +1,89 @@ +# Session: CAPMOX against a two-node PVE cluster + +**Date:** 2026-09-29 23:11 CEST + +## Question + +`pv1-1` became the PVE cluster `homelab`, and a second node `pv1-2` joined it. Does the CAPI/CAPMOX +configuration need changes to work against the cluster? + +## What was checked + +- PVE API (read-only GETs with the `capmox@pve!capi` token, 2026-09-29): `/cluster/status`, + `/nodes`, `/storage`, `/nodes//network`, `/nodes//storage`, `/cluster/resources`, + `/access/acl`. +- CAPMOX v0.9.1 source (Go proxy zip): scheduler, clone path, template lookup, + `templates/cluster-template.yaml`. +- PVE `qemu-server` `PVE/API2/Qemu.pm` (HEAD, not pinned to a PVE release): the clone target check. + +## Findings + +PVE state: + +| Item | pv1-1 | pv1-2 | +| --- | --- | --- | +| IP | 192.168.0.8 | 192.168.0.10 | +| CPU / RAM | 16 / 29.2 GiB | 12 / 40.9 GiB | +| `vmbr0` VLAN-aware | yes | **no** | +| `local-lvm` | local LVM-thin | local LVM-thin (not shared) | +| Templates | 101 `ubuntu-2404-kube-v1.35.9`, 9000, 9001 | none | + +- Storage definitions (`local`, `local-lvm`, `proxy-seeds`) are cluster-wide without a `nodes` + restriction, but none of them is shared. +- The `capmox@pve` ACLs (`/`, `/storage`, `/sdn/zones/localnetwork`) are cluster-wide, so they + cover `pv1-2` as well. There are no SDN zones. + +CAPMOX v0.9.1: + +- **Node choice.** If the ProxmoxMachine has its own `allowedNodes`, that list replaces the + ProxmoxCluster's (`internal/service/scheduler/vmscheduler.go:52-62`). + - The scheduler balances by how many machines of this cluster are already on each node + (`status.nodeLocations`), and only picks a node where the requested memory fits + (`vmscheduler.go:75-119`). + - `memoryAdjustment` is applied per node, against that node's own total memory. + - With empty `allowedNodes` the scheduler isn't called and the VM clones on the source node + (`internal/service/vmservice/vm.go:477`). +- **Clone.** `target` goes to the PVE clone call as-is (`pkg/proxmox/goproxmox/api_client.go:84-95`). + - PVE refuses a cross-node clone when any source disk is on non-shared storage: + `can't clone VM to node '' (VM uses local storage)`. + - Setting `full: true` or `storage` does not change that. + - v0.9.1 has no `localStorage` option. +- **`templateSelector`** searches the whole PVE cluster and needs exactly one match + (`api_client.go:149-212`). "One template per node with the same tags" fails with + `found 2 VM templates`. +- **API URL and credentials.** Everything goes through one API URL; nothing in the credentials + names a node. + - Cross-node requests should be forwarded by PVE (standard PVE behaviour, not checked in code). + - `192.168.0.8` is `pv1-1`, so a down `pv1-1` also means no API for CAPMOX. +- **Bridge.** `bridge`/`vlan` are written into `netX` without any check (`vm.go:338`). + +Conclusion: + +- Nothing has to change for the current setup to keep working. `lab1` pins `allowedNodes: [pv1-1]` + and `sourceNode: pv1-1`, which is decision #7. +- Using `pv1-2` requires two things: + 1. Make `vmbr0` on `pv1-2` VLAN-aware and trunk VLAN 4 on its switch port. + 2. Either put a template copy on `pv1-2`, or add shared storage. +- With a template copy per node: + - Workers get one ProxmoxMachineTemplate and one MachineDeployment per PVE node, each with + `sourceNode`, `templateID` (the node's own VMID) and `allowedNodes: []`. For + cluster-autoscaler that is one node group per PVE node. + - KubeadmControlPlane takes a single infrastructureRef, so the control plane stays on one node. + +## Open questions + +- How to get the template onto `pv1-2`: a second image-builder run (`PROXMOX_NODE=pv1-2`), or + `qm clone` + `qm migrate` / vzdump-restore? The image-builder run is reproducible; the copy is + faster. +- Is shared storage (NFS) worth adding instead? It gives one MachineDeployment spread over both + nodes, and the control plane can use both nodes too. +- Behaviour when a cross-node clone fails: the scheduler may keep picking the same node. This is + inferred from the code, not tested. + +## Follow-ups + +- The owner chose: no shared storage for now; copy template 101 to `pv1-2` as 9101; keep + `lab1-workers` and add `lab1-workers-pv1-2`. Plan: + [2026-09-29-2315-pv1-2-workers](../plans/2026-09-29-2315-pv1-2-workers.md). It covers the bridge + fix, the template copy, the second MD, the CA limits and a decision row superseding #7. +- Update the "Lab facts" in CLAUDE.md/AGENTS.md: PVE cluster `homelab`, `pv1-2` at 192.168.0.10.