Plan: lab1 workers on pv1-2 via per-node template copy

Session note on CAPMOX v0.9.1 against the 2-node PVE cluster, the plan and its execution log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plans-executions/2026-09-29-2315-pv1-2-workers.md

@@ -0,0 +1,23 @@
+# Execution log: lab1 workers on both PVE nodes (template copy per node)
+
+Pairs with [docs/plans/2026-09-29-2315-pv1-2-workers.md](../plans/2026-09-29-2315-pv1-2-workers.md).
+
+## Status
+
+- [x] Step 0 — Record the plan
+- [ ] Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner)
+- [ ] Step 2 — Template 9101 on `pv1-2` (owner)
+- [ ] Step 3 — Second worker MachineDeployment `lab1-workers-pv1-2`
+- [ ] Step 4 — Scale test across both nodes
+- [ ] Step 5 — Docs and history
+
+## Step 0 — Record the plan (2026-09-29 23:15 CEST)
+
+Plan and this log committed on their own, together with the
+[session note](../sessions/2026-09-29-2311-pve-two-node-cluster.md) that motivated them.
+
+Starting state (read-only PVE API):
+- cluster `homelab`: `pv1-1` 192.168.0.8, `pv1-2` 192.168.0.10
+- `vmbr0` on `pv1-2` has no `bridge_vlan_aware`
+- template 101 is only on `pv1-1`, with disks `scsi0` + `efidisk0`, both on `local-lvm`
+- `nextid` is 103

docs/plans/2026-09-29-2315-pv1-2-workers.md

@@ -0,0 +1,119 @@
+# Plan: lab1 workers on both PVE nodes (template copy per node)
+
+**Created:** 2026-09-29 23:15 CEST
+
+## Context
+
+`pv1-1` is now the PVE cluster `homelab`, and `pv1-2` (192.168.0.10, 12 cores, 40.9 GiB) has joined
+it. There is no shared storage, and the owner doesn't want any for now. CAPMOX v0.9.1 can't clone
+across nodes from `local-lvm`, and `templateSelector` needs exactly one match in the whole cluster
+([session note](../sessions/2026-09-29-2311-pve-two-node-cluster.md)). So each node needs its own
+template VMID, and workers get one MachineDeployment per PVE node.
+
+Two blockers found through the PVE API:
+- `vmbr0` on `pv1-2` is not VLAN-aware.
+- Template 101 exists only on `pv1-1`. Its config is two `local-lvm` disks (`scsi0`, `efidisk0`),
+  no ISO or `local` references, so an offline migration is possible.
+
+Owner's choices:
+- Copy template 101 rather than rebuilding it.
+- Keep `lab1-workers` as it is and add `lab1-workers-pv1-2`.
+
+## Steps
+
+### Step 0 — Record the plan
+Copy this plan to `docs/plans/<date +%Y-%m-%d-%H%M>-pv1-2-workers.md`, create the execution log
+`docs/plans-executions/<same name>` with the checklist, and commit both on their own to `main`.
+
+### Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner runs)
+In `/etc/network/interfaces` on `pv1-2`, add to the `iface vmbr0` stanza:
+```
+bridge-vlan-aware yes
+bridge-vids 2-4094
+```
+Then run `ifreload -a`. The switch port for `pv1-2` must carry VLAN 4 tagged, the same as `pv1-1`.
+
+Verify (read-only, by me): `GET /nodes/pv1-2/network/vmbr0` → `bridge_vlan_aware: 1`.
+
+### Step 2 — Template 9101 on `pv1-2` (owner runs on `pv1-1`)
+```bash
+qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm  # the clone stays on pv1-1 (local storage)
+qm migrate 9101 pv1-2 --with-local-disks                                        # offline: copies both LVs to pv1-2's local-lvm
+ssh root@192.168.0.10 qm template 9101                                          # on pv1-2
+```
+- VMID 9101 is outside CAPMOX's `nextid` range (103+), next to the existing 9000/9001 templates.
+- The name stays the same on purpose. CAPMOX addresses the template by `templateID`, not by name.
+
+Verify:
+- `/cluster/resources`: 9101 is on `pv1-2` with `template: 1`.
+- Smoke test from [image-builder-proxmox.md](../image-builder-proxmox.md#smoke-test-of-the-template),
+  run on `pv1-2` with `TPL=9101`. It also proves that VLAN 4 works on `pv1-2` (DHCP lease, guest
+  agent answers).
+
+### Step 3 — Second worker MachineDeployment (`configs/capi/lab1.yaml`)
+- New `ProxmoxMachineTemplate/lab1-worker-2g-pv1-2`: a copy of `lab1-worker-2g` with:
+  - `sourceNode: pv1-2`
+  - `templateID: 9101`
+  - `allowedNodes: [pv1-2]` (the machine-level list overrides the cluster's,
+    `vmscheduler.go:52-62`)
+- New `MachineDeployment/lab1-workers-pv1-2`: a copy of `lab1-workers` with:
+  - the same KubeadmConfigTemplate `lab1-worker` and the same capacity annotations
+  - the same rollout `maxSurge 0 / maxUnavailable 1`
+  - min 0 / max **4**
+- `lab1-workers`: max-size annotation 5 → **3**. Only an annotation changes, so no rollout.
+  `lab1-worker-2g` is not edited; templates are never edited in place.
+- `ProxmoxCluster.allowedNodes` stays `[pv1-1]`. The control plane (template 101) must not be
+  scheduled onto `pv1-2`.
+- IP pool check: .31–.39 has 9 addresses; 1 CP + 3 + 4 workers = 8, so one address is spare.
+- Update the header comment: generate command and hand edit #10 (second MD, per-node template),
+  plus #7 for the new max-size.
+
+Owner runs:
+```bash
+kubectl diff -f configs/capi/lab1.yaml   # expect: new PMT + MD, max-size 5→3 on lab1-workers
+kubectl apply -f configs/capi/lab1.yaml
+```
+- No change to cluster-autoscaler values: `autoDiscovery` picks up the new MD, and
+  `balance-similar-node-groups: true` plus the `least-waste` expander spread scale-ups across the
+  two groups.
+- The RBAC rule for `proxmoxmachinetemplates` already covers the new template.
+
+### Step 4 — Scale test
+1. Scale `inflate` so it needs about 4 workers (`configs/test/inflate.yaml`, 2 pods per worker).
+2. Expect:
+   - the CA log shows `discovered node group: MachineDeployment/default/lab1-workers-pv1-2 (min: 0, max: 4…)`
+   - the scale-up is balanced across both MDs
+   - the ProxmoxMachines land on `pv1-1` / `pv1-2` according to `status.proxmoxNode`
+   - the nodes go Ready (Cilium)
+3. Then scale down to 0 and confirm both groups shrink.
+Diagnosis goes through the `cluster-inspector` agent.
+
+### Step 5 — Docs and history
+- `docs/decisions.md`:
+  - new row #18: "Template copy per PVE node (101 on `pv1-1`, 9101 on `pv1-2`); one worker MD per
+    node". Alternatives: shared storage, a rebuild per node, `templateSelector`. Status
+    `proposed` until Step 4 passes, then `active`.
+  - #7 → `superseded by #18`
+- `docs/phase-1-proxmox-capi.md`:
+  - topology table (two MDs, nodes)
+  - Step 0: the storage/network paragraph (per-node template, VLAN-aware bridge on every node)
+  - Step 5 annotations
+  - troubleshooting rows:
+    - clone to another node → "VM uses local storage"
+    - VLAN not reachable on the new node
+- `docs/image-builder-proxmox.md`: a new section "Template on other PVE nodes" with the Step 2
+  commands. The result line then names both VMIDs.
+- `configs/capi/clusterctl.yaml.example`: fix the comments on `ALLOWED_NODES` / `SOURCENODE`.
+- CLAUDE.md, AGENTS.md "Lab facts" and README line 29: PVE cluster `homelab`, `pv1-2`
+  192.168.0.10, templates 101/9101. These files have uncommitted owner edits, so I stage only my
+  hunks (`git add -p`) or ask first.
+- Session note: add the outcome and a link to the plan.
+- `CHANGELOG.md` entry, only after the owner confirms Step 4 works.
+
+## Verification
+- PVE API (read-only): `vmbr0` VLAN-aware on both nodes; template 9101 on `pv1-2`.
+- Smoke-test VM on `pv1-2` gets a VLAN 4 address.
+- `clusterctl describe cluster lab1`: both MDs are healthy.
+- The Step 4 scale test puts workers on both PVE nodes, and they join and become Ready.
+- Rollback: the owner deletes `MachineDeployment/lab1-workers-pv1-2` and restores max 5 on
+  `lab1-workers`. Template 9101 can stay.

docs/sessions/2026-09-29-2311-pve-two-node-cluster.md

@@ -0,0 +1,89 @@
+# Session: CAPMOX against a two-node PVE cluster
+
+**Date:** 2026-09-29 23:11 CEST
+
+## Question
+
+`pv1-1` became the PVE cluster `homelab`, and a second node `pv1-2` joined it. Does the CAPI/CAPMOX
+configuration need changes to work against the cluster?
+
+## What was checked
+
+- PVE API (read-only GETs with the `capmox@pve!capi` token, 2026-09-29): `/cluster/status`,
+  `/nodes`, `/storage`, `/nodes/<n>/network`, `/nodes/<n>/storage`, `/cluster/resources`,
+  `/access/acl`.
+- CAPMOX v0.9.1 source (Go proxy zip): scheduler, clone path, template lookup,
+  `templates/cluster-template.yaml`.
+- PVE `qemu-server` `PVE/API2/Qemu.pm` (HEAD, not pinned to a PVE release): the clone target check.
+
+## Findings
+
+PVE state:
+
+| Item | pv1-1 | pv1-2 |
+| --- | --- | --- |
+| IP | 192.168.0.8 | 192.168.0.10 |
+| CPU / RAM | 16 / 29.2 GiB | 12 / 40.9 GiB |
+| `vmbr0` VLAN-aware | yes | **no** |
+| `local-lvm` | local LVM-thin | local LVM-thin (not shared) |
+| Templates | 101 `ubuntu-2404-kube-v1.35.9`, 9000, 9001 | none |
+
+- Storage definitions (`local`, `local-lvm`, `proxy-seeds`) are cluster-wide without a `nodes`
+  restriction, but none of them is shared.
+- The `capmox@pve` ACLs (`/`, `/storage`, `/sdn/zones/localnetwork`) are cluster-wide, so they
+  cover `pv1-2` as well. There are no SDN zones.
+
+CAPMOX v0.9.1:
+
+- **Node choice.** If the ProxmoxMachine has its own `allowedNodes`, that list replaces the
+  ProxmoxCluster's (`internal/service/scheduler/vmscheduler.go:52-62`).
+  - The scheduler balances by how many machines of this cluster are already on each node
+    (`status.nodeLocations`), and only picks a node where the requested memory fits
+    (`vmscheduler.go:75-119`).
+  - `memoryAdjustment` is applied per node, against that node's own total memory.
+  - With empty `allowedNodes` the scheduler isn't called and the VM clones on the source node
+    (`internal/service/vmservice/vm.go:477`).
+- **Clone.** `target` goes to the PVE clone call as-is (`pkg/proxmox/goproxmox/api_client.go:84-95`).
+  - PVE refuses a cross-node clone when any source disk is on non-shared storage:
+    `can't clone VM to node '<target>' (VM uses local storage)`.
+  - Setting `full: true` or `storage` does not change that.
+  - v0.9.1 has no `localStorage` option.
+- **`templateSelector`** searches the whole PVE cluster and needs exactly one match
+  (`api_client.go:149-212`). "One template per node with the same tags" fails with
+  `found 2 VM templates`.
+- **API URL and credentials.** Everything goes through one API URL; nothing in the credentials
+  names a node.
+  - Cross-node requests should be forwarded by PVE (standard PVE behaviour, not checked in code).
+  - `192.168.0.8` is `pv1-1`, so a down `pv1-1` also means no API for CAPMOX.
+- **Bridge.** `bridge`/`vlan` are written into `netX` without any check (`vm.go:338`).
+
+Conclusion:
+
+- Nothing has to change for the current setup to keep working. `lab1` pins `allowedNodes: [pv1-1]`
+  and `sourceNode: pv1-1`, which is decision #7.
+- Using `pv1-2` requires two things:
+  1. Make `vmbr0` on `pv1-2` VLAN-aware and trunk VLAN 4 on its switch port.
+  2. Either put a template copy on `pv1-2`, or add shared storage.
+- With a template copy per node:
+  - Workers get one ProxmoxMachineTemplate and one MachineDeployment per PVE node, each with
+    `sourceNode`, `templateID` (the node's own VMID) and `allowedNodes: [<node>]`. For
+    cluster-autoscaler that is one node group per PVE node.
+  - KubeadmControlPlane takes a single infrastructureRef, so the control plane stays on one node.
+
+## Open questions
+
+- How to get the template onto `pv1-2`: a second image-builder run (`PROXMOX_NODE=pv1-2`), or
+  `qm clone` + `qm migrate` / vzdump-restore? The image-builder run is reproducible; the copy is
+  faster.
+- Is shared storage (NFS) worth adding instead? It gives one MachineDeployment spread over both
+  nodes, and the control plane can use both nodes too.
+- Behaviour when a cross-node clone fails: the scheduler may keep picking the same node. This is
+  inferred from the code, not tested.
+
+## Follow-ups
+
+- The owner chose: no shared storage for now; copy template 101 to `pv1-2` as 9101; keep
+  `lab1-workers` and add `lab1-workers-pv1-2`. Plan:
+  [2026-09-29-2315-pv1-2-workers](../plans/2026-09-29-2315-pv1-2-workers.md). It covers the bridge
+  fix, the template copy, the second MD, the CA limits and a decision row superseding #7.
+- Update the "Lab facts" in CLAUDE.md/AGENTS.md: PVE cluster `homelab`, `pv1-2` at 192.168.0.10.