Plan: lab1 workers on pv1-2 via per-node template copy
Session note on CAPMOX v0.9.1 against the 2-node PVE cluster, the plan and its execution log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -0,0 +1,119 @@
|
||||
# Plan: lab1 workers on both PVE nodes (template copy per node)
|
||||
|
||||
**Created:** 2026-09-29 23:15 CEST
|
||||
|
||||
## Context
|
||||
|
||||
`pv1-1` is now the PVE cluster `homelab`, and `pv1-2` (192.168.0.10, 12 cores, 40.9 GiB) has joined
|
||||
it. There is no shared storage, and the owner doesn't want any for now. CAPMOX v0.9.1 can't clone
|
||||
across nodes from `local-lvm`, and `templateSelector` needs exactly one match in the whole cluster
|
||||
([session note](../sessions/2026-09-29-2311-pve-two-node-cluster.md)). So each node needs its own
|
||||
template VMID, and workers get one MachineDeployment per PVE node.
|
||||
|
||||
Two blockers found through the PVE API:
|
||||
- `vmbr0` on `pv1-2` is not VLAN-aware.
|
||||
- Template 101 exists only on `pv1-1`. Its config is two `local-lvm` disks (`scsi0`, `efidisk0`),
|
||||
no ISO or `local` references, so an offline migration is possible.
|
||||
|
||||
Owner's choices:
|
||||
- Copy template 101 rather than rebuilding it.
|
||||
- Keep `lab1-workers` as it is and add `lab1-workers-pv1-2`.
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 0 — Record the plan
|
||||
Copy this plan to `docs/plans/<date +%Y-%m-%d-%H%M>-pv1-2-workers.md`, create the execution log
|
||||
`docs/plans-executions/<same name>` with the checklist, and commit both on their own to `main`.
|
||||
|
||||
### Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner runs)
|
||||
In `/etc/network/interfaces` on `pv1-2`, add to the `iface vmbr0` stanza:
|
||||
```
|
||||
bridge-vlan-aware yes
|
||||
bridge-vids 2-4094
|
||||
```
|
||||
Then run `ifreload -a`. The switch port for `pv1-2` must carry VLAN 4 tagged, the same as `pv1-1`.
|
||||
|
||||
Verify (read-only, by me): `GET /nodes/pv1-2/network/vmbr0` → `bridge_vlan_aware: 1`.
|
||||
|
||||
### Step 2 — Template 9101 on `pv1-2` (owner runs on `pv1-1`)
|
||||
```bash
|
||||
qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm # the clone stays on pv1-1 (local storage)
|
||||
qm migrate 9101 pv1-2 --with-local-disks # offline: copies both LVs to pv1-2's local-lvm
|
||||
ssh root@192.168.0.10 qm template 9101 # on pv1-2
|
||||
```
|
||||
- VMID 9101 is outside CAPMOX's `nextid` range (103+), next to the existing 9000/9001 templates.
|
||||
- The name stays the same on purpose. CAPMOX addresses the template by `templateID`, not by name.
|
||||
|
||||
Verify:
|
||||
- `/cluster/resources`: 9101 is on `pv1-2` with `template: 1`.
|
||||
- Smoke test from [image-builder-proxmox.md](../image-builder-proxmox.md#smoke-test-of-the-template),
|
||||
run on `pv1-2` with `TPL=9101`. It also proves that VLAN 4 works on `pv1-2` (DHCP lease, guest
|
||||
agent answers).
|
||||
|
||||
### Step 3 — Second worker MachineDeployment (`configs/capi/lab1.yaml`)
|
||||
- New `ProxmoxMachineTemplate/lab1-worker-2g-pv1-2`: a copy of `lab1-worker-2g` with:
|
||||
- `sourceNode: pv1-2`
|
||||
- `templateID: 9101`
|
||||
- `allowedNodes: [pv1-2]` (the machine-level list overrides the cluster's,
|
||||
`vmscheduler.go:52-62`)
|
||||
- New `MachineDeployment/lab1-workers-pv1-2`: a copy of `lab1-workers` with:
|
||||
- the same KubeadmConfigTemplate `lab1-worker` and the same capacity annotations
|
||||
- the same rollout `maxSurge 0 / maxUnavailable 1`
|
||||
- min 0 / max **4**
|
||||
- `lab1-workers`: max-size annotation 5 → **3**. Only an annotation changes, so no rollout.
|
||||
`lab1-worker-2g` is not edited; templates are never edited in place.
|
||||
- `ProxmoxCluster.allowedNodes` stays `[pv1-1]`. The control plane (template 101) must not be
|
||||
scheduled onto `pv1-2`.
|
||||
- IP pool check: .31–.39 has 9 addresses; 1 CP + 3 + 4 workers = 8, so one address is spare.
|
||||
- Update the header comment: generate command and hand edit #10 (second MD, per-node template),
|
||||
plus #7 for the new max-size.
|
||||
|
||||
Owner runs:
|
||||
```bash
|
||||
kubectl diff -f configs/capi/lab1.yaml # expect: new PMT + MD, max-size 5→3 on lab1-workers
|
||||
kubectl apply -f configs/capi/lab1.yaml
|
||||
```
|
||||
- No change to cluster-autoscaler values: `autoDiscovery` picks up the new MD, and
|
||||
`balance-similar-node-groups: true` plus the `least-waste` expander spread scale-ups across the
|
||||
two groups.
|
||||
- The RBAC rule for `proxmoxmachinetemplates` already covers the new template.
|
||||
|
||||
### Step 4 — Scale test
|
||||
1. Scale `inflate` so it needs about 4 workers (`configs/test/inflate.yaml`, 2 pods per worker).
|
||||
2. Expect:
|
||||
- the CA log shows `discovered node group: MachineDeployment/default/lab1-workers-pv1-2 (min: 0, max: 4…)`
|
||||
- the scale-up is balanced across both MDs
|
||||
- the ProxmoxMachines land on `pv1-1` / `pv1-2` according to `status.proxmoxNode`
|
||||
- the nodes go Ready (Cilium)
|
||||
3. Then scale down to 0 and confirm both groups shrink.
|
||||
Diagnosis goes through the `cluster-inspector` agent.
|
||||
|
||||
### Step 5 — Docs and history
|
||||
- `docs/decisions.md`:
|
||||
- new row #18: "Template copy per PVE node (101 on `pv1-1`, 9101 on `pv1-2`); one worker MD per
|
||||
node". Alternatives: shared storage, a rebuild per node, `templateSelector`. Status
|
||||
`proposed` until Step 4 passes, then `active`.
|
||||
- #7 → `superseded by #18`
|
||||
- `docs/phase-1-proxmox-capi.md`:
|
||||
- topology table (two MDs, nodes)
|
||||
- Step 0: the storage/network paragraph (per-node template, VLAN-aware bridge on every node)
|
||||
- Step 5 annotations
|
||||
- troubleshooting rows:
|
||||
- clone to another node → "VM uses local storage"
|
||||
- VLAN not reachable on the new node
|
||||
- `docs/image-builder-proxmox.md`: a new section "Template on other PVE nodes" with the Step 2
|
||||
commands. The result line then names both VMIDs.
|
||||
- `configs/capi/clusterctl.yaml.example`: fix the comments on `ALLOWED_NODES` / `SOURCENODE`.
|
||||
- CLAUDE.md, AGENTS.md "Lab facts" and README line 29: PVE cluster `homelab`, `pv1-2`
|
||||
192.168.0.10, templates 101/9101. These files have uncommitted owner edits, so I stage only my
|
||||
hunks (`git add -p`) or ask first.
|
||||
- Session note: add the outcome and a link to the plan.
|
||||
- `CHANGELOG.md` entry, only after the owner confirms Step 4 works.
|
||||
|
||||
## Verification
|
||||
- PVE API (read-only): `vmbr0` VLAN-aware on both nodes; template 9101 on `pv1-2`.
|
||||
- Smoke-test VM on `pv1-2` gets a VLAN 4 address.
|
||||
- `clusterctl describe cluster lab1`: both MDs are healthy.
|
||||
- The Step 4 scale test puts workers on both PVE nodes, and they join and become Ready.
|
||||
- Rollback: the owner deletes `MachineDeployment/lab1-workers-pv1-2` and restores max 5 on
|
||||
`lab1-workers`. Template 9101 can stay.
|
||||
Reference in New Issue
Block a user