kacerrandClaude Opus 5.5
6ef55778ff
PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage: - template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50; lab1-workers renamed for symmetry (decision #18, #7 superseded) - CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19). Scale test: 16 pods -> 4 + 4 across both nodes. - runbook, image-builder copy procedure, fixes tables, future task to report three CA clusterapi bugs upstream, lab facts, changelog, execution log Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
101 lines
4.8 KiB
Markdown
101 lines
4.8 KiB
Markdown
# Session: CAPMOX against a two-node PVE cluster
|
|
|
|
**Date:** 2026-09-29 23:11 CEST
|
|
|
|
## Question
|
|
|
|
`pv1-1` became the PVE cluster `homelab`, and a second node `pv1-2` joined it. Does the CAPI/CAPMOX
|
|
configuration need changes to work against the cluster?
|
|
|
|
## What was checked
|
|
|
|
- PVE API (read-only GETs with the `capmox@pve!capi` token, 2026-09-29): `/cluster/status`,
|
|
`/nodes`, `/storage`, `/nodes/<n>/network`, `/nodes/<n>/storage`, `/cluster/resources`,
|
|
`/access/acl`.
|
|
- CAPMOX v0.9.1 source (Go proxy zip): scheduler, clone path, template lookup,
|
|
`templates/cluster-template.yaml`.
|
|
- PVE `qemu-server` `PVE/API2/Qemu.pm` (HEAD, not pinned to a PVE release): the clone target check.
|
|
|
|
## Findings
|
|
|
|
PVE state:
|
|
|
|
| Item | pv1-1 | pv1-2 |
|
|
| --- | --- | --- |
|
|
| IP | 192.168.0.8 | 192.168.0.10 |
|
|
| CPU / RAM | 16 / 29.2 GiB | 12 / 40.9 GiB |
|
|
| `vmbr0` VLAN-aware | yes | **no** |
|
|
| `local-lvm` | local LVM-thin | local LVM-thin (not shared) |
|
|
| Templates | 101 `ubuntu-2404-kube-v1.35.9`, 9000, 9001 | none |
|
|
|
|
- Storage definitions (`local`, `local-lvm`, `proxy-seeds`) are cluster-wide without a `nodes`
|
|
restriction, but none of them is shared.
|
|
- The `capmox@pve` ACLs (`/`, `/storage`, `/sdn/zones/localnetwork`) are cluster-wide, so they
|
|
cover `pv1-2` as well. There are no SDN zones.
|
|
|
|
CAPMOX v0.9.1:
|
|
|
|
- **Node choice.** If the ProxmoxMachine has its own `allowedNodes`, that list replaces the
|
|
ProxmoxCluster's (`internal/service/scheduler/vmscheduler.go:52-62`).
|
|
- The scheduler balances by how many machines of this cluster are already on each node
|
|
(`status.nodeLocations`), and only picks a node where the requested memory fits
|
|
(`vmscheduler.go:75-119`).
|
|
- `memoryAdjustment` is applied per node, against that node's own total memory.
|
|
- With empty `allowedNodes` the scheduler isn't called and the VM clones on the source node
|
|
(`internal/service/vmservice/vm.go:477`).
|
|
- **Clone.** `target` goes to the PVE clone call as-is (`pkg/proxmox/goproxmox/api_client.go:84-95`).
|
|
- PVE refuses a cross-node clone when any source disk is on non-shared storage:
|
|
`can't clone VM to node '<target>' (VM uses local storage)`.
|
|
- Setting `full: true` or `storage` does not change that.
|
|
- v0.9.1 has no `localStorage` option.
|
|
- **`templateSelector`** searches the whole PVE cluster and needs exactly one match
|
|
(`api_client.go:149-212`). "One template per node with the same tags" fails with
|
|
`found 2 VM templates`.
|
|
- **API URL and credentials.** Everything goes through one API URL; nothing in the credentials
|
|
names a node.
|
|
- Cross-node requests should be forwarded by PVE (standard PVE behaviour, not checked in code).
|
|
- `192.168.0.8` is `pv1-1`, so a down `pv1-1` also means no API for CAPMOX.
|
|
- **Bridge.** `bridge`/`vlan` are written into `netX` without any check (`vm.go:338`).
|
|
|
|
Conclusion:
|
|
|
|
- Nothing has to change for the current setup to keep working. `lab1` pins `allowedNodes: [pv1-1]`
|
|
and `sourceNode: pv1-1`, which is decision #7.
|
|
- Using `pv1-2` requires two things:
|
|
1. Make `vmbr0` on `pv1-2` VLAN-aware and trunk VLAN 4 on its switch port.
|
|
2. Either put a template copy on `pv1-2`, or add shared storage.
|
|
- With a template copy per node:
|
|
- Workers get one ProxmoxMachineTemplate and one MachineDeployment per PVE node, each with
|
|
`sourceNode`, `templateID` (the node's own VMID) and `allowedNodes: [<node>]`. For
|
|
cluster-autoscaler that is one node group per PVE node.
|
|
- KubeadmControlPlane takes a single infrastructureRef, so the control plane stays on one node.
|
|
|
|
## Open questions
|
|
|
|
- How to get the template onto `pv1-2`: a second image-builder run (`PROXMOX_NODE=pv1-2`), or
|
|
`qm clone` + `qm migrate` / vzdump-restore? The image-builder run is reproducible; the copy is
|
|
faster.
|
|
- Is shared storage (NFS) worth adding instead? It gives one MachineDeployment spread over both
|
|
nodes, and the control plane can use both nodes too.
|
|
- Behaviour when a cross-node clone fails: the scheduler may keep picking the same node. This is
|
|
inferred from the code, not tested.
|
|
|
|
## Follow-ups
|
|
|
|
- The owner chose: no shared storage for now; copy template 101 to `pv1-2` as 9101; keep
|
|
`lab1-workers` and add `lab1-workers-pv1-2`. Plan:
|
|
[2026-09-29-2315-pv1-2-workers](../plans/2026-09-29-2315-pv1-2-workers.md). It covers the bridge
|
|
fix, the template copy, the second MD, the CA limits and a decision row superseding #7.
|
|
- Update the "Lab facts" in CLAUDE.md/AGENTS.md: PVE cluster `homelab`, `pv1-2` at 192.168.0.10.
|
|
|
|
## Outcome (2026-09-30)
|
|
|
|
Done as planned, and confirmed on the lab:
|
|
- template 9101 on `pv1-2`;
|
|
- MDs `lab1-workers-pv1-1` (max 6) + `lab1-workers-pv1-2` (max 12);
|
|
- a balanced scale-up of 4 + 4 across both PVE nodes.
|
|
|
|
Two cluster-autoscaler findings from the scale test (no balancing, and throttling from uncached
|
|
discovery) led to decision #19. Details are in the
|
|
[execution log](../plans-executions/2026-09-29-2315-pv1-2-workers.md).
|