kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

lab1 workers on both PVE nodes: per-node template + MD, balanced CA

Browse Source 
PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage:
- template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and
  lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50;
  lab1-workers renamed for symmetry (decision #18, #7 superseded)
- CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels
  with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19).
  Scale test: 16 pods -> 4 + 4 across both nodes.
- runbook, image-builder copy procedure, fixes tables, future task to report
  three CA clusterapi bugs upstream, lab facts, changelog, execution log

Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-30 11:07:17 +02:00
co-authored by Claude Opus 5.5
parent 5e5b6c506a
commit 6ef55778ff
14 changed files with 476 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/decisions.md
+3 -1
View File
@@ -15,7 +15,7 @@ image-builder fixes log; their dates are when the decision was recorded, not nec
| 4 | 2026-09-24 | image-builder overrides go in a `PACKER_VAR_FILES` JSON | `PACKER_FLAGS` | flags were silently lost when `PACKER_FLAGS` was re-exported ([fixes log #4](image-builder-proxmox.md#fixes-log)) | active |
| 5 | 2026-09-24 | Cilium installed by Helm with `ipam.mode=kubernetes`, pod CIDR `10.244.0.0/16` | ClusterResourceSet CNI flavors; pod CIDR `192.168.0.0/16` | keeps the CNI version under our control; `192.168.0.0/16` overlaps the lab LAN | active |
| 6 | 2026-09-24 | Scale-from-zero via `capacity.cluster-autoscaler.kubernetes.io/*` annotations on the MachineDeployment | infrastructure provider publishing `status.capacity` | CAPMOX does not publish `status.capacity` on `ProxmoxMachineTemplate` | active |
| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | active |
| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | superseded by #18 |
| 8 | 2026-09-26 | Phase 2: spike CAPLV first; own provider if insufficient; KubeVirt as fallback ([phase-2](phase-2-kvm-provider.md#suggested-approach)) | CAPK + KubeVirt, Metal3, own provider from the start | CAPLV shares CAPI 1.12 with CAPMOX and fits plain libvirt hosts directly | proposed |
| 9 | 2026-09-26 | Phases 2 and 3 are independent and can be done in either order | strict sequence | no technical dependency between the KVM provider and hosted control planes | active |
| 10 | 2026-09-26 | Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog ([session](sessions/2026-09-26-2355-history-conventions.md)) | chat history only; raw transcripts committed | research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets | active |
@@ -26,3 +26,5 @@ image-builder fixes log; their dates are when the decision was recorded, not nec
| 15 | 2026-09-27 | `lab1` workers 4 vCPU / **4 GiB** (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker | 8 GiB workers with 2Gi inflate pods | owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape ([configs/capi/lab1.yaml](../configs/capi/lab1.yaml)) | superseded by #16 |
| 16 | 2026-09-28 | `lab1` workers 4 vCPU / **2 GiB** (new `ProxmoxMachineTemplate/lab1-worker-2g`, rolled in via the MD); inflate pods 1500m / **512Mi**, still CPU-bound at 2 per worker | keep 4 GiB and delete stopped VMs; CAPMOX `memoryAdjustment` overcommit | CAPMOX reserves the configured memory of every VM, stopped ones included, so 4 GiB workers ran out after the first scale-up ([troubleshooting](phase-1-proxmox-capi.md#troubleshooting)); 2 GiB (~1.78 GiB allocatable) still leaves room for kubelet/containerd plus real pods, and max 5 workers now fits the host | proposed |
| 17 | 2026-09-28 | CAPMOX memory overcommit on `lab1`: `ProxmoxCluster.spec.schedulerHints.memoryAdjustment: 160` (46.7 GiB reservable on the 29.16 GiB `pv1-1`); MD rollout `maxSurge 0 / maxUnavailable 1` | 100% (default); 120% (minimum sizing for max 5 × 2 GiB workers); 0 (memory check off); deleting stopped VMs | CAPMOX counts the configured memory of stopped VMs (12 GiB here), and no release can use real free memory; 160% keeps a bound, and host-side protection is KSM + swap, not ballooning ([session](sessions/2026-09-28-1007-capmox-memory-overcommit.md)) | proposed |
| 18 | 2026-09-30 | PVE is now a 2-node cluster (`pv1-1` + `pv1-2`), no shared storage: a **template copy per node** (101 on `pv1-1`, 9101 on `pv1-2`, copied by clone + offline migrate), **one worker MD per node** (`lab1-workers-pv1-1` max 6, `lab1-workers-pv1-2` max 12, PMT `allowedNodes: [<node>]`), control plane stays on `pv1-1`, node IP pool `.31–.50` | shared storage (NFS); a second image-builder run on `pv1-2`; `templateSelector` with the same tags on both nodes; keep `lab1-workers` and only add a second MD | PVE refuses cross-node clones from local storage and CAPMOX v0.9.1 always passes the scheduled node as target; `templateSelector` needs exactly one match cluster-wide; a copy is byte-identical; owner wants no shared storage for now and symmetric names ([plan](plans/2026-09-29-2315-pv1-2-workers.md), [session](sessions/2026-09-29-2311-pve-two-node-cluster.md)) | active |
| 19 | 2026-09-30 | cluster-autoscaler with two node groups: `balancing-label: node.cluster.x-k8s.io/pool` (label on both MDs + `capacity…/labels` annotation), `scan-interval: 1m`, `v: 2`; client QPS left at the default 5/10 | generic similarity comparator; `balancing-ignore-label` for `deployment-name`; `kube-client-qps 50 / burst 100`; `min-size 1` per group; `CAPI_VERSION=v1beta1` | the comparator fails between a cached real-node template and a from-zero template (6+2 instead of 4+4); CA 1.35.2 runs uncached discovery per zero-replica MD ~15× per loop, and the owner prefers fewer requests over a higher rate — pending pods still trigger loops immediately ([execution log](plans-executions/2026-09-29-2315-pv1-2-workers.md)) | active |
docs/future-tasks.md
+35 -1
View File
@@ -3,6 +3,40 @@
Deferred work, newest first. When a task is picked up it gets a plan in `docs/plans/`; then tick
it here and link the plan.
## - [ ] Report three cluster-autoscaler clusterapi-provider bugs upstream
**Added:** 2026-09-30 09:28 CEST · **Why deferred:** worked around in config (decision #19);
reporting is not blocking.
**Problem.** Three bugs in CA 1.35.2 `cloudprovider/clusterapi`, all still on master and 1.36.1 as
of 2026-09-29, found in [the pv1-2 execution log](plans-executions/2026-09-29-2315-pv1-2-workers.md):
1. **Uncached discovery.**
- `clusterapi_provider.go:195` builds a `memory.NewMemCacheClient`, but passes the raw
discovery client to the controller (`:209`).
- `getKindPreferredVersion` therefore does 3 GETs per MD at 0 replicas on every `NodeGroups()`
call, about 15× per loop. The loop took 18 s at 5 QPS.
2. **`Labels()` reads the wrong path.** It reads `spec.template.spec.metadata.labels`
(`clusterapi_unstructured.go:207`), which doesn't exist in CAPI v1beta2, so MD template labels
never reach the from-zero template node. The unit tests use the same wrong path.
3. **Empty os/arch labels.** For an infra template without `status`, as with CAPMOX,
`InstanceSystemInfo()` returns an empty struct. The template node then gets
`kubernetes.io/os=""` and `arch=""`, which override the generic `linux`/`amd64`.
**Options:**
- Issues only.
- Issues plus small PRs: pass the cached client, fix the path, skip empty nodeInfo fields.
- Carry a patched CA image: rejected, versions must stay upstream and pinned.
**Cost.** About 1–2 h for issues with source references. PRs would add the CLA and a test for each.
**Changes when done:**
- Link the issues/PRs in `docs/phase-1-proxmox-capi.md` troubleshooting.
- Once fixed in a 1.35.x patch release: bump CA in `docs/versions.md`, then revisit
`scan-interval` and the `capacity…/labels` annotation.
**Run by the owner:** nothing on the lab. Filing on GitHub is the owner's call, since it goes out
under their account.
## - [ ] Namespace per workload cluster (`lab1` out of `default`)
**Added:** 2026-09-28 09:29 CEST · **Why deferred:** keep phase 1 moving; `default` works for now.
@@ -36,7 +70,7 @@ next cluster.
(`helm uninstall ca -n default` first if it was installed in `default`).
- `ansible/group_vars/all.yml`: `inspector_workload_clusters` entry `{ name: lab1, namespace: lab1 }`.
- `docs/phase-1-proxmox-capi.md`: Steps 3, 5 and 6 (namespace in commands and log expectations,
`MachineDeployment/lab1/lab1-workers`).
`MachineDeployment/lab1/lab1-workers-pv1-{1,2}`).
- `.claude/agents/cluster-inspector.md`: the "Where things live" table says CA runs in `default`.
- `docs/decisions.md`: new row (namespace per workload cluster).
docs/image-builder-proxmox.md
+27 -1
View File
@@ -2,7 +2,8 @@
Result: Proxmox template `ubuntu-2404-kube-v1.35.9` on node `pv1-1` — Ubuntu 24.04.4, containerd,
kubeadm/kubelet/kubectl v1.35.9, pre-pulled control-plane images, cloud-init, qemu-guest-agent.
Its VMID is `TEMPLATE_VMID` for CAPMOX.
Its VMID (101) is `TEMPLATE_VMID` for CAPMOX. A byte-identical copy, 9101, sits on `pv1-2`
([Template on other PVE nodes](#template-on-other-pve-nodes)).
## Build host prerequisites (Ubuntu 24.04)
@@ -88,3 +89,28 @@ qm stop $ID && qm destroy $ID --purge
```
kubelet crash-looping before `kubeadm init/join` is expected.
## Template on other PVE nodes
Without shared storage, a template on `local-lvm` can only be cloned on its own node. That holds
for PVE and for CAPMOX: `can't clone VM to node '<target>' (VM uses local storage)`. Each additional
PVE node therefore gets its own copy under its own VMID, since VMIDs are cluster-wide.
Copy rather than rebuild: the copy is byte-identical, whereas a second Packer run would pick up
newer apt packages. The source must have no disk or CD-ROM on non-shared storage other than
`local-lvm`; check with `qm config 101`.
```bash
# on the node holding the template (pv1-1); 9101 = copy for pv1-2
qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm # clone stays on pv1-1
qm migrate 9101 pv1-2 --with-local-disks # offline: copies the LVs
ssh root@192.168.0.10 qm template 9101
```
Then run the smoke test on the target node with `TPL=9101`. It also proves that VLAN 4 works on
that node's `vmbr0`. The copy is referenced by `templateID: 9101` + `sourceNode: pv1-2` in that
node's `ProxmoxMachineTemplate`
([phase 1, Step 0](phase-1-proxmox-capi.md#step-0--proxmox-prep)).
A new template build (a new Kubernetes version) means building on `pv1-1` and then repeating the
copy for every other node, with a new VMID pair per version.
docs/phase-1-proxmox-capi.md
+71 -26
View File
@@ -1,7 +1,8 @@
# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler
Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload
cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
cluster `lab1` as Proxmox VMs on a two-node PVE cluster; cluster-autoscaler scales its workers 0–18
in one node group per PVE node, balanced, including scale-from-zero.
No custom code in this phase — only assembling upstream components.
Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md).
@@ -10,9 +11,11 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
| Role | Where | Size |
| --- | --- | --- |
| PVE cluster `homelab` | `pv1-1` 192.168.0.8 (API endpoint for CAPMOX), `pv1-2` 192.168.0.10; `local-lvm` on each, no shared storage | 16 / 12 cores, 29.2 / 40.9 GiB |
| Node template | `ubuntu-2404-kube-v1.35.9`: VMID 101 on `pv1-1`, copy 9101 on `pv1-2` | 20 GB |
| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 2 GB RAM |
| `lab1` control plane | 1 VM on `pv1-1`, VIP `.30` via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployments `lab1-workers-pv1-1` (min 0 / max 6) and `lab1-workers-pv1-2` (min 0 / max 12); node IP pool `.31–.50` | 4 vCPU, 2 GB RAM |
## Checklist
@@ -39,11 +42,24 @@ pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser #
Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`.
Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP
for the control plane endpoint in the same subnet.
The token and ACLs are cluster-wide in a PVE cluster, so a node that joins later needs nothing here.
Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node
requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template.
Network: `vmbr0` is VLAN-aware on **every** PVE node, and VMs live in VLAN 4. On a new node, add
`bridge-vlan-aware yes` + `bridge-vids 2-10` (or wider) to `iface vmbr0`, run `ifreload -a`, and
trunk VLAN 4 on its switch port. CAPMOX writes `bridge`/`vlan` into the VM without checking them.
Reserve a node IP range outside DHCP and one VIP for the control plane endpoint in the same subnet.
Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2.
- PVE refuses to clone a VM to another node from non-shared storage, and CAPMOX v0.9.1 always
passes the scheduled node as the clone target.
- `templateSelector` needs exactly one match in the whole PVE cluster, so it can't pick a
per-node copy.
- Without shared storage, therefore, every node gets its own template VMID (VMIDs are
cluster-wide) and its own worker MachineDeployment. Each MD's `ProxmoxMachineTemplate` sets
`sourceNode`, `templateID` and `allowedNodes: [<node>]`.
- `ProxmoxCluster.allowedNodes` stays `[pv1-1]` for the control plane. The copy procedure is in
[image-builder-proxmox.md](image-builder-proxmox.md#template-on-other-pve-nodes), and the
reasoning in [decision #18](decisions.md).
## Step 2 — management cluster
@@ -153,32 +169,47 @@ kubectl -n kube-system get pods -o wide # cilium DaemonSet 2/
## Step 5 — autoscaling annotations
Both worker MDs carry the same block. Only the name and max-size differ (6 on `pv1-1`, 12 on
`pv1-2`):
```yaml
metadata:
name: lab1-workers
name: lab1-workers-pv1-2
annotations:
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12"
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores
capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
capacity.cluster-autoscaler.kubernetes.io/maxPods: "110" # not "maximum-pods" (ignored)
capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
spec:
template:
metadata:
labels:
node.cluster.x-k8s.io/pool: workers # synced to the Node by CAPI; CA --balancing-label
```
They are part of `MachineDeployment/lab1-workers` in [configs/capi/lab1.yaml](../configs/capi/lab1.yaml);
re-apply the file (the only diff is these annotations; `replicas` isn't in the manifest, so it
stays as is):
They are part of both MDs in [configs/capi/lab1.yaml](../configs/capi/lab1.yaml). Re-apply the file.
`replicas` isn't in the manifest, so it stays as it is, and a new MD starts at the min-size:
```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl diff -f configs/capi/lab1.yaml # expect only the six annotations on lab1-workers
kubectl diff -f configs/capi/lab1.yaml
kubectl apply -f configs/capi/lab1.yaml
```
Capacity annotations are required for scale-from-zero because CAPMOX does not publish
`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints`
annotations if pods target the group by selector/toleration. They have no effect until
cluster-autoscaler runs (Step 6).
Scale-from-zero needs capacity annotations because CAPMOX does not publish `status.capacity`, and
they must match the `ProxmoxMachineTemplate`. The `.../labels` annotation is needed as well, for two
reasons in CA 1.35.2:
- It reads MD template labels from `spec.template.spec.metadata.labels`, a path that doesn't exist in
CAPI v1beta2.
- CAPMOX publishes no `status.nodeInfo`, so the from-zero template node would otherwise get
`kubernetes.io/os=""` and `arch=""`. A pod with an `os: linux` selector then can't trigger a
scale-up.
Add a `.../taints` annotation if pods target the group by toleration. None of this has any effect
until cluster-autoscaler runs (Step 6).
## Step 6 — cluster-autoscaler
@@ -196,14 +227,22 @@ doesn't cover infrastructure templates, and without them the first main loop han
troubleshooting). Health check: the log shows `Starting main loop` followed by regular
`Autoscaler loop triggered` lines, and `cluster-autoscaler-status` on lab1 leaves `Initializing`.
CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`. Its log must
show the node group; the line is logged at `v: 4` on every loop (`clusterapi_controller.go`,
CA 1.35.2):
CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`.
```bash
kubectl logs deploy/ca-clusterapi-cluster-autoscaler -n default --kubeconfig ~/.kube/lab-mgmt.kubeconfig \
| grep 'discovered node group'
# discovered node group: MachineDeployment/default/lab1-workers (min: 0, max: 5, replicas: 1)
**Settings in the values that matter with two node groups:**
- `balancing-label: node.cluster.x-k8s.io/pool`. CA compares groups only on this label, so a
scale-up is split across both PVE nodes.
- `scan-interval: 1m`. Unschedulable pods and finished scale-ups/downs still start a loop
immediately; only the idle scale-down evaluation runs less often.
- `v: 2`.
**Checking the node groups:** `discovered node group` is only logged at `v: 4`
(`clusterapi_controller.go:800`). At `v: 2`, check the node groups in the status ConfigMap (below)
instead. A balanced scale-up logs:
```text
Splitting scale-up between 2 similar node groups: {MachineDeployment/default/lab1-workers-pv1-2, MachineDeployment/default/lab1-workers-pv1-1}
Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}]
```
The status ConfigMap and the leader-election lock live on the workload cluster, not on the management
@@ -237,7 +276,13 @@ CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocata
| MD rollout never finishes and CA never scales down: CA logs `Node … will be skipped as it is currently under rollout`, new ProxmoxMachine `cannot reserve …B of memory` | deadlock. Default rollout surge (`maxSurge 1 / maxUnavailable 0`) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (`ScaleDownNodeUpgradeProcessor`) | `spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1}` on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory |
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
| `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` |
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maxPods (`maximum-pods` is not a CA key and is silently ignored) |
| a pod with `nodeSelector: kubernetes.io/os: linux` doesn't trigger scale-from-zero | CA 1.35.2 builds the template node's os/arch labels from the infra template's `status.nodeInfo`. CAPMOX has none, so they become `""`. MD template labels aren't read either (CA reads the non-existent v1beta2 path `spec.template.spec.metadata.labels`) | `capacity.cluster-autoscaler.kubernetes.io/labels: "kubernetes.io/os=linux,kubernetes.io/arch=amd64,…"` on the MD |
| scale-up goes into one group only: `No similar node groups found`, next loop the second group (6 + 2 instead of 4 + 4) | the generic comparator fails when one group's template comes from a real node (cached for `--node-info-cache-expire-time`, default 10 years: ~1.78 Gi allocatable, os/arch set) and the other from the synthetic from-zero template (2 Gi, os/arch empty). The mismatch reason is only logged at V(5). `cluster.x-k8s.io/deployment-name` is **not** the cause, because CA never reads it | same label on both MDs (`node.cluster.x-k8s.io/pool`: template label + capacity labels annotation) and `balancing-label: node.cluster.x-k8s.io/pool` in the CA values. This turns off the resource comparison, so keep the MDs' capacity identical |
| CA loop takes 8–26 s; the log is full of `Waited before sending request … client-side throttling` on `/api`, `/apis`, `/apis/infrastructure.cluster.x-k8s.io/v1alpha2` | the CA 1.35.2 clusterapi provider runs **uncached** discovery (`getKindPreferredVersion`) for every MD at 0 replicas, on every `NodeGroups()` call (about 15× per loop). The client-go default is 5 QPS / burst 10. There is no upstream fix up to 1.36.1 | fewer loops: `scan-interval: 1m` (scale-up is still triggered by pending pods at once). Alternatively `kube-client-qps`/`-burst` (more requests, not chosen). `v: 2` hides the noise. Upstream fix: [future tasks](future-tasks.md) |
| `W clusterstate.go:510 Failed to find acceptable ranges for MachineDeployment/…` | the MD was created during a loop: acceptable ranges are snapshotted at the start of the loop, and the group showed up in a later `NodeGroups()` call | none if it happens once. If it repeats every loop, the group is dropping out of `NodeGroups()` (scale-from-zero check failing: infra template unreadable, capacity annotations missing) |
| ProxmoxMachine `VMProvisionFailed` … `can't clone VM to node '<node>' (VM uses local storage)` | the scheduled node (`allowedNodes`) isn't the node holding the template, and the template is on non-shared storage. A cross-node clone is then impossible, whatever `full`/`storage` are set to | per-node template copy + per-node MD with `allowedNodes: [<node>]` (see Step 0); or shared storage |
| VMs on a new PVE node get no IP (no DHCP/IPAM connectivity on VLAN 4) | `vmbr0` on that node isn't VLAN-aware, or its switch port doesn't trunk VLAN 4 | `bridge-vlan-aware yes` + `bridge-vids` in `/etc/network/interfaces`, `ifreload -a`; check via `pvesh get /nodes/<node>/network/vmbr0` |
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
docs/plans-executions/2026-09-29-2315-pv1-2-workers.md
+165 -3
View File
@@ -7,9 +7,9 @@ Pairs with [docs/plans/2026-09-29-2315-pv1-2-workers.md](../plans/2026-09-29-231
- [x] Step 0 — Record the plan
- [x] Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner)
- [x] Step 2 — Template 9101 on `pv1-2` (owner)
- [ ] Step 3 — Second worker MachineDeployment `lab1-workers-pv1-2`
- [ ] Step 4 — Scale test across both nodes
- [ ] Step 5 — Docs and history
- [x] Step 3 — Second worker MachineDeployment `lab1-workers-pv1-2`
- [x] Step 4 — Scale test across both nodes
- [x] Step 5 — Docs and history
## Step 0 — Record the plan (2026-09-29 23:15 CEST)
@@ -58,3 +58,165 @@ Worth noting:
unique.
- The copied `efidisk0` reports `size=528K` (the source says 4M). This doesn't matter: the template
boots SeaBIOS.
## Step 3 — second worker MachineDeployment (prepared 2026-09-30, apply pending)
Changes in `configs/capi/lab1.yaml`:
- new `MachineDeployment/lab1-workers-pv1-2` and `ProxmoxMachineTemplate/lab1-worker-2g-pv1-2`
(`sourceNode: pv1-2`, `templateID: 9101`, `allowedNodes: [pv1-2]`)
- `ProxmoxCluster.allowedNodes` stays `[pv1-1]` for the control plane
- header hand edits #10 and #11
Deviation from the plan: the IP pool is enlarged and the max sizes are raised (owner's request,
"20 nodes"):
- `ipv4Config.addresses` `192.168.4.31-192.168.4.39` → `192.168.4.31-192.168.4.50`
- `lab1-workers` max 5 → **6**, `lab1-workers-pv1-2` max **12**
- 6 + 12 + 1 CP = 19 of 20 addresses. The spare covers the KCP rollout surge, and CAPMOX pools
hold a freed address for 60 s (`AddressReuseGracePeriodSeconds`, `pkg/kubernetes/ipam/ipam.go:318`).
- An ARP probe from `lab-mgmt` found `.32–.60` silent; `.30` (VIP) and `.31` (CP) share the CP VM's
MAC.
- The owner confirmed `.31–.50` is outside the VLAN 4 DHCP range.
- Changing the pool is safe: the ProxmoxCluster webhook only validates the addresses, and the
controller copies them into the InClusterIPPool on every reconcile (`ipam.go:323-324`).
Sizing:
- `pv1-1`: 6 × 2 GiB within ~17.7 GiB of reservable headroom (160 %).
- `pv1-2`: 12 × 2 GiB of 65.4 GiB reservable (40.9 GiB physical); 48 vCPU on 12 cores, which is
fine for the pause-container `inflate` tests but CPU-bound under real load.
`yq` parse check: 9 documents; both worker PMTs point at the right node, template and `allowedNodes`.
Applied by the owner (`kubectl diff` / `kubectl apply -f configs/capi/lab1.yaml`). The CA
discovered `lab1-workers (max 6)` and `lab1-workers-pv1-2 (max 12)`.
## Step 4 — scale test, first run (2026-09-30 ~06:45 UTC)
The `inflate` Deployment had 16 pods (1500m / 512Mi each, 2 per worker).
- **Result:** 8 workers Ready by 06:47Z.
- `lab1-workers` 0→6: VMIDs 103–108 on `pv1-1`, IPs .34–.39.
- `lab1-workers-pv1-2` 0→2: VMIDs 109–110 on `pv1-2`, IPs .32–.33.
- No `VMProvisionFailed`. `status.proxmoxNode` and providerIDs are consistent. The pool had 9 of
20 addresses in use.
- **Checked with:** the `cluster-inspector` agent (read-only). Root causes came from the source
(`upstream-researcher`, CA tag `cluster-autoscaler-1.35.2`, CAPI v1.12.11, CAPMOX v0.9.1).
Two problems.
**1. Not balanced.** The log shows `Best option to resize: lab1-workers` → `No similar node groups
found` → `0->6 (max: 6)`. Only in the next loop did `lab1-workers-pv1-2` get `0->2`.
- The inspector guessed the `cluster.x-k8s.io/deployment-name` label. The source rules that out:
CA never reads it.
- What the source does show in CA 1.35.2 `cloudprovider/clusterapi`:
- `Labels()` reads `spec.template.spec.metadata.labels`, which doesn't exist in v1beta2
(`clusterapi_unstructured.go:207`). MD template labels therefore never reach the from-zero
template node.
- CAPMOX `ProxmoxMachineTemplate` has no `status`, so `kubernetes.io/os` and `arch` become `""` on
the template (`clusterapi_nodegroup.go:393-405`). A pod with `nodeSelector: kubernetes.io/os:
linux` would not trigger a scale-up from zero.
- The capacity key is `maxPods`; our `maximum-pods` was ignored, and 110 only applied because it
is the default.
- `MixedTemplateNodeInfoProvider` caches a template built from a real node for
`--node-info-cache-expire-time` (default 87600h). A group that has had nodes keeps that
template: ~1.78Gi allocatable, os/arch set, `node-role` label. A from-zero group gets the
synthetic template: 2Gi, empty os/arch. The generic comparator rejects that pair.
- That this cache caused this particular decision is inferred. The reason for a mismatch is only
logged at V(5).
- **Fix (`lab1.yaml` hand edit #12):**
- both MDs carry the template label `node.cluster.x-k8s.io/pool: workers`, which CAPI syncs to the
Node;
- the annotation `capacity.cluster-autoscaler.kubernetes.io/labels:
node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64`;
- `maximum-pods` → `maxPods`;
- CA `balancing-label: node.cluster.x-k8s.io/pool`, which compares only that label and turns off
the resource checks.
**2. Loop 8–26 s, throttled.**
- The log is full of `Waited before sending request … client-side throttling` on `/api`, `/apis`
and `/apis/infrastructure.cluster.x-k8s.io/v1alpha2` against the lab-mgmt API.
- Cause:
- For every MD at 0 replicas, `CanScaleFromZero` → `readInfrastructureReferenceResource` →
`getKindPreferredVersion` runs through the provider's **uncached** discovery client. The cached
one is built at `clusterapi_provider.go:195` but not passed to the controller.
- That is 3 GETs per MD per `NodeGroups()` call, and `NodeGroups()` runs about 15× per loop.
- The client uses QPS 5 / burst 10 (`--kube-client-qps` / `--kube-client-burst` defaults).
- There is no upstream fix up to 1.36.1 or master (2026-09-29).
- The owner rejected raising QPS (50/100) and wants fewer requests instead. Fix in
`configs/cluster-autoscaler/values.yaml`:
- `scan-interval: 1m` (was 10s). `loop/trigger.go:74-100`: unschedulable pods and finished
scale-ups/downs still start a loop immediately. Only idle loops are spaced out, so scale-down
takes up to about +1 min.
- `v: 2` (chart default 4). The throttling and "discovered node group" lines are V(3)/V(4).
- Considered and not taken:
- `min-size: 1`: no MD at 0 means no discovery, but two idle VMs.
- `CAPI_VERSION=v1beta1`: v1beta1 refs carry `apiVersion`, so no discovery. But v1beta1 is
deprecated and the path is untested with CAPMOX.
- An upstream patch to pass the cached discovery client (a future task).
## Rename of the pv1-1 group (owner's request, 2026-09-30)
This deviates from the plan: the plan kept `lab1-workers`. The owner wants symmetric per-node
names now that node names show the MD name (`lab1-workers-zldvn-*` next to
`lab1-workers-pv1-2-nlv6s-*`).
Changes in `configs/capi/lab1.yaml` (hand edit #13):
- `MachineDeployment/lab1-workers` → `lab1-workers-pv1-1`.
- `ProxmoxMachineTemplate/lab1-worker-2g` → `lab1-worker-2g-pv1-1`, now with explicit
`allowedNodes: [pv1-1]`.
A rename creates new objects. `kubectl apply` doesn't prune, so the owner deletes the old MD and PMT
once the old MD has scaled to 0. The timing is right because the CA was already scaling every worker
down (`inflate` at 0). The new MD comes up with 0 replicas: `replicas` is unset, and the CAPI webhook
defaults it to the autoscaler min-size annotation.
## Step 4 — re-test after the fixes (2026-09-30 07:23 UTC)
The owner applied the changes in this order:
1. `kubectl apply -f configs/capi/lab1.yaml`: new `lab1-workers-pv1-1` + `lab1-worker-2g-pv1-1`, the
pool label, `maxPods`, os/arch.
2. Deleted the old `MachineDeployment/lab1-workers` and `ProxmoxMachineTemplate/lab1-worker-2g`.
3. `helm upgrade --install ca … --version 9.59.0 -f configs/cluster-autoscaler/values.yaml`:
`scan-interval 1m`, `v 2`, `balancing-label`.
4. Scaled `inflate` 0 → 16.
The CA log (`--since=15m`):
```
orchestrator.go:752] Splitting scale-up between 2 similar node groups: {…/lab1-workers-pv1-2, …/lab1-workers-pv1-1}
orchestrator.go:263] Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}]
```
- **Placement:** ProxmoxMachines `lab1-workers-pv1-1-*` on `pv1-1` (VMIDs 103, 104, 107, 110) and
`lab1-workers-pv1-2-*` on `pv1-2` (105, 106, 108, 109).
- **Nodes:** all 8 Ready about 1–1.5 min after creation, with labels
`node.cluster.x-k8s.io/pool=workers` and `kubernetes.io/os=linux`.
- **Loop timing:** idle loops now run on the `1m0s timer`.
- **One transient warning:** `W clusterstate.go:510 Failed to find acceptable ranges for
MachineDeployment/default/lab1-workers-pv1-1`, once, in the loop during which the new MD was
created. Acceptable ranges are snapshotted at the start of the loop, and the group appeared in a
later `NodeGroups()` call. Harmless; it did not repeat.
Worth noting:
- `ProxmoxMachineTemplate/lab1-worker` (4 GiB, decision #15) is still in the cluster, and no MD
references it. Deleting it is left to the owner.
## Step 5 — docs and history (2026-09-30 09:28 CEST)
- `docs/decisions.md`: #7 → `superseded by #18`. #18 (template copy + MD per PVE node, pool
`.31–.50`, symmetric names) and #19 (CA `balancing-label`, `scan-interval 1m`, `v 2`, QPS left at
the default) are both `active`, since the lab confirmed them in Step 4.
- `docs/phase-1-proxmox-capi.md`:
- intro and topology (PVE cluster, templates, both MDs, pool);
- Step 0: bridge on every node, per-node template;
- Step 5: annotations incl. `maxPods`, `labels`, the pool label;
- Step 6: the settings for two groups, and what a balanced scale-up logs;
- troubleshooting: 7 rows (maxPods, empty os/arch, no balancing, throttling, the
acceptable-ranges warning, cross-node clone, VLAN on a new node).
- `docs/image-builder-proxmox.md`: section "Template on other PVE nodes"; the result line names
101/9101.
- `configs/capi/clusterctl.yaml.example`: comments on `TEMPLATE_VMID` / `ALLOWED_NODES`.
- `docs/future-tasks.md`: report the three CA clusterapi bugs upstream. The namespace task now uses
the new MD names.
- `README.md` (lab inventory, key decision 7), `CLAUDE.md` / `AGENTS.md` (lab facts):
PVE cluster, `pv1-2`, template 9101.
- `.claude/settings.json`: two inspector allowlist rules `md lab1-workers` → `md lab1-workers-pv1-1`.
- `CHANGELOG.md`: entry added after the owner confirmed the setup works (2026-09-30 11:05 CEST). The owner's pending #16/#17 work went in first as its own commit (`5e5b6c5`).
docs/sessions/2026-09-29-2311-pve-two-node-cluster.md
+11
View File
@@ -87,3 +87,14 @@ Conclusion:
[2026-09-29-2315-pv1-2-workers](../plans/2026-09-29-2315-pv1-2-workers.md). It covers the bridge
fix, the template copy, the second MD, the CA limits and a decision row superseding #7.
- Update the "Lab facts" in CLAUDE.md/AGENTS.md: PVE cluster `homelab`, `pv1-2` at 192.168.0.10.
## Outcome (2026-09-30)
Done as planned, and confirmed on the lab:
- template 9101 on `pv1-2`;
- MDs `lab1-workers-pv1-1` (max 6) + `lab1-workers-pv1-2` (max 12);
- a balanced scale-up of 4 + 4 across both PVE nodes.
Two cluster-autoscaler findings from the scale test (no balancing, and throttling from uncached
discovery) led to decision #19. Details are in the
[execution log](../plans-executions/2026-09-29-2315-pv1-2-workers.md).