kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

lab1 workers on both PVE nodes: per-node template + MD, balanced CA

Browse Source 
PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage:
- template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and
  lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50;
  lab1-workers renamed for symmetry (decision #18, #7 superseded)
- CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels
  with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19).
  Scale test: 16 pods -> 4 + 4 across both nodes.
- runbook, image-builder copy procedure, fixes tables, future task to report
  three CA clusterapi bugs upstream, lab facts, changelog, execution log

Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-30 11:07:17 +02:00
co-authored by Claude Opus 5.5
parent 5e5b6c506a
commit 6ef55778ff
14 changed files with 476 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-1-proxmox-capi.md
+71 -26
View File
@@ -1,7 +1,8 @@
# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler
Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload
cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
cluster `lab1` as Proxmox VMs on a two-node PVE cluster; cluster-autoscaler scales its workers 0–18
in one node group per PVE node, balanced, including scale-from-zero.
No custom code in this phase — only assembling upstream components.
Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md).
@@ -10,9 +11,11 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
| Role | Where | Size |
| --- | --- | --- |
| PVE cluster `homelab` | `pv1-1` 192.168.0.8 (API endpoint for CAPMOX), `pv1-2` 192.168.0.10; `local-lvm` on each, no shared storage | 16 / 12 cores, 29.2 / 40.9 GiB |
| Node template | `ubuntu-2404-kube-v1.35.9`: VMID 101 on `pv1-1`, copy 9101 on `pv1-2` | 20 GB |
| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 2 GB RAM |
| `lab1` control plane | 1 VM on `pv1-1`, VIP `.30` via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployments `lab1-workers-pv1-1` (min 0 / max 6) and `lab1-workers-pv1-2` (min 0 / max 12); node IP pool `.31–.50` | 4 vCPU, 2 GB RAM |
## Checklist
@@ -39,11 +42,24 @@ pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser #
Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`.
Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP
for the control plane endpoint in the same subnet.
The token and ACLs are cluster-wide in a PVE cluster, so a node that joins later needs nothing here.
Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node
requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template.
Network: `vmbr0` is VLAN-aware on **every** PVE node, and VMs live in VLAN 4. On a new node, add
`bridge-vlan-aware yes` + `bridge-vids 2-10` (or wider) to `iface vmbr0`, run `ifreload -a`, and
trunk VLAN 4 on its switch port. CAPMOX writes `bridge`/`vlan` into the VM without checking them.
Reserve a node IP range outside DHCP and one VIP for the control plane endpoint in the same subnet.
Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2.
- PVE refuses to clone a VM to another node from non-shared storage, and CAPMOX v0.9.1 always
passes the scheduled node as the clone target.
- `templateSelector` needs exactly one match in the whole PVE cluster, so it can't pick a
per-node copy.
- Without shared storage, therefore, every node gets its own template VMID (VMIDs are
cluster-wide) and its own worker MachineDeployment. Each MD's `ProxmoxMachineTemplate` sets
`sourceNode`, `templateID` and `allowedNodes: [<node>]`.
- `ProxmoxCluster.allowedNodes` stays `[pv1-1]` for the control plane. The copy procedure is in
[image-builder-proxmox.md](image-builder-proxmox.md#template-on-other-pve-nodes), and the
reasoning in [decision #18](decisions.md).
## Step 2 — management cluster
@@ -153,32 +169,47 @@ kubectl -n kube-system get pods -o wide # cilium DaemonSet 2/
## Step 5 — autoscaling annotations
Both worker MDs carry the same block. Only the name and max-size differ (6 on `pv1-1`, 12 on
`pv1-2`):
```yaml
metadata:
name: lab1-workers
name: lab1-workers-pv1-2
annotations:
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12"
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores
capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
capacity.cluster-autoscaler.kubernetes.io/maxPods: "110" # not "maximum-pods" (ignored)
capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
spec:
template:
metadata:
labels:
node.cluster.x-k8s.io/pool: workers # synced to the Node by CAPI; CA --balancing-label
```
They are part of `MachineDeployment/lab1-workers` in [configs/capi/lab1.yaml](../configs/capi/lab1.yaml);
re-apply the file (the only diff is these annotations; `replicas` isn't in the manifest, so it
stays as is):
They are part of both MDs in [configs/capi/lab1.yaml](../configs/capi/lab1.yaml). Re-apply the file.
`replicas` isn't in the manifest, so it stays as it is, and a new MD starts at the min-size:
```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl diff -f configs/capi/lab1.yaml # expect only the six annotations on lab1-workers
kubectl diff -f configs/capi/lab1.yaml
kubectl apply -f configs/capi/lab1.yaml
```
Capacity annotations are required for scale-from-zero because CAPMOX does not publish
`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints`
annotations if pods target the group by selector/toleration. They have no effect until
cluster-autoscaler runs (Step 6).
Scale-from-zero needs capacity annotations because CAPMOX does not publish `status.capacity`, and
they must match the `ProxmoxMachineTemplate`. The `.../labels` annotation is needed as well, for two
reasons in CA 1.35.2:
- It reads MD template labels from `spec.template.spec.metadata.labels`, a path that doesn't exist in
CAPI v1beta2.
- CAPMOX publishes no `status.nodeInfo`, so the from-zero template node would otherwise get
`kubernetes.io/os=""` and `arch=""`. A pod with an `os: linux` selector then can't trigger a
scale-up.
Add a `.../taints` annotation if pods target the group by toleration. None of this has any effect
until cluster-autoscaler runs (Step 6).
## Step 6 — cluster-autoscaler
@@ -196,14 +227,22 @@ doesn't cover infrastructure templates, and without them the first main loop han
troubleshooting). Health check: the log shows `Starting main loop` followed by regular
`Autoscaler loop triggered` lines, and `cluster-autoscaler-status` on lab1 leaves `Initializing`.
CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`. Its log must
show the node group; the line is logged at `v: 4` on every loop (`clusterapi_controller.go`,
CA 1.35.2):
CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`.
```bash
kubectl logs deploy/ca-clusterapi-cluster-autoscaler -n default --kubeconfig ~/.kube/lab-mgmt.kubeconfig \
| grep 'discovered node group'
# discovered node group: MachineDeployment/default/lab1-workers (min: 0, max: 5, replicas: 1)
**Settings in the values that matter with two node groups:**
- `balancing-label: node.cluster.x-k8s.io/pool`. CA compares groups only on this label, so a
scale-up is split across both PVE nodes.
- `scan-interval: 1m`. Unschedulable pods and finished scale-ups/downs still start a loop
immediately; only the idle scale-down evaluation runs less often.
- `v: 2`.
**Checking the node groups:** `discovered node group` is only logged at `v: 4`
(`clusterapi_controller.go:800`). At `v: 2`, check the node groups in the status ConfigMap (below)
instead. A balanced scale-up logs:
```text
Splitting scale-up between 2 similar node groups: {MachineDeployment/default/lab1-workers-pv1-2, MachineDeployment/default/lab1-workers-pv1-1}
Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}]
```
The status ConfigMap and the leader-election lock live on the workload cluster, not on the management
@@ -237,7 +276,13 @@ CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocata
| MD rollout never finishes and CA never scales down: CA logs `Node … will be skipped as it is currently under rollout`, new ProxmoxMachine `cannot reserve …B of memory` | deadlock. Default rollout surge (`maxSurge 1 / maxUnavailable 0`) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (`ScaleDownNodeUpgradeProcessor`) | `spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1}` on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory |
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
| `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` |
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maxPods (`maximum-pods` is not a CA key and is silently ignored) |
| a pod with `nodeSelector: kubernetes.io/os: linux` doesn't trigger scale-from-zero | CA 1.35.2 builds the template node's os/arch labels from the infra template's `status.nodeInfo`. CAPMOX has none, so they become `""`. MD template labels aren't read either (CA reads the non-existent v1beta2 path `spec.template.spec.metadata.labels`) | `capacity.cluster-autoscaler.kubernetes.io/labels: "kubernetes.io/os=linux,kubernetes.io/arch=amd64,…"` on the MD |
| scale-up goes into one group only: `No similar node groups found`, next loop the second group (6 + 2 instead of 4 + 4) | the generic comparator fails when one group's template comes from a real node (cached for `--node-info-cache-expire-time`, default 10 years: ~1.78 Gi allocatable, os/arch set) and the other from the synthetic from-zero template (2 Gi, os/arch empty). The mismatch reason is only logged at V(5). `cluster.x-k8s.io/deployment-name` is **not** the cause, because CA never reads it | same label on both MDs (`node.cluster.x-k8s.io/pool`: template label + capacity labels annotation) and `balancing-label: node.cluster.x-k8s.io/pool` in the CA values. This turns off the resource comparison, so keep the MDs' capacity identical |
| CA loop takes 8–26 s; the log is full of `Waited before sending request … client-side throttling` on `/api`, `/apis`, `/apis/infrastructure.cluster.x-k8s.io/v1alpha2` | the CA 1.35.2 clusterapi provider runs **uncached** discovery (`getKindPreferredVersion`) for every MD at 0 replicas, on every `NodeGroups()` call (about 15× per loop). The client-go default is 5 QPS / burst 10. There is no upstream fix up to 1.36.1 | fewer loops: `scan-interval: 1m` (scale-up is still triggered by pending pods at once). Alternatively `kube-client-qps`/`-burst` (more requests, not chosen). `v: 2` hides the noise. Upstream fix: [future tasks](future-tasks.md) |
| `W clusterstate.go:510 Failed to find acceptable ranges for MachineDeployment/…` | the MD was created during a loop: acceptable ranges are snapshotted at the start of the loop, and the group showed up in a later `NodeGroups()` call | none if it happens once. If it repeats every loop, the group is dropping out of `NodeGroups()` (scale-from-zero check failing: infra template unreadable, capacity annotations missing) |
| ProxmoxMachine `VMProvisionFailed` … `can't clone VM to node '<node>' (VM uses local storage)` | the scheduled node (`allowedNodes`) isn't the node holding the template, and the template is on non-shared storage. A cross-node clone is then impossible, whatever `full`/`storage` are set to | per-node template copy + per-node MD with `allowedNodes: [<node>]` (see Step 0); or shared storage |
| VMs on a new PVE node get no IP (no DHCP/IPAM connectivity on VLAN 4) | `vmbr0` on that node isn't VLAN-aware, or its switch port doesn't trunk VLAN 4 | `bridge-vlan-aware yes` + `bridge-vids` in `/etc/network/interfaces`, `ifreload -a`; check via `pvesh get /nodes/<node>/network/vmbr0` |
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |