kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

lab1 workers on both PVE nodes: per-node template + MD, balanced CA

Browse Source 
PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage:
- template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and
  lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50;
  lab1-workers renamed for symmetry (decision #18, #7 superseded)
- CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels
  with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19).
  Scale test: 16 pods -> 4 + 4 across both nodes.
- runbook, image-builder copy procedure, fixes tables, future task to report
  three CA clusterapi bugs upstream, lab facts, changelog, execution log

Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-30 11:07:17 +02:00
co-authored by Claude Opus 5.5
parent 5e5b6c506a
commit 6ef55778ff
14 changed files with 476 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
configs/capi/clusterctl.yaml.example
+3 -2
View File
@@ -4,8 +4,9 @@
## workload defaults
PROXMOX_SOURCENODE: "pv1-1" # node holding the template
TEMPLATE_VMID: "<vmid of ubuntu-2404-kube-v1.35.9>"
ALLOWED_NODES: "[pv1-1]" # more nodes only with shared storage
TEMPLATE_VMID: "<vmid of ubuntu-2404-kube-v1.35.9>" # 101 on pv1-1 (copy 9101 on pv1-2)
ALLOWED_NODES: "[pv1-1]" # local-lvm: only the template's node. More PVE nodes = one
# worker MD + PMT per node by hand (lab1.yaml, decision #18)
VM_SSH_KEYS: "ssh-ed25519 AAAA..."
## network (VLAN 4 on vmbr0 — add `vlan: 4` to the network device in ProxmoxMachineTemplates)
configs/capi/lab1.yaml
+106 -7
View File
@@ -20,6 +20,17 @@
# 7. MachineDeployment lab1-workers: autoscaler min/max + capacity annotations (runbook Step 5)
# 8. MachineDeployment lab1-workers: rollout maxSurge 0 / maxUnavailable 1 (memory-tight host)
# 9. ProxmoxCluster lab1: schedulerHints.memoryAdjustment 160 (CAPMOX memory overcommit, decision #17)
# 10. Second worker group on PVE node pv1-2 (decision #18): MachineDeployment lab1-workers-pv1-2 +
# ProxmoxMachineTemplate lab1-worker-2g-pv1-2 (template copy 9101, allowedNodes [pv1-2]).
# local-lvm is node-local, so every node needs its own template VMID and its own MD.
# 11. ProxmoxCluster lab1: node IP pool .31-.39 -> .31-.50 (20 addresses). Max sizes 6 (pv1-1) +
# 12 (pv1-2) + 1 CP = 19; one address stays free for the KCP rollout surge (CAPMOX pools also
# hold a freed address for 60 s before reuse).
# 12. Both worker MDs: label node.cluster.x-k8s.io/pool=workers (template + capacity labels
# annotation) for CA --balancing-label; capacity key maximum-pods -> maxPods.
# 13. Symmetric per-node names (owner's request): MachineDeployment lab1-workers -> lab1-workers-pv1-1,
# ProxmoxMachineTemplate lab1-worker-2g -> lab1-worker-2g-pv1-1 (+ explicit allowedNodes [pv1-1]).
# A rename is a new object: the old MD and PMT are deleted by hand once the old MD is at 0.
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
@@ -59,7 +70,7 @@ spec:
- 1.1.1.1
ipv4Config:
addresses:
- 192.168.4.31-192.168.4.39
- 192.168.4.31-192.168.4.50
gateway: 192.168.4.8
prefix: 24
# CAPMOX reserves the configured memory of every non-template VM, stopped ones included, against
@@ -235,15 +246,18 @@ kind: MachineDeployment
metadata:
annotations:
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
# scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "6"
# scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g-pv1-1
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets 1 × numCores 4
capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" # bootVolume sizeGb 40
capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
capacity.cluster-autoscaler.kubernetes.io/maxPods: "110" # CA key; "maximum-pods" is ignored
# Labels for the from-zero template node. CA 1.35.2 reads MD labels from a path v1beta2 doesn't
# have, and CAPMOX publishes no nodeInfo, so without this os/arch are "" on the template.
capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
labels:
cluster.x-k8s.io/cluster-name: lab1
name: lab1-workers
name: lab1-workers-pv1-1
namespace: default
spec:
clusterName: lab1
@@ -263,6 +277,8 @@ spec:
labels:
cluster.x-k8s.io/cluster-name: lab1
node-role.kubernetes.io/node: ""
# CA --balancing-label; node.cluster.x-k8s.io/* is synced to the Node by CAPI
node.cluster.x-k8s.io/pool: workers
spec:
bootstrap:
configRef:
@@ -273,17 +289,21 @@ spec:
infrastructureRef:
apiGroup: infrastructure.cluster.x-k8s.io
kind: ProxmoxMachineTemplate
name: lab1-worker-2g
name: lab1-worker-2g-pv1-1
version: v1.35.9
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: ProxmoxMachineTemplate
metadata:
name: lab1-worker-2g
name: lab1-worker-2g-pv1-1
namespace: default
spec:
template:
spec:
# Pinned like the pv1-2 template, independent of ProxmoxCluster.allowedNodes: template 101
# lives on pv1-1's local-lvm.
allowedNodes:
- pv1-1
disks:
bootVolume:
disk: scsi0
@@ -301,6 +321,85 @@ spec:
sourceNode: pv1-1
templateID: 101
---
# Worker group on pv1-2. Keep in sync with lab1-workers-pv1-1 / lab1-worker-2g-pv1-1 except node,
# template and max-size: CA --balancing-label skips resource checks, so capacity must stay identical.
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
annotations:
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12"
# scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g-pv1-2
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets 1 × numCores 4
capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" # bootVolume sizeGb 40
capacity.cluster-autoscaler.kubernetes.io/maxPods: "110" # CA key; "maximum-pods" is ignored
# Labels for the from-zero template node. CA 1.35.2 reads MD labels from a path v1beta2 doesn't
# have, and CAPMOX publishes no nodeInfo, so without this os/arch are "" on the template.
capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
labels:
cluster.x-k8s.io/cluster-name: lab1
name: lab1-workers-pv1-2
namespace: default
spec:
clusterName: lab1
rollout:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
selector:
matchLabels: null
template:
metadata:
labels:
cluster.x-k8s.io/cluster-name: lab1
node-role.kubernetes.io/node: ""
# CA --balancing-label; node.cluster.x-k8s.io/* is synced to the Node by CAPI
node.cluster.x-k8s.io/pool: workers
spec:
bootstrap:
configRef:
apiGroup: bootstrap.cluster.x-k8s.io
kind: KubeadmConfigTemplate
name: lab1-worker
clusterName: lab1
infrastructureRef:
apiGroup: infrastructure.cluster.x-k8s.io
kind: ProxmoxMachineTemplate
name: lab1-worker-2g-pv1-2
version: v1.35.9
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: ProxmoxMachineTemplate
metadata:
name: lab1-worker-2g-pv1-2
namespace: default
spec:
template:
spec:
# Overrides ProxmoxCluster.allowedNodes ([pv1-1], kept for the control plane): template 9101
# lives on pv1-2's local-lvm, and PVE refuses cross-node clones from local storage.
allowedNodes:
- pv1-2
disks:
bootVolume:
disk: scsi0
sizeGb: 40
full: true
memoryMiB: 2048
network:
networkDevices:
- bridge: vmbr0
model: virtio
name: net0
vlan: 4
numCores: 4
numSockets: 1
sourceNode: pv1-2
templateID: 9101
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
metadata: