kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
cluster-autoscaler-with-capi/configs/capi/lab1.yaml
T
kacerrandClaude Opus 5.5 6ef55778ff lab1 workers on both PVE nodes: per-node template + MD, balanced CA 
PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage:
- template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and
  lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50;
  lab1-workers renamed for symmetry (decision #18, #7 superseded)
- CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels
  with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19).
  Scale test: 16 pods -> 4 + 4 across both nodes.
- runbook, image-builder copy procedure, fixes tables, future task to report
  three CA clusterapi bugs upstream, lab facts, changelog, execution log

Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:07:17 +02:00

421 lines
14 KiB
YAML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Workload cluster lab1 (phase-1 Step 3). Generated 2026-09-27 with clusterctl v1.12.11:
#
# PROXMOX_SOURCENODE=pv1-1 TEMPLATE_VMID=101 ALLOWED_NODES="[pv1-1]" \
# VM_SSH_KEYS="$(cat ~/.ssh/id_ed25519.pub)" \
# CONTROL_PLANE_ENDPOINT_IP=192.168.4.30 NODE_IP_RANGES="[192.168.4.31-192.168.4.39]" \
# GATEWAY=192.168.4.8 IP_PREFIX=24 DNS_SERVERS="[8.8.8.8, 1.1.1.1]" BRIDGE=vmbr0 \
# BOOT_VOLUME_DEVICE=scsi0 BOOT_VOLUME_SIZE=40 NUM_SOCKETS=1 NUM_CORES=2 MEMORY_MIB=4096 \
# clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --kubernetes-version v1.35.9 \
# --control-plane-machine-count 1 --worker-machine-count 1 --target-namespace default
#
# Hand edits (docs/phase-1-proxmox-capi.md Step 3):
# 1. pod CIDR 192.168.0.0/16 -> 10.244.0.0/16 (overlaps the lab LAN)
# 2. kube-vip v0.7.1 -> v1.2.4 (kube-vip-prepare.sh kept)
# 3. `format: qcow2` removed from both ProxmoxMachineTemplates (local-lvm is LVM-thin, raw only)
# 4. `vlan: 4` on net0 of both ProxmoxMachineTemplates
# 5. worker template: 1 socket x 4 cores, 2048 MiB, named lab1-worker-2g (owner's choice, decision #16;
# first 4096 MiB as lab1-worker). Templates are never edited in place: a new size = a new
# template name + MD infrastructureRef, which rolls the workers.
# 6. MachineDeployment lab1-workers: `replicas` removed (the autoscaler owns it)
# 7. MachineDeployment lab1-workers: autoscaler min/max + capacity annotations (runbook Step 5)
# 8. MachineDeployment lab1-workers: rollout maxSurge 0 / maxUnavailable 1 (memory-tight host)
# 9. ProxmoxCluster lab1: schedulerHints.memoryAdjustment 160 (CAPMOX memory overcommit, decision #17)
# 10. Second worker group on PVE node pv1-2 (decision #18): MachineDeployment lab1-workers-pv1-2 +
# ProxmoxMachineTemplate lab1-worker-2g-pv1-2 (template copy 9101, allowedNodes [pv1-2]).
# local-lvm is node-local, so every node needs its own template VMID and its own MD.
# 11. ProxmoxCluster lab1: node IP pool .31-.39 -> .31-.50 (20 addresses). Max sizes 6 (pv1-1) +
# 12 (pv1-2) + 1 CP = 19; one address stays free for the KCP rollout surge (CAPMOX pools also
# hold a freed address for 60 s before reuse).
# 12. Both worker MDs: label node.cluster.x-k8s.io/pool=workers (template + capacity labels
# annotation) for CA --balancing-label; capacity key maximum-pods -> maxPods.
# 13. Symmetric per-node names (owner's request): MachineDeployment lab1-workers -> lab1-workers-pv1-1,
# ProxmoxMachineTemplate lab1-worker-2g -> lab1-worker-2g-pv1-1 (+ explicit allowedNodes [pv1-1]).
# A rename is a new object: the old MD and PMT are deleted by hand once the old MD is at 0.
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
labels:
cluster.x-k8s.io/cluster-name: lab1
name: lab1
namespace: default
spec:
clusterNetwork:
pods:
cidrBlocks:
- 10.244.0.0/16
controlPlaneRef:
apiGroup: controlplane.cluster.x-k8s.io
kind: KubeadmControlPlane
name: lab1-control-plane
infrastructureRef:
apiGroup: infrastructure.cluster.x-k8s.io
kind: ProxmoxCluster
name: lab1
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: ProxmoxCluster
metadata:
labels:
cluster.x-k8s.io/cluster-name: lab1
name: lab1
namespace: default
spec:
allowedNodes:
- pv1-1
controlPlaneEndpoint:
host: 192.168.4.30
port: 6443
dnsServers:
- 8.8.8.8
- 1.1.1.1
ipv4Config:
addresses:
- 192.168.4.31-192.168.4.50
gateway: 192.168.4.8
prefix: 24
# CAPMOX reserves the configured memory of every non-template VM, stopped ones included, against
# host total × memoryAdjustment%. 160% of 29.16 GiB = 46.7 GiB (~17.7 GiB headroom today).
# Real overcommit protection on pv1-1: KSM (ksmtuned) + 8 GiB swap.
schedulerHints:
memoryAdjustment: 160
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlane
metadata:
name: lab1-control-plane
namespace: default
spec:
kubeadmConfigSpec:
files:
- content: |
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
name: kube-vip
namespace: kube-system
spec:
containers:
- args:
- manager
env:
- name: cp_enable
value: "true"
- name: vip_interface
value: ""
- name: address
value: 192.168.4.30
- name: port
value: "6443"
- name: vip_arp
value: "true"
- name: vip_leaderelection
value: "true"
- name: vip_leaseduration
value: "15"
- name: vip_renewdeadline
value: "10"
- name: vip_retryperiod
value: "2"
image: ghcr.io/kube-vip/kube-vip:v1.2.4
imagePullPolicy: IfNotPresent
name: kube-vip
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
- NET_RAW
volumeMounts:
- mountPath: /etc/kubernetes/admin.conf
name: kubeconfig
hostAliases:
- hostnames:
- localhost
- kubernetes
ip: 127.0.0.1
hostNetwork: true
volumes:
- hostPath:
path: /etc/kubernetes/admin.conf
type: FileOrCreate
name: kubeconfig
status: {}
owner: root:root
path: /etc/kubernetes/manifests/kube-vip.yaml
- content: |
#!/bin/bash
# Copyright 2020 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -e
# Configure the workaround required for kubeadm init with kube-vip:
# xref: https://github.com/kube-vip/kube-vip/issues/684
# Nothing to do for kubernetes < v1.29
KUBEADM_MINOR="$(kubeadm version -o short | cut -d '.' -f 2)"
if [[ "$KUBEADM_MINOR" -lt "29" ]]; then
exit 0
fi
IS_KUBEADM_INIT="false"
# cloud-init kubeadm init
if [[ -f /run/kubeadm/kubeadm.yaml ]]; then
IS_KUBEADM_INIT="true"
fi
# ignition kubeadm init
if [[ -f /etc/kubeadm.sh ]] && grep -q -e "kubeadm init" /etc/kubeadm.sh; then
IS_KUBEADM_INIT="true"
fi
if [[ "$IS_KUBEADM_INIT" == "true" ]]; then
sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' \
/etc/kubernetes/manifests/kube-vip.yaml
fi
owner: root:root
path: /etc/kube-vip-prepare.sh
permissions: "0700"
initConfiguration:
nodeRegistration:
kubeletExtraArgs:
- name: provider-id
value: proxmox://'{{ ds.meta_data.instance_id }}'
joinConfiguration:
nodeRegistration:
kubeletExtraArgs:
- name: provider-id
value: proxmox://'{{ ds.meta_data.instance_id }}'
preKubeadmCommands:
- /etc/kube-vip-prepare.sh
users:
- name: root
sshAuthorizedKeys:
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINQxxkqmvtVI+8c5BkTaJ5c2HfBFRXJWMmEcevvfP9tV
jan.novak@Jans-MacBook-Air.local
machineTemplate:
spec:
infrastructureRef:
apiGroup: infrastructure.cluster.x-k8s.io
kind: ProxmoxMachineTemplate
name: lab1-control-plane
replicas: 1
version: v1.35.9
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: ProxmoxMachineTemplate
metadata:
name: lab1-control-plane
namespace: default
spec:
template:
spec:
disks:
bootVolume:
disk: scsi0
sizeGb: 40
full: true
memoryMiB: 4096
network:
networkDevices:
- bridge: vmbr0
model: virtio
name: net0
vlan: 4
numCores: 2
numSockets: 1
sourceNode: pv1-1
templateID: 101
---
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
annotations:
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "6"
# scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g-pv1-1
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets 1 × numCores 4
capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" # bootVolume sizeGb 40
capacity.cluster-autoscaler.kubernetes.io/maxPods: "110" # CA key; "maximum-pods" is ignored
# Labels for the from-zero template node. CA 1.35.2 reads MD labels from a path v1beta2 doesn't
# have, and CAPMOX publishes no nodeInfo, so without this os/arch are "" on the template.
capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
labels:
cluster.x-k8s.io/cluster-name: lab1
name: lab1-workers-pv1-1
namespace: default
spec:
clusterName: lab1
# Delete an old worker before creating its replacement. The default surge (1/0) deadlocks on a full
# host: CAPMOX can't reserve memory for the surge VM, and CA won't scale down the old workers while
# the MD is rolling out (clusterapi ScaleDownNodeUpgradeProcessor).
rollout:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
selector:
matchLabels: null
template:
metadata:
labels:
cluster.x-k8s.io/cluster-name: lab1
node-role.kubernetes.io/node: ""
# CA --balancing-label; node.cluster.x-k8s.io/* is synced to the Node by CAPI
node.cluster.x-k8s.io/pool: workers
spec:
bootstrap:
configRef:
apiGroup: bootstrap.cluster.x-k8s.io
kind: KubeadmConfigTemplate
name: lab1-worker
clusterName: lab1
infrastructureRef:
apiGroup: infrastructure.cluster.x-k8s.io
kind: ProxmoxMachineTemplate
name: lab1-worker-2g-pv1-1
version: v1.35.9
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: ProxmoxMachineTemplate
metadata:
name: lab1-worker-2g-pv1-1
namespace: default
spec:
template:
spec:
# Pinned like the pv1-2 template, independent of ProxmoxCluster.allowedNodes: template 101
# lives on pv1-1's local-lvm.
allowedNodes:
- pv1-1
disks:
bootVolume:
disk: scsi0
sizeGb: 40
full: true
memoryMiB: 2048
network:
networkDevices:
- bridge: vmbr0
model: virtio
name: net0
vlan: 4
numCores: 4
numSockets: 1
sourceNode: pv1-1
templateID: 101
---
# Worker group on pv1-2. Keep in sync with lab1-workers-pv1-1 / lab1-worker-2g-pv1-1 except node,
# template and max-size: CA --balancing-label skips resource checks, so capacity must stay identical.
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
annotations:
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12"
# scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g-pv1-2
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets 1 × numCores 4
capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" # bootVolume sizeGb 40
capacity.cluster-autoscaler.kubernetes.io/maxPods: "110" # CA key; "maximum-pods" is ignored
# Labels for the from-zero template node. CA 1.35.2 reads MD labels from a path v1beta2 doesn't
# have, and CAPMOX publishes no nodeInfo, so without this os/arch are "" on the template.
capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
labels:
cluster.x-k8s.io/cluster-name: lab1
name: lab1-workers-pv1-2
namespace: default
spec:
clusterName: lab1
rollout:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
selector:
matchLabels: null
template:
metadata:
labels:
cluster.x-k8s.io/cluster-name: lab1
node-role.kubernetes.io/node: ""
# CA --balancing-label; node.cluster.x-k8s.io/* is synced to the Node by CAPI
node.cluster.x-k8s.io/pool: workers
spec:
bootstrap:
configRef:
apiGroup: bootstrap.cluster.x-k8s.io
kind: KubeadmConfigTemplate
name: lab1-worker
clusterName: lab1
infrastructureRef:
apiGroup: infrastructure.cluster.x-k8s.io
kind: ProxmoxMachineTemplate
name: lab1-worker-2g-pv1-2
version: v1.35.9
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: ProxmoxMachineTemplate
metadata:
name: lab1-worker-2g-pv1-2
namespace: default
spec:
template:
spec:
# Overrides ProxmoxCluster.allowedNodes ([pv1-1], kept for the control plane): template 9101
# lives on pv1-2's local-lvm, and PVE refuses cross-node clones from local storage.
allowedNodes:
- pv1-2
disks:
bootVolume:
disk: scsi0
sizeGb: 40
full: true
memoryMiB: 2048
network:
networkDevices:
- bridge: vmbr0
model: virtio
name: net0
vlan: 4
numCores: 4
numSockets: 1
sourceNode: pv1-2
templateID: 9101
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
metadata:
name: lab1-worker
namespace: default
spec:
template:
spec:
joinConfiguration:
nodeRegistration:
kubeletExtraArgs:
- name: provider-id
value: proxmox://'{{ ds.meta_data.instance_id }}'
users:
- name: root
sshAuthorizedKeys:
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINQxxkqmvtVI+8c5BkTaJ5c2HfBFRXJWMmEcevvfP9tV
jan.novak@Jans-MacBook-Air.local
Reference in New Issue View Git Blame Copy Permalink