kacerrandClaude Opus 5.5
6ef55778ff
PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage: - template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50; lab1-workers renamed for symmetry (decision #18, #7 superseded) - CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19). Scale test: 16 pods -> 4 + 4 across both nodes. - runbook, image-builder copy procedure, fixes tables, future task to report three CA clusterapi bugs upstream, lab facts, changelog, execution log Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
43 lines
2.6 KiB
YAML
43 lines
2.6 KiB
YAML
# helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler --version 9.59.0 -n default -f values.yaml
|
|
# must be installed in the same namespace as the <cluster>-kubeconfig secret
|
|
cloudProvider: clusterapi
|
|
clusterAPIMode: kubeconfig-incluster # workload via kubeconfig, management in-cluster
|
|
clusterAPIKubeconfigSecret: lab1-kubeconfig
|
|
autoDiscovery:
|
|
clusterName: lab1
|
|
namespace: default
|
|
image:
|
|
tag: v1.35.2 # chart default appVersion is 1.35.0
|
|
rbac:
|
|
# CA reads the infrastructure machine template (capacity for scale-from-zero). The chart's
|
|
# ClusterRole only covers cluster.x-k8s.io; without this the first main loop blocks forever on
|
|
# "waiting for cache sync on infrastructure resource" and CA never scales.
|
|
additionalRules:
|
|
- apiGroups: [infrastructure.cluster.x-k8s.io]
|
|
resources: [proxmoxmachinetemplates]
|
|
verbs: [get, list, watch]
|
|
extraArgs:
|
|
expander: least-waste
|
|
# Aggressive lab scale-down profile (CA 1.35 defaults in brackets): idle worker gone in ~3-4 min
|
|
# (scan-interval 1m adds up to a minute to the 1m unneeded time).
|
|
# Keep delay-after-add >= node startup (~2-3 min here) or bursts flap up/down.
|
|
scale-down-unneeded-time: 1m # [10m]
|
|
scale-down-delay-after-add: 2m # [10m] cluster-wide cooldown after any scale-up
|
|
scale-down-delay-after-failure: 1m # [3m]
|
|
unremovable-node-recheck-timeout: 1m # [5m]
|
|
scale-down-utilization-threshold: "0.6" # [0.5]
|
|
max-graceful-termination-sec: "60" # [600] drain wait per pod
|
|
balance-similar-node-groups: true
|
|
# One worker MD per PVE node. Compare groups only on this label: the generic comparator fails when
|
|
# one group has a real-node template (cached, 2Gi VM = ~1.78Gi allocatable, os/arch set) and the
|
|
# other a from-zero one. Disables the resource checks, so keep the MDs' capacity identical.
|
|
balancing-label: node.cluster.x-k8s.io/pool
|
|
# Fewer API requests instead of a higher kube-client-qps: every MD at 0 replicas costs ~3 uncached
|
|
# discovery GETs per NodeGroups() call (~15x per loop). Unschedulable pods and scale-ups/downs still
|
|
# trigger a loop immediately; only idle loops (scale-down evaluation) are spaced out.
|
|
scan-interval: 1m # [10s]
|
|
# Cilium taints new nodes NoSchedule until its agent is up. Without this, CA simulates the freshly
|
|
# registered (still tainted) node, finds the pending pods don't fit and adds one node too many.
|
|
startup-taint: node.cilium.io/agent-not-ready
|
|
v: 2 # [chart 4] 4 logs every throttled request and MD lookup
|