cluster-autoscaler-with-capi

Node autoscaling for Kubernetes clusters running as VMs on a home-lab hypervisor fleet (Proxmox VE + vanilla KVM/libvirt), driven by Cluster API and cluster-autoscaler — growing, in later phases, into a "poor man's Kubernetes-as-a-Service".

Status: Phase 1 in progress — VM template built and smoke-tested; management cluster running (k3s + CAPI via cluster-api-operator, built by ansible/mgmt.yml); first workload cluster lab1 next. Versions verified 2026-09-24/26/27, see docs/versions.md.

Goals

• Workload clusters whose worker pool scales 0 → N → 0 on demand, with VMs created and destroyed on the hypervisors automatically.
• Everything declarative: clusters are CAPI objects in a management cluster; no hand-made VMs.
• Learn Cluster API internals (contracts, providers, reconcile flow) by assembling, then extending it.
• End state: self-service tenant clusters (hosted control planes + autoscaled VM workers).

Non-goals (for now)

• HA management cluster, production SLAs, multi-site.
• GitOps / fleet management (may come later on top).
• Writing our own autoscaler — cluster-autoscaler's clusterapi provider does the job.

Lab inventory

Hypervisor type	Management API	CAPI infrastructure provider
Proxmox VE (e.g. pv1-1, 192.168.0.8)	PVE REST API :8006	CAPMOX (phase 1)
Ubuntu 24.04 + KVM/libvirt	libvirt RPC (qemu+ssh/qemu+tls)	none mature upstream → phase 2

VM network: vmbr0 (VLAN-aware), VLAN 4, build host 192.168.4.181.

Architecture (phase 1)

flowchart LR
  subgraph MGMT[mgmt cluster: k3s VM]
    CAPI[CAPI core + kubeadm bootstrap/CP]
    CAPMOX[CAPMOX]
    IPAM[IPAM in-cluster]
    CA[cluster-autoscaler]
  end
  PVE[Proxmox VE API]
  subgraph LAB1[workload cluster lab1]
    CP[control plane VM + kube-vip VIP]
    W[worker VMs - MachineDeployment 0..N]
  end
  CA -- scales MachineDeployment --> CAPI
  CAPI --> CAPMOX
  CAPMOX -- clone template, cloud-init ISO --> PVE
  PVE --> CP
  PVE --> W
  CA -- watches pending pods via lab1-kubeconfig --> LAB1

cluster-autoscaler runs in the management cluster in kubeconfig-incluster mode: it reads pending pods/nodes from the workload cluster through the CAPI-generated kubeconfig secret and changes replicas on MachineDeployments via the in-cluster management API.

Roadmap

Phase	Scope	Exit criteria	Doc
1	k3s mgmt cluster (manual) → CAPI + CAPMOX → workload cluster on Proxmox VMs → cluster-autoscaler	scale-up 1→3, scale-down to 0, scale-from-zero all work	phase-1-proxmox-capi.md
2	CAPI provider for vanilla KVM/libvirt hosts (evaluate CAPLV / KubeVirt / Metal3 / own provider)	worker MachineDeployment on Ubuntu KVM hosts, autoscaled	phase-2-kvm-provider.md
3	Hosted control planes with Kamaji, tenant clusters on demand ("poor man's KaaS")	new tenant cluster from one manifest/ClusterClass, workers autoscaled	phase-3-kamaji-kaas.md

Phases 2 and 3 are independent and can be done in either order.

Repository layout

.
├── README.md                     # this file
├── CLAUDE.md                     # context + rules for Claude Code sessions
├── CHANGELOG.md                  # milestones and significant changes, newest first
├── ansible/                      # mgmt.yml: management VM + k3s + cert-manager + CAPI operator/providers
├── .claude/
│   ├── agents/                   # upstream-researcher, cluster-inspector (read-only)
│   ├── commands/                 # /ship
│   └── cluster-inspector/        # RBAC, kubeconfig minting script, gitignored kubeconfigs/
├── docs/
│   ├── decisions.md              # decision log: alternatives, why, status
│   ├── plans/                    # plans (plan mode), one file per plan
│   ├── plans-executions/         # per-plan execution logs with exact commands
│   ├── sessions/                 # research/design conversation summaries
│   ├── prompts/                  # prompts/specs that kicked off work
│   ├── reviews/                  # review findings
│   ├── versions.md               # pinned versions + compatibility reasoning
│   ├── phase-1-proxmox-capi.md   # runbook for phase 1
│   ├── image-builder-proxmox.md  # building the node VM template + fixes log
│   ├── phase-2-kvm-provider.md   # KVM/libvirt provider options
│   └── phase-3-kamaji-kaas.md    # hosted control planes / KaaS design notes
└── configs/
    ├── image-builder/            # packer var file for the Proxmox template
    ├── capi/                     # clusterctl config example
    ├── cluster-autoscaler/       # Helm values
    └── test/                     # workloads for scale tests

Secrets (PVE token secret, kubeconfigs) never go into the repo — see .gitignore.

Key decisions so far

Short summary; the full log with alternatives and status is docs/decisions.md.

1. CAPI v1.12.x, not the latest 1.14.x — CAPMOX v0.9.x is only tested against CAPI 1.11/1.12. This caps workload Kubernetes at 1.35 and cluster-autoscaler at 1.35.x.
2. Management cluster = k3s on a Proxmox VM, not kind — CAPI controllers and the autoscaler must run permanently.
3. Node images via kubernetes-sigs/image-builder (Ubuntu 24.04, Proxmox builder), overrides in a PACKER_VAR_FILES JSON instead of PACKER_FLAGS.
4. CNI installed by Helm (Cilium, ipam.mode=kubernetes), not via ClusterResourceSet flavors, to keep its version under our control.
5. Scale-from-zero via capacity annotations on the MachineDeployment — CAPMOX does not publish status.capacity on ProxmoxMachineTemplate.
6. Management cluster from one idempotent Ansible playbook, CAPI providers installed by cluster-api-operator instead of clusterctl init — re-runs are no-ops, upgrades are version bumps.