kacerrandClaude Opus 5.5
6ef55778ff
PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage: - template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50; lab1-workers renamed for symmetry (decision #18, #7 superseded) - CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19). Scale test: 16 pods -> 4 + 4 across both nodes. - runbook, image-builder copy procedure, fixes tables, future task to report three CA clusterapi bugs upstream, lab facts, changelog, execution log Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
124 lines
6.4 KiB
Markdown
124 lines
6.4 KiB
Markdown
# cluster-autoscaler-with-capi
|
|
|
|
Node autoscaling for Kubernetes clusters running as VMs on a home-lab hypervisor fleet
|
|
(Proxmox VE + vanilla KVM/libvirt), driven by **Cluster API** and **cluster-autoscaler** —
|
|
growing, in later phases, into a "poor man's Kubernetes-as-a-Service".
|
|
|
|
> Status: **Phase 1 in progress** — VM template built and smoke-tested; management cluster
|
|
> running (k3s + CAPI via cluster-api-operator, built by [ansible/mgmt.yml](ansible/mgmt.yml));
|
|
> first workload cluster `lab1` next. Versions verified 2026-09-24/26/27, see [docs/versions.md](docs/versions.md).
|
|
|
|
## Goals
|
|
|
|
- Workload clusters whose worker pool scales **0 → N → 0** on demand, with VMs created and
|
|
destroyed on the hypervisors automatically.
|
|
- Everything declarative: clusters are CAPI objects in a management cluster; no hand-made VMs.
|
|
- Learn Cluster API internals (contracts, providers, reconcile flow) by assembling, then extending it.
|
|
- End state: self-service tenant clusters (hosted control planes + autoscaled VM workers).
|
|
|
|
## Non-goals (for now)
|
|
|
|
- HA management cluster, production SLAs, multi-site.
|
|
- GitOps / fleet management (may come later on top).
|
|
- Writing our own autoscaler — cluster-autoscaler's `clusterapi` provider does the job.
|
|
|
|
## Lab inventory
|
|
|
|
| Hypervisor type | Management API | CAPI infrastructure provider |
|
|
| --- | --- | --- |
|
|
| Proxmox VE cluster `homelab`: `pv1-1` 192.168.0.8, `pv1-2` 192.168.0.10 | PVE REST API :8006 | CAPMOX (phase 1) |
|
|
| Ubuntu 24.04 + KVM/libvirt | libvirt RPC (`qemu+ssh`/`qemu+tls`) | none mature upstream → phase 2 |
|
|
|
|
VM network: `vmbr0` (VLAN-aware), VLAN 4, build host 192.168.4.181.
|
|
|
|
## Architecture (phase 1)
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph MGMT[mgmt cluster: k3s VM]
|
|
CAPI[CAPI core + kubeadm bootstrap/CP]
|
|
CAPMOX[CAPMOX]
|
|
IPAM[IPAM in-cluster]
|
|
CA[cluster-autoscaler]
|
|
end
|
|
PVE[Proxmox VE API]
|
|
subgraph LAB1[workload cluster lab1]
|
|
CP[control plane VM + kube-vip VIP]
|
|
W[worker VMs - MachineDeployment 0..N]
|
|
end
|
|
CA -- scales MachineDeployment --> CAPI
|
|
CAPI --> CAPMOX
|
|
CAPMOX -- clone template, cloud-init ISO --> PVE
|
|
PVE --> CP
|
|
PVE --> W
|
|
CA -- watches pending pods via lab1-kubeconfig --> LAB1
|
|
```
|
|
|
|
cluster-autoscaler runs in the management cluster in `kubeconfig-incluster` mode: it reads
|
|
pending pods/nodes from the workload cluster through the CAPI-generated kubeconfig secret and
|
|
changes `replicas` on MachineDeployments via the in-cluster management API.
|
|
|
|
## Roadmap
|
|
|
|
| Phase | Scope | Exit criteria | Doc |
|
|
| --- | --- | --- | --- |
|
|
| **1** | k3s mgmt cluster (manual) → CAPI + CAPMOX → workload cluster on Proxmox VMs → cluster-autoscaler | scale-up 1→3, scale-down to 0, scale-from-zero all work | [phase-1-proxmox-capi.md](docs/phase-1-proxmox-capi.md) |
|
|
| **2** | CAPI provider for vanilla KVM/libvirt hosts (evaluate CAPLV / KubeVirt / Metal3 / own provider) | worker MachineDeployment on Ubuntu KVM hosts, autoscaled | [phase-2-kvm-provider.md](docs/phase-2-kvm-provider.md) |
|
|
| **3** | Hosted control planes with Kamaji, tenant clusters on demand ("poor man's KaaS") | new tenant cluster from one manifest/ClusterClass, workers autoscaled | [phase-3-kamaji-kaas.md](docs/phase-3-kamaji-kaas.md) |
|
|
|
|
Phases 2 and 3 are independent and can be done in either order.
|
|
|
|
## Repository layout
|
|
|
|
```
|
|
.
|
|
├── README.md # this file
|
|
├── CLAUDE.md # context + rules for Claude Code sessions
|
|
├── CHANGELOG.md # milestones and significant changes, newest first
|
|
├── ansible/ # mgmt.yml: management VM + k3s + cert-manager + CAPI operator/providers
|
|
├── .claude/
|
|
│ ├── agents/ # upstream-researcher, cluster-inspector (read-only)
|
|
│ ├── commands/ # /ship
|
|
│ └── cluster-inspector/ # RBAC, kubeconfig minting script, gitignored kubeconfigs/
|
|
├── docs/
|
|
│ ├── decisions.md # decision log: alternatives, why, status
|
|
│ ├── future-tasks.md # deferred work with context, newest first
|
|
│ ├── plans/ # plans (plan mode), one file per plan
|
|
│ ├── plans-executions/ # per-plan execution logs with exact commands
|
|
│ ├── sessions/ # research/design conversation summaries
|
|
│ ├── prompts/ # prompts/specs that kicked off work
|
|
│ ├── reviews/ # review findings
|
|
│ ├── versions.md # pinned versions + compatibility reasoning
|
|
│ ├── phase-1-proxmox-capi.md # runbook for phase 1
|
|
│ ├── image-builder-proxmox.md # building the node VM template + fixes log
|
|
│ ├── phase-2-kvm-provider.md # KVM/libvirt provider options
|
|
│ └── phase-3-kamaji-kaas.md # hosted control planes / KaaS design notes
|
|
└── configs/
|
|
├── image-builder/ # packer var file for the Proxmox template
|
|
├── capi/ # clusterctl config example
|
|
├── cluster-autoscaler/ # Helm values
|
|
└── test/ # workloads for scale tests
|
|
```
|
|
|
|
Secrets (PVE token secret, kubeconfigs) never go into the repo — see `.gitignore`.
|
|
|
|
## Key decisions so far
|
|
|
|
Short summary; the full log with alternatives and status is [docs/decisions.md](docs/decisions.md).
|
|
|
|
1. **CAPI v1.12.x, not the latest 1.14.x** — CAPMOX v0.9.x is only tested against CAPI 1.11/1.12.
|
|
This caps workload Kubernetes at **1.35** and cluster-autoscaler at **1.35.x**.
|
|
2. **Management cluster = k3s on a Proxmox VM**, not kind — CAPI controllers and the autoscaler
|
|
must run permanently.
|
|
3. **Node images via kubernetes-sigs/image-builder** (Ubuntu 24.04, Proxmox builder), overrides in
|
|
a `PACKER_VAR_FILES` JSON instead of `PACKER_FLAGS`.
|
|
4. **CNI installed by Helm** (Cilium, `ipam.mode=kubernetes`), not via ClusterResourceSet flavors,
|
|
to keep its version under our control.
|
|
5. **Scale-from-zero via capacity annotations** on the MachineDeployment — CAPMOX does not publish
|
|
`status.capacity` on `ProxmoxMachineTemplate`.
|
|
6. **Management cluster from one idempotent Ansible playbook**, CAPI providers installed by
|
|
**cluster-api-operator** instead of `clusterctl init` — re-runs are no-ops, upgrades are version bumps.
|
|
7. **One template copy and one worker MachineDeployment per PVE node** (no shared storage):
|
|
`local-lvm` templates can't be cloned across nodes. cluster-autoscaler balances the groups on a
|
|
shared `node.cluster.x-k8s.io/pool` label.
|