Initial import: phase docs, configs, Claude conventions

Phase 1-3 runbooks and design notes, version pins, config templates, project-history conventions, cluster-inspector read-only credentials setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 00:58:52 +02:00
commit 5fd998c5f9
26 changed files with 1124 additions and 0 deletions
@@ -0,0 +1,115 @@
+# cluster-autoscaler-with-capi
+
+Node autoscaling for Kubernetes clusters running as VMs on a home-lab hypervisor fleet
+(Proxmox VE + vanilla KVM/libvirt), driven by **Cluster API** and **cluster-autoscaler** —
+growing, in later phases, into a "poor man's Kubernetes-as-a-Service".
+
+> Status: **Phase 1 in progress** — VM template built and smoke-tested; management cluster
+> and first workload cluster next. Versions verified 2026-09-24/26, see [docs/versions.md](docs/versions.md).
+
+## Goals
+
+- Workload clusters whose worker pool scales **0 → N → 0** on demand, with VMs created and
+  destroyed on the hypervisors automatically.
+- Everything declarative: clusters are CAPI objects in a management cluster; no hand-made VMs.
+- Learn Cluster API internals (contracts, providers, reconcile flow) by assembling, then extending it.
+- End state: self-service tenant clusters (hosted control planes + autoscaled VM workers).
+
+## Non-goals (for now)
+
+- HA management cluster, production SLAs, multi-site.
+- GitOps / fleet management (may come later on top).
+- Writing our own autoscaler — cluster-autoscaler's `clusterapi` provider does the job.
+
+## Lab inventory
+
+| Hypervisor type | Management API | CAPI infrastructure provider |
+| --- | --- | --- |
+| Proxmox VE (e.g. `pv1-1`, 192.168.0.8) | PVE REST API :8006 | CAPMOX (phase 1) |
+| Ubuntu 24.04 + KVM/libvirt | libvirt RPC (`qemu+ssh`/`qemu+tls`) | none mature upstream → phase 2 |
+
+VM network: `vmbr0` (VLAN-aware), VLAN 4, build host 192.168.4.181.
+
+## Architecture (phase 1)
+
+```mermaid
+flowchart LR
+  subgraph MGMT[mgmt cluster: k3s VM]
+    CAPI[CAPI core + kubeadm bootstrap/CP]
+    CAPMOX[CAPMOX]
+    IPAM[IPAM in-cluster]
+    CA[cluster-autoscaler]
+  end
+  PVE[Proxmox VE API]
+  subgraph LAB1[workload cluster lab1]
+    CP[control plane VM + kube-vip VIP]
+    W[worker VMs - MachineDeployment 0..N]
+  end
+  CA -- scales MachineDeployment --> CAPI
+  CAPI --> CAPMOX
+  CAPMOX -- clone template, cloud-init ISO --> PVE
+  PVE --> CP
+  PVE --> W
+  CA -- watches pending pods via lab1-kubeconfig --> LAB1
+```
+
+cluster-autoscaler runs in the management cluster in `kubeconfig-incluster` mode: it reads
+pending pods/nodes from the workload cluster through the CAPI-generated kubeconfig secret and
+changes `replicas` on MachineDeployments via the in-cluster management API.
+
+## Roadmap
+
+| Phase | Scope | Exit criteria | Doc |
+| --- | --- | --- | --- |
+| **1** | k3s mgmt cluster (manual) → CAPI + CAPMOX → workload cluster on Proxmox VMs → cluster-autoscaler | scale-up 1→3, scale-down to 0, scale-from-zero all work | [phase-1-proxmox-capi.md](docs/phase-1-proxmox-capi.md) |
+| **2** | CAPI provider for vanilla KVM/libvirt hosts (evaluate CAPLV / KubeVirt / Metal3 / own provider) | worker MachineDeployment on Ubuntu KVM hosts, autoscaled | [phase-2-kvm-provider.md](docs/phase-2-kvm-provider.md) |
+| **3** | Hosted control planes with Kamaji, tenant clusters on demand ("poor man's KaaS") | new tenant cluster from one manifest/ClusterClass, workers autoscaled | [phase-3-kamaji-kaas.md](docs/phase-3-kamaji-kaas.md) |
+
+Phases 2 and 3 are independent and can be done in either order.
+
+## Repository layout
+
+```
+.
+├── README.md                     # this file
+├── CLAUDE.md                     # context + rules for Claude Code sessions
+├── CHANGELOG.md                  # milestones and significant changes, newest first
+├── .claude/
+│   ├── agents/                   # upstream-researcher, cluster-inspector (read-only)
+│   ├── commands/                 # /ship
+│   └── cluster-inspector/        # RBAC, kubeconfig minting script, gitignored kubeconfigs/
+├── docs/
+│   ├── decisions.md              # decision log: alternatives, why, status
+│   ├── plans/                    # plans (plan mode), one file per plan
+│   ├── plans-executions/         # per-plan execution logs with exact commands
+│   ├── sessions/                 # research/design conversation summaries
+│   ├── prompts/                  # prompts/specs that kicked off work
+│   ├── reviews/                  # review findings
+│   ├── versions.md               # pinned versions + compatibility reasoning
+│   ├── phase-1-proxmox-capi.md   # runbook for phase 1
+│   ├── image-builder-proxmox.md  # building the node VM template + fixes log
+│   ├── phase-2-kvm-provider.md   # KVM/libvirt provider options
+│   └── phase-3-kamaji-kaas.md    # hosted control planes / KaaS design notes
+└── configs/
+    ├── image-builder/            # packer var file for the Proxmox template
+    ├── capi/                     # clusterctl config example
+    ├── cluster-autoscaler/       # Helm values
+    └── test/                     # workloads for scale tests
+```
+
+Secrets (PVE token secret, kubeconfigs) never go into the repo — see `.gitignore`.
+
+## Key decisions so far
+
+Short summary; the full log with alternatives and status is [docs/decisions.md](docs/decisions.md).
+
+1. **CAPI v1.12.x, not the latest 1.14.x** — CAPMOX v0.9.x is only tested against CAPI 1.11/1.12.
+   This caps workload Kubernetes at **1.35** and cluster-autoscaler at **1.35.x**.
+2. **Management cluster = k3s on a Proxmox VM**, not kind — CAPI controllers and the autoscaler
+   must run permanently.
+3. **Node images via kubernetes-sigs/image-builder** (Ubuntu 24.04, Proxmox builder), overrides in
+   a `PACKER_VAR_FILES` JSON instead of `PACKER_FLAGS`.
+4. **CNI installed by Helm** (Cilium, `ipam.mode=kubernetes`), not via ClusterResourceSet flavors,
+   to keep its version under our control.
+5. **Scale-from-zero via capacity annotations** on the MachineDeployment — CAPMOX does not publish
+   `status.capacity` on `ProxmoxMachineTemplate`.