cluster-autoscaler-with-capi/docs/phase-3-kamaji-kaas.md

Phase 3 — Hosted control planes with Kamaji ("poor man's KaaS")

Goal: create a tenant Kubernetes cluster from one manifest. Its control plane runs as pods in the management cluster (Kamaji), and its workers are autoscaled Proxmox (later also KVM) VMs. Status: design notes.

Why Kamaji

With kubeadm control planes every tenant costs ≥ 1 VM (3 for HA) just for apiserver/etcd. Kamaji runs each tenant control plane as a Deployment in the management cluster, backed by a shared datastore. That cuts the control plane to a few hundred MB of RAM, gives it seconds to start, and makes upgrades a rolling Deployment update.

Architecture

flowchart LR
  subgraph MGMT[management cluster - 3 nodes]
    CAPI[CAPI core + CAPMOX]
    KCP[Kamaji CP provider]
    KAM[Kamaji controller]
    DS[(kamaji-etcd datastore)]
    LB[MetalLB pool on VLAN 4]
    subgraph T1[ns tenant-a]
      TCP1[TenantControlPlane pods<br/>apiserver, ctrl-mgr, scheduler]
      CA1[cluster-autoscaler tenant-a]
    end
  end
  PVE[Proxmox VE]
  W1[tenant-a worker VMs]
  KCP --> KAM --> TCP1
  TCP1 --> DS
  LB -. VIP :6443 .- TCP1
  CAPI -- MachineDeployment --> PVE --> W1
  W1 -- kubelet join via VIP --> TCP1
  CA1 -- scales MD --> CAPI

CAPI object wiring per tenant:

• Cluster.spec.controlPlaneRef → KamajiControlPlane, which creates the Kamaji TenantControlPlane.
• Cluster.spec.infrastructureRef → ProxmoxCluster with externalManagedControlPlane: true, so the Kamaji provider patches the control plane endpoint. The field exists in CAPMOX v1alpha2.
• Workers: a regular MachineDeployment + KubeadmConfigTemplate + ProxmoxMachineTemplate, with autoscaler annotations as in phase 1.
• Konnectivity handles apiserver → kubelet traffic, because the control plane isn't on the worker network.

References: Kamaji + Proxmox infra provider docs, CLASTIX Helm chart capi-kamaji-proxmox (0.0.4) as a wiring example.

Management cluster changes vs. phase 1

Area	Phase 1	Phase 3
Nodes	1× k3s	3 nodes: tenant control planes and their datastore live here
LoadBalancer	none needed (kube-vip in workload CP)	MetalLB (or kube-vip cloud provider) with an IP pool on VLAN 4, one VIP per tenant
Datastore	—	kamaji-etcd chart 0.18.1 (etcd 3.6.12); alternatives: PostgreSQL/MySQL via kine
Storage	—	CSI for etcd PVCs

k3s' built-in servicelb (klipper) binds host ports, so it can't hand out many :6443 endpoints. Use MetalLB.

Autoscaling per tenant

A cluster-autoscaler instance in clusterapi mode serves one workload cluster (one kubeconfig). So each tenant gets its own CA release in its namespace in the management cluster (kubeconfig-incluster, secret <tenant>-kubeconfig). Automating that is our first real Go code:

• tenant-operator (small controller in this repo): watches Cluster objects labelled kaas.lab/autoscaling=enabled, renders a per-tenant cluster-autoscaler Deployment + RBAC, and cleans up on delete. The alternative is CAAPH (cluster-api-addon-provider-helm), but that installs into the workload cluster, and we want CA in the management cluster.

Self-service UX

• ClusterClass lab-tenant with variables: Kubernetes version, worker size, min/max workers, VLAN.
• A tenant cluster is one Cluster with spec.topology referencing the class.
• A namespace per tenant with RBAC limited to its own Cluster; the kubeconfig is handed out from the <tenant>-kubeconfig secret.

Open questions and risks

• CAPI version skew: Kamaji CP provider v0.21.0 is built against CAPI v1.13.4, while CAPMOX v0.9.x is tested on v1.12. All providers share one CAPI core, so either find a Kamaji CP provider release built on 1.12 or wait for CAPMOX to support 1.13+. Check each provider's metadata.yaml contract before installing.
• Kamaji OSS release model: OSS publishes edge releases only (latest tag edge-26.2.5, Helm chart 0.0.0+latest). Stable versioned builds are commercial (CLASTIX), so pin by edge tag and image digest.
• Tenant Kubernetes version range is bounded by the Kamaji release and by our node images. We need one image-builder template per supported minor.
• Backup and restore of the shared datastore (etcd snapshots) becomes critical: it holds every tenant.

Exit criteria

• Tenant cluster created from a single Cluster + ClusterClass, control plane as pods in mgmt.
• Workers join via Konnectivity, and CNI works.
• Per-tenant cluster-autoscaler deployed automatically, scale 0 → N → 0 works.
• Deleting the Cluster removes the control plane, VMs, VIP and autoscaler.