# Phase 3 — Hosted control planes with Kamaji ("poor man's KaaS")

Goal: create a tenant Kubernetes cluster from one manifest. Its control plane runs as pods in the
management cluster (Kamaji), and its workers are autoscaled Proxmox (later also KVM) VMs.
Status: **design notes**.

## Why Kamaji

With kubeadm control planes every tenant costs ≥ 1 VM (3 for HA) just for apiserver/etcd. Kamaji
runs each tenant control plane as a Deployment in the management cluster, backed by a shared
datastore. That cuts the control plane to a few hundred MB of RAM, gives it seconds to start, and
makes upgrades a rolling Deployment update.

## Architecture

```mermaid
flowchart LR
  subgraph MGMT[management cluster - 3 nodes]
    CAPI[CAPI core + CAPMOX]
    KCP[Kamaji CP provider]
    KAM[Kamaji controller]
    DS[(kamaji-etcd datastore)]
    LB[MetalLB pool on VLAN 4]
    subgraph T1[ns tenant-a]
      TCP1[TenantControlPlane pods<br/>apiserver, ctrl-mgr, scheduler]
      CA1[cluster-autoscaler tenant-a]
    end
  end
  PVE[Proxmox VE]
  W1[tenant-a worker VMs]
  KCP --> KAM --> TCP1
  TCP1 --> DS
  LB -. VIP :6443 .- TCP1
  CAPI -- MachineDeployment --> PVE --> W1
  W1 -- kubelet join via VIP --> TCP1
  CA1 -- scales MD --> CAPI
```

CAPI object wiring per tenant:

- `Cluster.spec.controlPlaneRef` → `KamajiControlPlane`, which creates the Kamaji `TenantControlPlane`.
- `Cluster.spec.infrastructureRef` → `ProxmoxCluster` with **`externalManagedControlPlane: true`**,
  so the Kamaji provider patches the control plane endpoint. The field exists in CAPMOX v1alpha2.
- Workers: a regular `MachineDeployment` + `KubeadmConfigTemplate` + `ProxmoxMachineTemplate`, with
  autoscaler annotations as in phase 1.
- Konnectivity handles apiserver → kubelet traffic, because the control plane isn't on the worker network.

References: [Kamaji + Proxmox infra provider docs](https://kamaji.clastix.io/cluster-api/proxmox-infra-provider/),
CLASTIX Helm chart `capi-kamaji-proxmox` (0.0.4) as a wiring example.

## Management cluster changes vs. phase 1

| Area | Phase 1 | Phase 3 |
| --- | --- | --- |
| Nodes | 1× k3s | 3 nodes: tenant control planes and their datastore live here |
| LoadBalancer | none needed (kube-vip in workload CP) | MetalLB (or kube-vip cloud provider) with an IP pool on VLAN 4, one VIP per tenant |
| Datastore | — | `kamaji-etcd` chart 0.18.1 (etcd 3.6.12); alternatives: PostgreSQL/MySQL via kine |
| Storage | — | CSI for etcd PVCs |

k3s' built-in servicelb (klipper) binds host ports, so it can't hand out many `:6443` endpoints. Use MetalLB.

## Autoscaling per tenant

A cluster-autoscaler instance in `clusterapi` mode serves **one** workload cluster (one kubeconfig).
So each tenant gets its own CA release in its namespace in the management cluster
(`kubeconfig-incluster`, secret `<tenant>-kubeconfig`). Automating that is our first real Go code:

- **tenant-operator** (small controller in this repo): watches `Cluster` objects labelled
  `kaas.lab/autoscaling=enabled`, renders a per-tenant cluster-autoscaler Deployment + RBAC, and
  cleans up on delete. The alternative is CAAPH (cluster-api-addon-provider-helm), but that installs
  into the *workload* cluster, and we want CA in the management cluster.

## Self-service UX

- `ClusterClass` `lab-tenant` with variables: Kubernetes version, worker size, min/max workers, VLAN.
- A tenant cluster is one `Cluster` with `spec.topology` referencing the class.
- A namespace per tenant with RBAC limited to its own `Cluster`; the kubeconfig is handed out from the
  `<tenant>-kubeconfig` secret.

## Open questions and risks

- **CAPI version skew:** Kamaji CP provider v0.21.0 is built against CAPI **v1.13.4**, while CAPMOX
  v0.9.x is tested on **v1.12**. All providers share one CAPI core, so either find a Kamaji CP
  provider release built on 1.12 or wait for CAPMOX to support 1.13+. Check each provider's
  `metadata.yaml` contract before installing.
- **Kamaji OSS release model:** OSS publishes edge releases only (latest tag `edge-26.2.5`, Helm chart
  `0.0.0+latest`). Stable versioned builds are commercial (CLASTIX), so pin by edge tag and image digest.
- Tenant Kubernetes version range is bounded by the Kamaji release and by our node images. We need
  one image-builder template per supported minor.
- Backup and restore of the shared datastore (etcd snapshots) becomes critical: it holds every tenant.

## Exit criteria

- [ ] Tenant cluster created from a single `Cluster` + `ClusterClass`, control plane as pods in mgmt.
- [ ] Workers join via Konnectivity, and CNI works.
- [ ] Per-tenant cluster-autoscaler deployed automatically, scale 0 → N → 0 works.
- [ ] Deleting the `Cluster` removes the control plane, VMs, VIP and autoscaler.