kacerrandClaude Opus 5.5
5fd998c5f9
Phase 1-3 runbooks and design notes, version pins, config templates, project-history conventions, cluster-inspector read-only credentials setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
98 lines
4.6 KiB
Markdown
98 lines
4.6 KiB
Markdown
# Phase 3 — Hosted control planes with Kamaji ("poor man's KaaS")
|
||
|
||
Goal: create a tenant Kubernetes cluster from one manifest. Its control plane runs as pods in the
|
||
management cluster (Kamaji), and its workers are autoscaled Proxmox (later also KVM) VMs.
|
||
Status: **design notes**.
|
||
|
||
## Why Kamaji
|
||
|
||
With kubeadm control planes every tenant costs ≥ 1 VM (3 for HA) just for apiserver/etcd. Kamaji
|
||
runs each tenant control plane as a Deployment in the management cluster, backed by a shared
|
||
datastore. That cuts the control plane to a few hundred MB of RAM, gives it seconds to start, and
|
||
makes upgrades a rolling Deployment update.
|
||
|
||
## Architecture
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph MGMT[management cluster - 3 nodes]
|
||
CAPI[CAPI core + CAPMOX]
|
||
KCP[Kamaji CP provider]
|
||
KAM[Kamaji controller]
|
||
DS[(kamaji-etcd datastore)]
|
||
LB[MetalLB pool on VLAN 4]
|
||
subgraph T1[ns tenant-a]
|
||
TCP1[TenantControlPlane pods<br/>apiserver, ctrl-mgr, scheduler]
|
||
CA1[cluster-autoscaler tenant-a]
|
||
end
|
||
end
|
||
PVE[Proxmox VE]
|
||
W1[tenant-a worker VMs]
|
||
KCP --> KAM --> TCP1
|
||
TCP1 --> DS
|
||
LB -. VIP :6443 .- TCP1
|
||
CAPI -- MachineDeployment --> PVE --> W1
|
||
W1 -- kubelet join via VIP --> TCP1
|
||
CA1 -- scales MD --> CAPI
|
||
```
|
||
|
||
CAPI object wiring per tenant:
|
||
|
||
- `Cluster.spec.controlPlaneRef` → `KamajiControlPlane`, which creates the Kamaji `TenantControlPlane`.
|
||
- `Cluster.spec.infrastructureRef` → `ProxmoxCluster` with **`externalManagedControlPlane: true`**,
|
||
so the Kamaji provider patches the control plane endpoint. The field exists in CAPMOX v1alpha2.
|
||
- Workers: a regular `MachineDeployment` + `KubeadmConfigTemplate` + `ProxmoxMachineTemplate`, with
|
||
autoscaler annotations as in phase 1.
|
||
- Konnectivity handles apiserver → kubelet traffic, because the control plane isn't on the worker network.
|
||
|
||
References: [Kamaji + Proxmox infra provider docs](https://kamaji.clastix.io/cluster-api/proxmox-infra-provider/),
|
||
CLASTIX Helm chart `capi-kamaji-proxmox` (0.0.4) as a wiring example.
|
||
|
||
## Management cluster changes vs. phase 1
|
||
|
||
| Area | Phase 1 | Phase 3 |
|
||
| --- | --- | --- |
|
||
| Nodes | 1× k3s | 3 nodes: tenant control planes and their datastore live here |
|
||
| LoadBalancer | none needed (kube-vip in workload CP) | MetalLB (or kube-vip cloud provider) with an IP pool on VLAN 4, one VIP per tenant |
|
||
| Datastore | — | `kamaji-etcd` chart 0.18.1 (etcd 3.6.12); alternatives: PostgreSQL/MySQL via kine |
|
||
| Storage | — | CSI for etcd PVCs |
|
||
|
||
k3s' built-in servicelb (klipper) binds host ports, so it can't hand out many `:6443` endpoints. Use MetalLB.
|
||
|
||
## Autoscaling per tenant
|
||
|
||
A cluster-autoscaler instance in `clusterapi` mode serves **one** workload cluster (one kubeconfig).
|
||
So each tenant gets its own CA release in its namespace in the management cluster
|
||
(`kubeconfig-incluster`, secret `<tenant>-kubeconfig`). Automating that is our first real Go code:
|
||
|
||
- **tenant-operator** (small controller in this repo): watches `Cluster` objects labelled
|
||
`kaas.lab/autoscaling=enabled`, renders a per-tenant cluster-autoscaler Deployment + RBAC, and
|
||
cleans up on delete. The alternative is CAAPH (cluster-api-addon-provider-helm), but that installs
|
||
into the *workload* cluster, and we want CA in the management cluster.
|
||
|
||
## Self-service UX
|
||
|
||
- `ClusterClass` `lab-tenant` with variables: Kubernetes version, worker size, min/max workers, VLAN.
|
||
- A tenant cluster is one `Cluster` with `spec.topology` referencing the class.
|
||
- A namespace per tenant with RBAC limited to its own `Cluster`; the kubeconfig is handed out from the
|
||
`<tenant>-kubeconfig` secret.
|
||
|
||
## Open questions and risks
|
||
|
||
- **CAPI version skew:** Kamaji CP provider v0.21.0 is built against CAPI **v1.13.4**, while CAPMOX
|
||
v0.9.x is tested on **v1.12**. All providers share one CAPI core, so either find a Kamaji CP
|
||
provider release built on 1.12 or wait for CAPMOX to support 1.13+. Check each provider's
|
||
`metadata.yaml` contract before installing.
|
||
- **Kamaji OSS release model:** OSS publishes edge releases only (latest tag `edge-26.2.5`, Helm chart
|
||
`0.0.0+latest`). Stable versioned builds are commercial (CLASTIX), so pin by edge tag and image digest.
|
||
- Tenant Kubernetes version range is bounded by the Kamaji release and by our node images. We need
|
||
one image-builder template per supported minor.
|
||
- Backup and restore of the shared datastore (etcd snapshots) becomes critical: it holds every tenant.
|
||
|
||
## Exit criteria
|
||
|
||
- [ ] Tenant cluster created from a single `Cluster` + `ClusterClass`, control plane as pods in mgmt.
|
||
- [ ] Workers join via Konnectivity, and CNI works.
|
||
- [ ] Per-tenant cluster-autoscaler deployed automatically, scale 0 → N → 0 works.
|
||
- [ ] Deleting the `Cluster` removes the control plane, VMs, VIP and autoscaler.
|