Initial import: phase docs, configs, Claude conventions
Phase 1-3 runbooks and design notes, version pins, config templates, project-history conventions, cluster-inspector read-only credentials setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -0,0 +1,97 @@
|
||||
# Phase 3 — Hosted control planes with Kamaji ("poor man's KaaS")
|
||||
|
||||
Goal: create a tenant Kubernetes cluster from one manifest. Its control plane runs as pods in the
|
||||
management cluster (Kamaji), and its workers are autoscaled Proxmox (later also KVM) VMs.
|
||||
Status: **design notes**.
|
||||
|
||||
## Why Kamaji
|
||||
|
||||
With kubeadm control planes every tenant costs ≥ 1 VM (3 for HA) just for apiserver/etcd. Kamaji
|
||||
runs each tenant control plane as a Deployment in the management cluster, backed by a shared
|
||||
datastore. That cuts the control plane to a few hundred MB of RAM, gives it seconds to start, and
|
||||
makes upgrades a rolling Deployment update.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph MGMT[management cluster - 3 nodes]
|
||||
CAPI[CAPI core + CAPMOX]
|
||||
KCP[Kamaji CP provider]
|
||||
KAM[Kamaji controller]
|
||||
DS[(kamaji-etcd datastore)]
|
||||
LB[MetalLB pool on VLAN 4]
|
||||
subgraph T1[ns tenant-a]
|
||||
TCP1[TenantControlPlane pods<br/>apiserver, ctrl-mgr, scheduler]
|
||||
CA1[cluster-autoscaler tenant-a]
|
||||
end
|
||||
end
|
||||
PVE[Proxmox VE]
|
||||
W1[tenant-a worker VMs]
|
||||
KCP --> KAM --> TCP1
|
||||
TCP1 --> DS
|
||||
LB -. VIP :6443 .- TCP1
|
||||
CAPI -- MachineDeployment --> PVE --> W1
|
||||
W1 -- kubelet join via VIP --> TCP1
|
||||
CA1 -- scales MD --> CAPI
|
||||
```
|
||||
|
||||
CAPI object wiring per tenant:
|
||||
|
||||
- `Cluster.spec.controlPlaneRef` → `KamajiControlPlane`, which creates the Kamaji `TenantControlPlane`.
|
||||
- `Cluster.spec.infrastructureRef` → `ProxmoxCluster` with **`externalManagedControlPlane: true`**,
|
||||
so the Kamaji provider patches the control plane endpoint. The field exists in CAPMOX v1alpha2.
|
||||
- Workers: a regular `MachineDeployment` + `KubeadmConfigTemplate` + `ProxmoxMachineTemplate`, with
|
||||
autoscaler annotations as in phase 1.
|
||||
- Konnectivity handles apiserver → kubelet traffic, because the control plane isn't on the worker network.
|
||||
|
||||
References: [Kamaji + Proxmox infra provider docs](https://kamaji.clastix.io/cluster-api/proxmox-infra-provider/),
|
||||
CLASTIX Helm chart `capi-kamaji-proxmox` (0.0.4) as a wiring example.
|
||||
|
||||
## Management cluster changes vs. phase 1
|
||||
|
||||
| Area | Phase 1 | Phase 3 |
|
||||
| --- | --- | --- |
|
||||
| Nodes | 1× k3s | 3 nodes: tenant control planes and their datastore live here |
|
||||
| LoadBalancer | none needed (kube-vip in workload CP) | MetalLB (or kube-vip cloud provider) with an IP pool on VLAN 4, one VIP per tenant |
|
||||
| Datastore | — | `kamaji-etcd` chart 0.18.1 (etcd 3.6.12); alternatives: PostgreSQL/MySQL via kine |
|
||||
| Storage | — | CSI for etcd PVCs |
|
||||
|
||||
k3s' built-in servicelb (klipper) binds host ports, so it can't hand out many `:6443` endpoints. Use MetalLB.
|
||||
|
||||
## Autoscaling per tenant
|
||||
|
||||
A cluster-autoscaler instance in `clusterapi` mode serves **one** workload cluster (one kubeconfig).
|
||||
So each tenant gets its own CA release in its namespace in the management cluster
|
||||
(`kubeconfig-incluster`, secret `<tenant>-kubeconfig`). Automating that is our first real Go code:
|
||||
|
||||
- **tenant-operator** (small controller in this repo): watches `Cluster` objects labelled
|
||||
`kaas.lab/autoscaling=enabled`, renders a per-tenant cluster-autoscaler Deployment + RBAC, and
|
||||
cleans up on delete. The alternative is CAAPH (cluster-api-addon-provider-helm), but that installs
|
||||
into the *workload* cluster, and we want CA in the management cluster.
|
||||
|
||||
## Self-service UX
|
||||
|
||||
- `ClusterClass` `lab-tenant` with variables: Kubernetes version, worker size, min/max workers, VLAN.
|
||||
- A tenant cluster is one `Cluster` with `spec.topology` referencing the class.
|
||||
- A namespace per tenant with RBAC limited to its own `Cluster`; the kubeconfig is handed out from the
|
||||
`<tenant>-kubeconfig` secret.
|
||||
|
||||
## Open questions and risks
|
||||
|
||||
- **CAPI version skew:** Kamaji CP provider v0.21.0 is built against CAPI **v1.13.4**, while CAPMOX
|
||||
v0.9.x is tested on **v1.12**. All providers share one CAPI core, so either find a Kamaji CP
|
||||
provider release built on 1.12 or wait for CAPMOX to support 1.13+. Check each provider's
|
||||
`metadata.yaml` contract before installing.
|
||||
- **Kamaji OSS release model:** OSS publishes edge releases only (latest tag `edge-26.2.5`, Helm chart
|
||||
`0.0.0+latest`). Stable versioned builds are commercial (CLASTIX), so pin by edge tag and image digest.
|
||||
- Tenant Kubernetes version range is bounded by the Kamaji release and by our node images. We need
|
||||
one image-builder template per supported minor.
|
||||
- Backup and restore of the shared datastore (etcd snapshots) becomes critical: it holds every tenant.
|
||||
|
||||
## Exit criteria
|
||||
|
||||
- [ ] Tenant cluster created from a single `Cluster` + `ClusterClass`, control plane as pods in mgmt.
|
||||
- [ ] Workers join via Konnectivity, and CNI works.
|
||||
- [ ] Per-tenant cluster-autoscaler deployed automatically, scale 0 → N → 0 works.
|
||||
- [ ] Deleting the `Cluster` removes the control plane, VMs, VIP and autoscaler.
|
||||
Reference in New Issue
Block a user