kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial import: phase docs, configs, Claude conventions

Browse Source 
Phase 1-3 runbooks and design notes, version pins, config templates,
project-history conventions, cluster-inspector read-only credentials setup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-27 00:58:52 +02:00
co-authored by Claude Opus 5.5
commit 5fd998c5f9
26 changed files with 1124 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/decisions.md
+22
View File
@@ -0,0 +1,22 @@
# Decisions
Every direction chosen, rejected or reversed, newest at the bottom. Rows are never rewritten:
a reversal is a new row, and the old row's status becomes `superseded by #N`. Status is
`active`, `proposed` (not yet confirmed by lab work) or `superseded by #N`.
Rows 1–9 were back-filled on 2026-09-26 from the README, `versions.md`, the phase docs and the
image-builder fixes log; their dates are when the decision was recorded, not necessarily made.
| # | Date | Decision | Alternatives considered | Why | Status |
| --- | --- | --- | --- | --- | --- |
| 1 | 2026-09-24 | Pin CAPI core to 1.12.x ([versions.md](versions.md)) | latest CAPI 1.14.x | CAPMOX v0.9.x is only tested against CAPI 1.11/1.12; caps workload Kubernetes and cluster-autoscaler at 1.35 | active |
| 2 | 2026-09-24 | Management cluster is k3s on a Proxmox VM | kind on a workstation | CAPI controllers and the autoscaler must run permanently | active |
| 3 | 2026-09-24 | Node images built with kubernetes-sigs/image-builder (Ubuntu 24.04, Proxmox builder) | — | upstream-maintained CAPI images | active |
| 4 | 2026-09-24 | image-builder overrides go in a `PACKER_VAR_FILES` JSON | `PACKER_FLAGS` | flags were silently lost when `PACKER_FLAGS` was re-exported ([fixes log #4](image-builder-proxmox.md#fixes-log)) | active |
| 5 | 2026-09-24 | Cilium installed by Helm with `ipam.mode=kubernetes`, pod CIDR `10.244.0.0/16` | ClusterResourceSet CNI flavors; pod CIDR `192.168.0.0/16` | keeps the CNI version under our control; `192.168.0.0/16` overlaps the lab LAN | active |
| 6 | 2026-09-24 | Scale-from-zero via `capacity.cluster-autoscaler.kubernetes.io/*` annotations on the MachineDeployment | infrastructure provider publishing `status.capacity` | CAPMOX does not publish `status.capacity` on `ProxmoxMachineTemplate` | active |
| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | active |
| 8 | 2026-09-26 | Phase 2: spike CAPLV first; own provider if insufficient; KubeVirt as fallback ([phase-2](phase-2-kvm-provider.md#suggested-approach)) | CAPK + KubeVirt, Metal3, own provider from the start | CAPLV shares CAPI 1.12 with CAPMOX and fits plain libvirt hosts directly | proposed |
| 9 | 2026-09-26 | Phases 2 and 3 are independent and can be done in either order | strict sequence | no technical dependency between the KVM provider and hosted control planes | active |
| 10 | 2026-09-26 | Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog ([session](sessions/2026-09-26-2355-history-conventions.md)) | chat history only; raw transcripts committed | research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets | active |
| 11 | 2026-09-27 | Cluster inspection by agents only via read-only ServiceAccount kubeconfigs minted by `.claude/cluster-inspector/make-kubeconfig.sh` (short-lived TokenRequest tokens, gitignored) | admin kubeconfig + permission allowlist; long-lived SA token Secret; RBAC under `configs/` | RBAC (`view`, no Secrets) enforces read-only regardless of which kubectl verb an agent runs — also closes the `kubectl get secret` gap from the history-conventions session; short-lived tokens limit leak impact | active |
docs/image-builder-proxmox.md
+85
View File
@@ -0,0 +1,85 @@
# Node VM template with image-builder (Proxmox)
Result: Proxmox template `ubuntu-2404-kube-v1.35.9` on node `pv1-1` — Ubuntu 24.04.4, containerd,
kubeadm/kubelet/kubectl v1.35.9, pre-pulled control-plane images, cloud-init, qemu-guest-agent.
Its VMID is `TEMPLATE_VMID` for CAPMOX.
## Build host prerequisites (Ubuntu 24.04)
```bash
sudo apt install -y make git curl unzip ca-certificates openssl openssh-client \
python3 python3-venv python3-pip
python3 -m venv ~/ib-venv && source ~/ib-venv/bin/activate
pip install ansible-core==2.18.18 # avoids PEP 668 / --break-system-packages
git clone https://github.com/kubernetes-sigs/image-builder.git ~/image-builder
cd ~/image-builder/images/capi && git checkout v0.1.55
make deps-proxmox # python/ansible/packer 1.9.5 checks + packer init
```
## Build
Overrides live in a var file ([configs/image-builder/proxmox-lab.json.example](../configs/image-builder/proxmox-lab.json.example)),
passed via `PACKER_VAR_FILES` (last on the packer command line → highest precedence).
```bash
export PROXMOX_URL="https://192.168.0.8:8006/api2/json"
export PROXMOX_USERNAME='capmox@pve!capi'
export PROXMOX_TOKEN="<secret>"
export PROXMOX_NODE="pv1-1" # PVE node *name* (pvesh get /nodes), not IP; template lands here
export PROXMOX_ISO_POOL="local" # storage with content 'iso' (pvesm status --content iso)
export PROXMOX_STORAGE_POOL="local-lvm" # storage with content 'images' (pvesm status --content images)
export PROXMOX_BRIDGE="vmbr0"
export PROXMOX_VLAN="4"
export PACKER_VAR_FILES=~/proxmox-lab.json
unset PACKER_FLAGS ISO_FILE
export SSH_PASSWORD=labpass # optional: ssh builder@<vm> while provisioning runs
ON_ERROR_ASK=1 PACKER_LOG=1 PACKER_LOG_PATH=packer.log make build-proxmox-ubuntu-2404
```
The build VM (VLAN 4) needs DHCP and must reach the Packer HTTP server on the build host
(ports 8000–9000) to fetch the autoinstall `user-data`.
## What the build does
1. `set-ssh-password` renders the autoinstall `user-data` with a hashed password for user `builder`.
2. Packer creates VM `packer-<uuid>` (2 vCPU, 2 GB, 20 GB), attaches the ISO, serves `http/24.04/`.
3. After `boot_wait` it types the GRUB boot command over VNC: `autoinstall ds=nocloud-net;s=http://<host>:<port>/24.04/`.
4. Subiquity installs Ubuntu and reboots; Packer waits for SSH (up to 2h), IP via guest agent.
5. Provisioners: wait for cloud-init → ansible `firstboot.yml` → reboot → ansible `node.yml`
(containerd, k8s binaries, images, sysctl, cloud-init config) → goss tests → lock `builder`, shutdown.
6. VM converted to template `ubuntu-2404-kube-v1.35.9`.
Observe it in the PVE noVNC console; `DEBUG=1` steps through each Packer stage; `make -n` prints the
exact `packer build` command. Typical duration 20–40 min.
## Fixes log
| # | Symptom | Root cause | Fix |
| --- | --- | --- | --- |
| 1 | `POST .../storage/local/upload: write: broken pipe` | pveproxy dropped the 3.17 GiB upload mid-stream — missing `Datastore.AllocateTemplate` (token had only `PVEVMAdmin`) or no spool space in `/var/tmp` (exact cause not pinned down) | `PVEDatastoreAdmin` on `/storage`, check `df -h /var/tmp`; then avoid re-uploads with `iso_file` |
| 2 | `net0.bridge: invalid bridge ID 'vmbr0.4@vmbr0'` | `PROXMOX_BRIDGE` set to the `ip link` name of a VLAN sub-interface; PVE wants a bridge name + separate tag | `PROXMOX_BRIDGE=vmbr0`, `PROXMOX_VLAN=4` (→ `bridge=vmbr0,tag=4`) |
| 3 | `403 Permission check failed (/sdn/zones/localnetwork/vmbr0.4, SDN.Use)` | since PVE 8 every Linux bridge is an object in SDN zone `localnetwork`; `PVEVMAdmin` lacks `SDN.Use` | `PVESDNUser` on `/sdn/zones/localnetwork` |
| 4 | `unsupported format 'qcow2' at .../LvmThinPlugin.pm line 102` | image-builder defaults `disk_format` to qcow2; LVM-thin is raw-only. `--var disk_format=raw` in `PACKER_FLAGS` was lost when `PACKER_FLAGS` was re-exported | `"disk_format": "raw"` in the `PACKER_VAR_FILES` JSON |
| 5 | `one of iso_file, iso_url, or ... cd_content must be specified for boot_iso` | packer-plugin-proxmox v1.2.4 requires **exactly one** ISO source; `ubuntu-2404.json` sets `iso_file` from `$ISO_FILE` *and* a hardcoded `iso_url` → count 2. Misleading error; README tip is broken (worth an upstream issue) | set `iso_file` and `"iso_url": ""` in the var file, unset `ISO_FILE` |
Notes carried into CAPMOX (phase 1): same ACL set, `vlan: 4` on the network device, no
`format: qcow2` in `ProxmoxMachineTemplate`s.
## Smoke test of the template
The template has no usable login (`builder` is locked) — access only via cloud-init (NoCloud,
which the PVE Cloud-Init drive provides):
```bash
TPL=<template vmid>; ID=9001
qm clone $TPL $ID --name smoke-k8s --full 1 --storage local-lvm
qm set $ID --ide2 local-lvm:cloudinit
qm set $ID --ciuser jan --sshkeys /root/jan.pub --ipconfig0 ip=dhcp
qm set $ID --net0 virtio,bridge=vmbr0,tag=4
qm start $ID && qm guest cmd $ID network-get-interfaces
# inside: cloud-init status --long; kubeadm version -o short; crictl images
qm stop $ID && qm destroy $ID --purge
```
kubelet crash-looping before `kubeadm init/join` is expected.
docs/phase-1-proxmox-capi.md
+173
View File
@@ -0,0 +1,173 @@
# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler
Goal: a manually installed k3s management cluster runs CAPI + CAPMOX and creates workload
cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
No custom code in this phase — only assembling upstream components.
Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md).
## Topology
| Role | Where | Size |
| --- | --- | --- |
| Management cluster | 1 VM on Proxmox, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 8 GB RAM |
## Checklist
- [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
- [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
- [ ] Step 2 — management cluster + `clusterctl init`
- [ ] Step 3 — workload cluster `lab1`
- [ ] Step 4 — CNI (Cilium)
- [ ] Step 5 — MachineDeployment autoscaling annotations
- [ ] Step 6 — cluster-autoscaler
- [ ] Step 7 — scale tests
## Step 0 — Proxmox prep
```bash
pveum user add capmox@pve
pveum user token add capmox@pve capi -privsep 0 # token inherits user ACLs
pveum aclmod / -user capmox@pve -role PVEVMAdmin
pveum aclmod / -user capmox@pve -role PVEAuditor # Sys.Audit etc.
pveum aclmod /storage -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks
pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser # NIC on any local bridge (PVE 8+)
```
Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`.
Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP
for the control plane endpoint in the same subnet.
Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node
requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template.
## Step 2 — management cluster
```bash
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.8+k3s1 sh -s - --disable traefik
mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config
curl -L -o clusterctl https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-linux-amd64
sudo install -m 0755 clusterctl /usr/local/bin/clusterctl
```
`~/.cluster-api/clusterctl.yaml` — start from [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example).
```bash
clusterctl init \
--core cluster-api:v1.12.11 \
--bootstrap kubeadm:v1.12.11 \
--control-plane kubeadm:v1.12.11 \
--infrastructure proxmox:v0.9.1 \
--ipam in-cluster:v1.1.1
```
Always pin versions explicitly — without them clusterctl installs the newest core (1.14.x) and
breaks the tested CAPMOX combination.
## Step 3 — workload cluster `lab1`
```bash
clusterctl generate cluster lab1 \
--infrastructure proxmox:v0.9.1 \
--kubernetes-version v1.35.9 \
--control-plane-machine-count 1 \
--worker-machine-count 1 > lab1.yaml
```
Edit `lab1.yaml` before applying:
1. **Pod CIDR** — `Cluster.spec.clusterNetwork.pods.cidrBlocks` defaults to `192.168.0.0/16`, which
overlaps the lab network → use e.g. `10.244.0.0/16`.
2. **kube-vip** — bump `ghcr.io/kube-vip/kube-vip:v0.7.1` → `v1.2.4`; keep `kube-vip-prepare.sh`
(handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 8192`.
6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.
```bash
kubectl apply -f lab1.yaml
clusterctl describe cluster lab1 --show-conditions all
kubectl get proxmoxmachines,machines -w
clusterctl get kubeconfig lab1 > lab1.kubeconfig
```
Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses`). providerID is set by
the CAPMOX template as `proxmox://{{ ds.meta_data.instance_id }}` — don't touch it, the autoscaler
matches Nodes to Machines by it.
## Step 4 — Cilium
```bash
export KUBECONFIG=$PWD/lab1.kubeconfig
helm repo add cilium https://helm.cilium.io && helm repo update
helm install cilium cilium/cilium --version 1.20.2 -n kube-system --set ipam.mode=kubernetes
```
`ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default.
## Step 5 — autoscaling annotations
```yaml
metadata:
name: lab1-workers
annotations:
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores
capacity.cluster-autoscaler.kubernetes.io/memory: "8Gi"
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
```
Capacity annotations are required for scale-from-zero because CAPMOX does not publish
`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints`
annotations if pods target the group by selector/toleration.
## Step 6 — cluster-autoscaler
Install into the **same namespace as the `lab1-kubeconfig` secret** (here `default`), values in
[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):
```bash
helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update
helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \
-f configs/cluster-autoscaler/values.yaml
```
The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5.
## Step 7 — scale tests
Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 2Gi each).
- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear.
- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.
`kubectl describe pod <pending>` shows `TriggeredScaleUp` / `NotTriggerScaleUp` events — fastest diagnostic.
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
```bash
kubectl -n capmox-system logs deploy/capmox-controller-manager -f
kubectl -n capi-system logs deploy/capi-controller-manager -f
ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100'
```
docs/phase-2-kvm-provider.md
+62
View File
@@ -0,0 +1,62 @@
# Phase 2 — CAPI provider for vanilla KVM/libvirt hosts
Goal: autoscaled worker VMs on the Ubuntu 24.04 KVM/libvirt hypervisors, managed by the same
management cluster as phase 1. Status: **design / evaluation**.
## Problem
There is no mature, SIG-maintained CAPI infrastructure provider for plain libvirt. Proxmox does not
use libvirt (it drives QEMU via `qm`), so one libvirt provider cannot cover both hypervisor types.
## Options
| Option | How it works | Pros | Cons |
| --- | --- | --- | --- |
| **A. CAPLV** ([joshuagrisham/cluster-api-provider-libvirt](https://github.com/joshuagrisham/cluster-api-provider-libvirt), v0.2.1, CAPI v1.12.1) | talks libvirt RPC via pure-Go `digitalocean/go-libvirt` | same CAPI core as CAPMOX; direct fit for our hosts | young, single maintainer; feature gaps unknown (IPAM, placement, capacity) |
| **B. CAPK + KubeVirt** (CAPK v0.11.2 on CAPI v1.11.4, KubeVirt v1.9.0) | Ubuntu hosts become a Kubernetes "infra cluster" running KubeVirt; CAPK creates `VirtualMachine`s there | kubernetes-sigs project, mature, live migration, CDI for images | replaces plain libvirt management with a whole k8s + KubeVirt stack on the hypervisors; nested networking design (bridge/Multus) |
| **C. Metal3** (CAPM3 v1.14.1) + sushy-tools | VMs pre-defined in libvirt, exposed as "bare metal" via Redfish emulator, provisioned by Ironic | production-grade bare-metal flow | Ironic is heavy; VM pool is fixed → not real elasticity |
| **D. Own provider** | kubebuilder project, `LibvirtCluster/LibvirtMachine/LibvirtMachineTemplate` | full control, best learning value; can add multi-backend and `status.capacity` | we own maintenance |
## Suggested approach
1. **Spike A (≈ 1 week):** deploy CAPLV next to CAPMOX (both on CAPI 1.12), one MachineDeployment
on one Ubuntu host. Check: cloud-init/bootstrap handling, IP assignment (DHCP vs CAPI IPAM),
`providerID` format, delete/cleanup of volumes, behaviour under cluster-autoscaler.
2. If A is insufficient → **D**, reusing CAPLV's patterns, or contribute upstream.
3. Keep **B** as the alternative if we're willing to turn the Ubuntu hosts into a KubeVirt cluster
(it may pay off together with phase 3).
## Own provider sketch (option D)
Reconcile of `LibvirtMachine`:
1. Wait for `Machine.spec.bootstrap.dataSecretName`.
2. Pick a host (placement: one MachineDeployment per host, or least-loaded by free memory).
3. Create a qcow2 overlay volume on the base image (backing file) in the host's storage pool.
4. Build a NoCloud ISO from the bootstrap secret (+ network-config from CAPI IPAM `IPAddressClaim`).
5. `virDomainDefineXML` + start; wait for IP (guest agent / DHCP leases).
6. Set `spec.providerID = libvirt://<host>/<domain-uuid>`, `status.addresses`, `ready`.
7. Finalizer: destroy domain, delete overlay + ISO.
`LibvirtMachineTemplate.status.capacity` (cpu/memory) → scale-from-zero without annotations.
Connection: `qemu+ssh://` or `qemu+tls://` per host, credentials in a Secret.
## Mixing providers in one workload cluster
`Cluster.spec.infrastructureRef` points to **one** InfraCluster kind, and most providers' machine
reconcilers look up *their own* InfraCluster kind through it — a `ProxmoxMachine` in a cluster whose
InfraCluster is a `LibvirtCluster` won't reconcile. Choices:
- **Separate workload clusters per provider** — simplest, no hacks.
- **Tolerant own provider** — our `LibvirtMachine` works regardless of the InfraCluster kind, so a
CAPMOX-based cluster can also get libvirt worker MachineDeployments.
- **Multi-backend provider** — one `HypervisorCluster` kind with Proxmox API + libvirt backends.
cluster-autoscaler is indifferent: it scales any MachineDeployment carrying the min/max annotations,
whatever its infrastructure kind. The priority expander can prefer one hypervisor pool over another.
## Exit criteria
- [ ] Worker MachineDeployment on an Ubuntu KVM host joins a cluster managed from the phase-1 mgmt cluster.
- [ ] cluster-autoscaler scales it 0 → N → 0.
- [ ] VM and volumes fully cleaned up on scale-down.
docs/phase-3-kamaji-kaas.md
+97
View File
@@ -0,0 +1,97 @@
# Phase 3 — Hosted control planes with Kamaji ("poor man's KaaS")
Goal: create a tenant Kubernetes cluster from one manifest. Its control plane runs as pods in the
management cluster (Kamaji), and its workers are autoscaled Proxmox (later also KVM) VMs.
Status: **design notes**.
## Why Kamaji
With kubeadm control planes every tenant costs ≥ 1 VM (3 for HA) just for apiserver/etcd. Kamaji
runs each tenant control plane as a Deployment in the management cluster, backed by a shared
datastore. That cuts the control plane to a few hundred MB of RAM, gives it seconds to start, and
makes upgrades a rolling Deployment update.
## Architecture
```mermaid
flowchart LR
subgraph MGMT[management cluster - 3 nodes]
CAPI[CAPI core + CAPMOX]
KCP[Kamaji CP provider]
KAM[Kamaji controller]
DS[(kamaji-etcd datastore)]
LB[MetalLB pool on VLAN 4]
subgraph T1[ns tenant-a]
TCP1[TenantControlPlane pods<br/>apiserver, ctrl-mgr, scheduler]
CA1[cluster-autoscaler tenant-a]
end
end
PVE[Proxmox VE]
W1[tenant-a worker VMs]
KCP --> KAM --> TCP1
TCP1 --> DS
LB -. VIP :6443 .- TCP1
CAPI -- MachineDeployment --> PVE --> W1
W1 -- kubelet join via VIP --> TCP1
CA1 -- scales MD --> CAPI
```
CAPI object wiring per tenant:
- `Cluster.spec.controlPlaneRef` → `KamajiControlPlane`, which creates the Kamaji `TenantControlPlane`.
- `Cluster.spec.infrastructureRef` → `ProxmoxCluster` with **`externalManagedControlPlane: true`**,
so the Kamaji provider patches the control plane endpoint. The field exists in CAPMOX v1alpha2.
- Workers: a regular `MachineDeployment` + `KubeadmConfigTemplate` + `ProxmoxMachineTemplate`, with
autoscaler annotations as in phase 1.
- Konnectivity handles apiserver → kubelet traffic, because the control plane isn't on the worker network.
References: [Kamaji + Proxmox infra provider docs](https://kamaji.clastix.io/cluster-api/proxmox-infra-provider/),
CLASTIX Helm chart `capi-kamaji-proxmox` (0.0.4) as a wiring example.
## Management cluster changes vs. phase 1
| Area | Phase 1 | Phase 3 |
| --- | --- | --- |
| Nodes | 1× k3s | 3 nodes: tenant control planes and their datastore live here |
| LoadBalancer | none needed (kube-vip in workload CP) | MetalLB (or kube-vip cloud provider) with an IP pool on VLAN 4, one VIP per tenant |
| Datastore | — | `kamaji-etcd` chart 0.18.1 (etcd 3.6.12); alternatives: PostgreSQL/MySQL via kine |
| Storage | — | CSI for etcd PVCs |
k3s' built-in servicelb (klipper) binds host ports, so it can't hand out many `:6443` endpoints. Use MetalLB.
## Autoscaling per tenant
A cluster-autoscaler instance in `clusterapi` mode serves **one** workload cluster (one kubeconfig).
So each tenant gets its own CA release in its namespace in the management cluster
(`kubeconfig-incluster`, secret `<tenant>-kubeconfig`). Automating that is our first real Go code:
- **tenant-operator** (small controller in this repo): watches `Cluster` objects labelled
`kaas.lab/autoscaling=enabled`, renders a per-tenant cluster-autoscaler Deployment + RBAC, and
cleans up on delete. The alternative is CAAPH (cluster-api-addon-provider-helm), but that installs
into the *workload* cluster, and we want CA in the management cluster.
## Self-service UX
- `ClusterClass` `lab-tenant` with variables: Kubernetes version, worker size, min/max workers, VLAN.
- A tenant cluster is one `Cluster` with `spec.topology` referencing the class.
- A namespace per tenant with RBAC limited to its own `Cluster`; the kubeconfig is handed out from the
`<tenant>-kubeconfig` secret.
## Open questions and risks
- **CAPI version skew:** Kamaji CP provider v0.21.0 is built against CAPI **v1.13.4**, while CAPMOX
v0.9.x is tested on **v1.12**. All providers share one CAPI core, so either find a Kamaji CP
provider release built on 1.12 or wait for CAPMOX to support 1.13+. Check each provider's
`metadata.yaml` contract before installing.
- **Kamaji OSS release model:** OSS publishes edge releases only (latest tag `edge-26.2.5`, Helm chart
`0.0.0+latest`). Stable versioned builds are commercial (CLASTIX), so pin by edge tag and image digest.
- Tenant Kubernetes version range is bounded by the Kamaji release and by our node images. We need
one image-builder template per supported minor.
- Backup and restore of the shared datastore (etcd snapshots) becomes critical: it holds every tenant.
## Exit criteria
- [ ] Tenant cluster created from a single `Cluster` + `ClusterClass`, control plane as pods in mgmt.
- [ ] Workers join via Konnectivity, and CNI works.
- [ ] Per-tenant cluster-autoscaler deployed automatically, scale 0 → N → 0 works.
- [ ] Deleting the `Cluster` removes the control plane, VMs, VIP and autoscaler.
docs/plans-executions/.gitkeep
View File
docs/plans/.gitkeep
View File
docs/prompts/.gitkeep
View File
docs/reviews/.gitkeep
View File
docs/sessions/2026-09-26-2355-history-conventions.md
+51
View File
@@ -0,0 +1,51 @@
# Session: which conventions to adopt from egress-proxies-operator
**Created:** 2026-09-26 23:55
## Question
> look into project in /Users/jan.novak/srv/go/egress-proxies-operator and suggest which rules i
> should copy over into this project
>
> this project is sort of research/experimentation project, so i want to keep history of
> conversations, decisions, plans, etc.
## What was checked
`egress-proxies-operator`: `CLAUDE.md`, `.claude/settings.json` + `settings.local.json`,
`.claude/commands/{ship,release}.md`, `.claude/agents/*`, `AGENTS.md`, and samples from
`docs/plans/`, `docs/plans-executions/`, `docs/prompts/`, `docs/reviews/`, `CHANGELOG.md`.
## Findings
Adopted, adapted to a docs/lab repo:
- Plans in `docs/plans/` with the "commit the plan first, on its own" ordering rule.
- Execution logs in `docs/plans-executions/`, especially "exact commands with one line of
context, deviations called out", which suits lab runbook work better than Go code.
- `docs/prompts/`, `docs/reviews/`, top-inserted `CHANGELOG.md` with real `date` timestamps.
- Gitea + `tea pr create` MR flow, but only for phase-3 Go code; docs go straight to `main`.
- `/ship` command; a small read-only permission allowlist (`kubectl get/describe/logs`,
`clusterctl describe`, `helm list`, git read commands).
Not adopted: Go and testing conventions, CI/CD, kubebuilder gotchas, `AGENTS.md`, the two
operator reviewer agents (candidates for phase 3), `/release` (no tags or images here),
`.junie/.codex/.agents` duplicates, and the broad `kubectl *` / `curl *` permissions.
New, not in the operator repo, because decisions there are buried in plan context sections:
- `docs/decisions.md`: one table with alternatives and status, back-filled with 9 rows.
- `docs/sessions/`: summaries of conversations that did not produce a plan (this file is the first).
## Open questions
- ~~Whether `kubectl get *` in the allowlist is too broad: `kubectl get secret -o yaml` would
print the `lab1-kubeconfig` secret without a prompt.~~ Resolved by
[decision #11](../decisions.md): a parallel session introduced read-only `cluster-inspector`
kubeconfigs, so the `kubectl` / `clusterctl` / `helm` allowlist entries were dropped from
`.claude/settings.json` again.
## Follow-ups
- [Decision #10](../decisions.md)
- `CLAUDE.md` sections "Project history" and "Git"
docs/versions.md
+44
View File
@@ -0,0 +1,44 @@
# Versions
Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go module proxy,
`dl.k8s.io`, `registry.k8s.io`, upstream Helm indexes and git tags. Re-verify before every deploy.
## Phase 1 — pinned
| Component | Pinned | Latest available | Why pinned there |
| --- | --- | --- | --- |
| Cluster API core, kubeadm bootstrap + control plane, clusterctl | **v1.12.11** | v1.14.2 | CAPMOX v0.9 compatibility table lists CAPI v1.11 / v1.12 only; built against v1.12.11 |
| CAPMOX (ionos-cloud/cluster-api-provider-proxmox) | **v0.9.1** | v0.9.1 | API `v1alpha2`, contract `v1beta2` |
| IPAM provider in-cluster | **v1.1.1** | v1.1.1 | CAPMOX v0.9 requires ≥ 1.1.0 |
| Workload Kubernetes | **v1.35.9** | v1.37.1 | CAPI 1.12 supports workload clusters up to 1.35 |
| Management cluster (k3s) | **v1.35.8+k3s1** | v1.37.0+k3s1 | CAPI 1.12 supports management clusters 1.31–1.35 |
| cluster-autoscaler | **v1.35.2** | v1.36.1 | CA minor must match workload Kubernetes minor |
| cluster-autoscaler Helm chart | **9.59.0** | 9.59.0 | default appVersion 1.35.0 → override `image.tag` |
| Cilium | **1.20.2** | 1.20.2 | |
| kube-vip | **v1.2.4** | v1.2.4 | CAPMOX template still ships v0.7.1 |
| image-builder | **v0.1.55** | v0.1.55 | |
| Packer (installed by image-builder) | **1.9.5** | — | last MPL-licensed release, pinned by image-builder |
| packer-plugin-proxmox | **v1.2.4** | v1.2.4 | pulled by `packer init` (`>= 1.1.2`) |
| ansible-core (image-builder) | **2.18.18** | — | pinned by image-builder `hack/utils.sh` |
| Node OS | Ubuntu 24.04.4 | — | ISO sha256 `e907d92e…a138433` |
### Upgrade path
When CAPMOX adds CAPI 1.13/1.14 to its compatibility table: bump CAPI core → k3s mgmt to 1.36+ →
rebuild template with Kubernetes 1.36/1.37 → cluster-autoscaler to the matching minor.
## Phase 2 / 3 — candidates (not pinned yet)
| Component | Latest | Built against CAPI | Notes |
| --- | --- | --- | --- |
| CAPLV (joshuagrisham/cluster-api-provider-libvirt) | v0.2.1 (2026-02) | v1.12.1 | pure-Go `digitalocean/go-libvirt`, single maintainer |
| CAPK (cluster-api-provider-kubevirt) | v0.11.2 (2026-02) | v1.11.4 | needs KubeVirt on the hypervisors |
| KubeVirt | v1.9.0 (2026-07) | — | |
| CAPM3 (Metal3) | v1.14.1 (2026-09) | — | needs Ironic + BMO + sushy-tools for VMs |
| cluster-api-control-plane-provider-kamaji | v0.21.0 (2026-09-16) | **v1.13.4** | ⚠ newer CAPI than CAPMOX v0.9 is tested with |
| Kamaji (OSS) | `edge-26.2.5`, Helm chart `0.0.0+latest` | — | OSS ships edge releases only; last semver tag v1.0.0 (2024) |
| kamaji-etcd Helm chart | 0.18.1 (etcd 3.6.12) | — | datastore for hosted control planes |
| CLASTIX `capi-kamaji-proxmox` chart | 0.0.4 (2025-10) | — | reference for Kamaji + CAPMOX wiring |
The CAPI version skew between CAPMOX (1.12) and the Kamaji CP provider (1.13) is the main open
compatibility question for phase 3 — all providers in one management cluster share one CAPI core.