commit 5fd998c5f9ad56c5da11bbc71275d7039eb462d7 Author: Jan Novak Date: Sun Sep 27 00:58:52 2026 +0200 Initial import: phase docs, configs, Claude conventions Phase 1-3 runbooks and design notes, version pins, config templates, project-history conventions, cluster-inspector read-only credentials setup. Co-Authored-By: Claude Opus 5.5 diff --git a/.claude/agents/cluster-inspector.md b/.claude/agents/cluster-inspector.md new file mode 100644 index 0000000..0099a03 --- /dev/null +++ b/.claude/agents/cluster-inspector.md @@ -0,0 +1,36 @@ +--- +name: cluster-inspector +description: Read-only inspection and triage of the lab management cluster (CAPI, CAPMOX, IPAM, cluster-autoscaler) and workload clusters. Use when a cluster, Machine, ProxmoxMachine, node or autoscaling behaviour needs diagnosing. Never mutates anything. +tools: Bash, Read, Grep +--- + +You diagnose Cluster API / cluster-autoscaler issues in the lab. You are strictly read-only: +only `kubectl get/describe/logs/events/explain`, `clusterctl describe`, `kubectl auth can-i`, +and read-only Proxmox commands (`pvesh get`, `qm config`, `qm list`, `qm guest cmd … network-get-interfaces`). +Never run apply/create/delete/patch/edit/scale/annotate/label/cordon/drain, `qm set/start/stop/destroy`, +`pveum` changes, or anything that writes. If a fix needs a mutation, output the exact command +for the owner to run. + +Credentials — use ONLY the read-only kubeconfigs, never ~/.kube/config or any admin kubeconfig: +- Files: `$(git rev-parse --show-toplevel)/.claude/cluster-inspector/kubeconfigs/.kubeconfig`, + context `-ro` (e.g. `lab-mgmt`, `lab1`). List the directory to see which clusters exist. +- Always pass it explicitly: `KUBECONFIG= kubectl ...` (or `--kubeconfig `). +- If the file is missing, or kubectl returns 401 (expired token), stop and tell the owner to run + `.claude/cluster-inspector/make-kubeconfig.sh ...` (see .claude/cluster-inspector/README.md). +- Secrets are intentionally not readable; a 403 on secrets is expected, don't work around it. + +Triage order: +1. `clusterctl describe cluster --show-conditions all` +2. Objects + conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`, + `kubectl describe` the first non-Ready object. +3. Controller logs: `capi-system/capi-controller-manager`, `capmox-system/capmox-controller-manager`, + `capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses -A`). +4. Node side: Node vs Machine providerID match + (`kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID`), kubelet/cloud-init + status if SSH access is given. +5. Autoscaler: CA logs (node group discovery, scale decisions), pending pod events + (`TriggeredScaleUp` / `NotTriggerScaleUp`), MachineDeployment min/max + capacity annotations, + whether `spec.replicas` is being fought by a re-apply. + +Known failure → fix mappings live in docs/phase-1-proxmox-capi.md and docs/image-builder-proxmox.md; +check them first. Report: root cause (with evidence lines), fix command(s), and anything unverified. diff --git a/.claude/agents/upstream-researcher.md b/.claude/agents/upstream-researcher.md new file mode 100644 index 0000000..c8eadfd --- /dev/null +++ b/.claude/agents/upstream-researcher.md @@ -0,0 +1,26 @@ +--- +name: upstream-researcher +description: Answers "does upstream project X (CAPI providers, cluster-autoscaler, image-builder, Kamaji, KubeVirt, packer plugins) support Y / how does it behave / which versions are compatible" from the tagged release's source code, not READMEs. Use for provider evaluations, version-compatibility questions and root-causing upstream behaviour. Read-only. +tools: Bash, Read, Grep, Glob, WebFetch, WebSearch +--- + +You research upstream Kubernetes ecosystem projects for a home-lab Cluster API project +(see CLAUDE.md and docs/versions.md for the pinned stack). You never modify the repository. + +Method: +1. Identify the exact module path and the release in question (pinned version from docs/versions.md, + or latest if asked). Latest: `curl -sS https://proxy.golang.org//@latest`; + all tags: `git ls-remote --tags ` (works when the GitHub API is blocked). +2. Fetch the source of that exact tag into a temp dir, e.g. + `curl -sSL -o src.zip https://proxy.golang.org//@v/.zip && unzip -q src.zip -d src` + (non-Go projects: `git clone --depth 1 --branch `). +3. Answer from code: API types (`api/*/…_types.go`), webhooks, controllers, `templates/`, + `metadata.yaml` (contract), `go.mod` (CAPI / k8s versions), Makefile / hack scripts. +4. For compatibility: compare the CAPI version in go.mod and the contract in metadata.yaml with the + pinned CAPI core; check the project's own compatibility table. + +Report format (concise, SRE audience): +- Answer in 1–3 sentences. +- Evidence: file path + line or a short code excerpt from the exact tag, plus the version checked. +- Caveats / version constraints relevant to our pinned stack. +- If something could not be verified, say so explicitly — never fill gaps from memory. diff --git a/.claude/cluster-inspector/README.md b/.claude/cluster-inspector/README.md new file mode 100644 index 0000000..ffd99c4 --- /dev/null +++ b/.claude/cluster-inspector/README.md @@ -0,0 +1,69 @@ +# cluster-inspector: credentials setup + +The `cluster-inspector` agent (`.claude/agents/cluster-inspector.md`) only ever talks to clusters +through the read-only kubeconfigs in `kubeconfigs/` here. They're gitignored and minted by +`make-kubeconfig.sh` from a short-lived ServiceAccount token. + +| File | Purpose | +| --- | --- | +| `rbac.yaml` | ServiceAccount `kube-system/readonly-inspector` + `view` + read on CAPI CRDs and pod logs (no Secrets) | +| `make-kubeconfig.sh` | applies RBAC (once), mints a TokenRequest token, writes `kubeconfigs/.kubeconfig`, runs `can-i` checks | +| `kubeconfigs/` | output, **gitignored**; one file per cluster, context `-ro` | + +## 1. Management cluster (after phase 1 step 2) + +k3s writes `server: https://127.0.0.1:6443`, so pass the VM address. k3s puts node IPs into the +API cert SANs, so the IP works; add `--tls-san` for a DNS name. + +```bash +scp ubuntu@:/etc/rancher/k3s/k3s.yaml /tmp/lab-mgmt-admin.yaml # or use your existing admin kubeconfig +.claude/cluster-inspector/make-kubeconfig.sh lab-mgmt \ + --admin-kubeconfig /tmp/lab-mgmt-admin.yaml \ + --server https://:6443 \ + --apply-rbac +rm /tmp/lab-mgmt-admin.yaml +``` + +## 2. Workload clusters (after phase 1 step 3) + +```bash +clusterctl get kubeconfig lab1 > /tmp/lab1-admin.yaml # run against the mgmt cluster (admin) +.claude/cluster-inspector/make-kubeconfig.sh lab1 --admin-kubeconfig /tmp/lab1-admin.yaml --apply-rbac +rm /tmp/lab1-admin.yaml +``` + +## Expected verification output + +``` + list pods yes + get machines.cluster.x-k8s.io yes # "no" on workload clusters (no CAPI CRDs), fine + delete machines.cluster.x-k8s.io no + patch machinedeployments.cluster.x-k8s.io no + get secrets no +``` + +Anything else means the RBAC is wrong. Don't use the agent until it matches. + +## Refresh / revoke + +- **Refresh:** re-run the same command without `--apply-rbac`. The default lifetime is 7 days + (`--duration`); the API server may cap it (`--service-account-max-token-expiration`). +- **Revoke everything:** `kubectl delete sa -n kube-system readonly-inspector` (as admin). That + invalidates every minted token; re-apply `rbac.yaml` to start over. + +## Making read-only the default for this repo (recommended) + +RBAC only protects you when the read-only kubeconfig is actually used. Your admin +`~/.kube/config` is still reachable from any shell. To make Claude Code sessions in this repo +default to the read-only files, add this to `.claude/settings.local.json` (personal, not committed): + +```json +{ + "env": { + "KUBECONFIG": "/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig:/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab1.kubeconfig" + } +} +``` + +Then plain `kubectl --context lab-mgmt-ro …` / `--context lab1-ro …` works everywhere in the +session, and mutations fail on RBAC even if something tries them. diff --git a/.claude/cluster-inspector/make-kubeconfig.sh b/.claude/cluster-inspector/make-kubeconfig.sh new file mode 100755 index 0000000..9d38f80 --- /dev/null +++ b/.claude/cluster-inspector/make-kubeconfig.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# Mint a read-only kubeconfig for the cluster-inspector agent. +# +# Usage: make-kubeconfig.sh [options] +# cluster name, e.g. lab-mgmt or lab1 -> kubeconfigs/.kubeconfig +# --admin-kubeconfig PATH admin kubeconfig used to apply RBAC / mint the token (default: $KUBECONFIG or ~/.kube/config) +# --context CTX context inside the admin kubeconfig +# --server URL override API server URL (k3s.yaml says https://127.0.0.1:6443) +# --duration DUR token lifetime (default 168h); re-run the script to refresh +# --apply-rbac kubectl apply rbac.yaml first (needed once per cluster) +set -euo pipefail +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; server=""; duration="168h"; apply=0 +while [[ $# -gt 0 ]]; do + case "$1" in + --admin-kubeconfig) admin="$2"; shift 2 ;; + --context) ctx="$2"; shift 2 ;; + --server) server="$2"; shift 2 ;; + --duration) duration="$2"; shift 2 ;; + --apply-rbac) apply=1; shift ;; + -h|--help) sed -n '2,10p' "$0"; exit 0 ;; + -*) echo "unknown option $1" >&2; exit 1 ;; + *) name="$1"; shift ;; + esac +done +[[ -n "$name" ]] || { sed -n '2,10p' "$0" >&2; exit 1; } + +k=(kubectl --kubeconfig "$admin") +[[ -n "$ctx" ]] && k+=(--context "$ctx") + +if (( apply )); then "${k[@]}" apply -f "$here/rbac.yaml"; fi + +token="$("${k[@]}" -n kube-system create token readonly-inspector --duration "$duration")" +[[ -n "$server" ]] || server="$("${k[@]}" config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.server}')" +ca="$("${k[@]}" config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')" +[[ -n "$ca" ]] || { echo "no certificate-authority-data in admin kubeconfig" >&2; exit 1; } + +mkdir -p "$here/kubeconfigs" +out="$here/kubeconfigs/$name.kubeconfig" +umask 077 +cat > "$out" </dev/null || true)" +done diff --git a/.claude/cluster-inspector/rbac.yaml b/.claude/cluster-inspector/rbac.yaml new file mode 100644 index 0000000..2bea717 --- /dev/null +++ b/.claude/cluster-inspector/rbac.yaml @@ -0,0 +1,43 @@ +# Read-only identity for the cluster-inspector agent. +# Apply to the management cluster and to every workload cluster the agent should inspect +# (CAPI rules are simply unused on workload clusters). +# Built-in `view` excludes Secrets; capi-readonly adds read access to CAPI CRDs and pod logs. +# Tokens are short-lived TokenRequest tokens minted by make-kubeconfig.sh — no Secret-based token. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: readonly-inspector + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: capi-readonly +rules: + - apiGroups: + - cluster.x-k8s.io + - infrastructure.cluster.x-k8s.io + - controlplane.cluster.x-k8s.io + - bootstrap.cluster.x-k8s.io + - ipam.cluster.x-k8s.io + - addons.cluster.x-k8s.io + - clusterctl.cluster.x-k8s.io + resources: ["*"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["pods/log"] + verbs: ["get", "list"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: readonly-inspector-view +roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: view } +subjects: [{ kind: ServiceAccount, name: readonly-inspector, namespace: kube-system }] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: readonly-inspector-capi +roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: capi-readonly } +subjects: [{ kind: ServiceAccount, name: readonly-inspector, namespace: kube-system }] diff --git a/.claude/commands/ship.md b/.claude/commands/ship.md new file mode 100644 index 0000000..ca9b33e --- /dev/null +++ b/.claude/commands/ship.md @@ -0,0 +1,23 @@ +--- +description: Commit current changes and push to the remote +--- + +Commit the current working-tree changes and push them. + +1. Run `git status` and `git diff` to see what will be committed. +2. Check the staged content for secrets: Proxmox token secrets, kubeconfig contents, anything + `.gitignore` would have caught by name but that was pasted into a doc. Stop and report if found. +3. Check the current branch (`git branch --show-current`). Docs, configs and history files may go + straight to `main`. If on `main` and the diff contains phase-3 Go code or something the user + called a feature, stop and tell the user to branch first (CLAUDE.md "Git"). +4. If an approved plan is in progress, make sure its execution log in `docs/plans-executions/` is + updated and its checklist ticked for the steps included in this commit. +5. Stage the changed files (`git add -A` is fine here). +6. Commit with a short message inferred from the diff, ending with the `Co-Authored-By:` trailer + for Claude. Pass the message via a HEREDOC. +7. Push with `git push`; if the branch has no upstream, `git push -u origin HEAD`. +8. If the branch isn't `main`, remind the user to open an MR with `tea pr create` rather than + opening one automatically. +9. Report the commit hash and push result. + +If there are no changes, stop and say so; do not create an empty commit. diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..8781956 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,14 @@ +{ + "permissions": { + "allow": [ + "Bash(git status:*)", + "Bash(git diff:*)", + "Bash(git log:*)", + "Bash(git branch:*)", + "Bash(git add:*)", + "Bash(date:*)", + "Bash(tea pr create:*)", + "Read(//private/tmp/**)" + ] + } +} diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..0772492 --- /dev/null +++ b/.gitignore @@ -0,0 +1,12 @@ +# credentials & generated kubeconfigs +.claude/cluster-inspector/kubeconfigs/ +.claude/settings.local.json +*.kubeconfig +kubeconfig* +*secret* +clusterctl.yaml +!configs/capi/clusterctl.yaml.example +.env +# packer / image-builder output +packer.log +*.iso diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..c580cf0 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,19 @@ +# Changelog + +## 2026-09-27 00:02 CEST — Read-only cluster-inspector credentials next to the agent + +- Moved inspector RBAC from `configs/rbac/` to `.claude/cluster-inspector/` with `make-kubeconfig.sh` + (applies RBAC once, mints a short-lived TokenRequest token, writes gitignored + `kubeconfigs/.kubeconfig`, runs `can-i` checks) and a setup README. +- Dropped the long-lived `service-account-token` Secret; tokens default to 7 days, refresh by re-running. +- `cluster-inspector` agent now reads only `.claude/cluster-inspector/kubeconfigs/*`; `.gitignore` + covers that dir and `.claude/settings.local.json`. +- Files: `.claude/cluster-inspector/`, `.claude/agents/cluster-inspector.md`, `CLAUDE.md`, `README.md`, + `.gitignore`, `docs/decisions.md`. + +## 2026-09-26 23:55 CEST — Project history conventions + +- Adopted plan / execution-log / prompt / review / changelog conventions from + egress-proxies-operator, plus new `docs/decisions.md` (back-filled) and `docs/sessions/`. +- Git flow: docs straight to `main`, phase-3 code via `feat/` branch + `tea` MR; `/ship` command. +- Files: `CLAUDE.md`, `docs/decisions.md`, `docs/sessions/`, `.claude/`. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..9f791a3 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,98 @@ +# CLAUDE.md + +Home-lab research project: cluster-autoscaler + Cluster API on Proxmox VE (phase 1), then a +KVM/libvirt provider (phase 2) and Kamaji hosted control planes / "poor man's KaaS" (phase 3). +Start with README.md; details live in docs/. + +## Owner context + +Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux, distributed systems). + +## Lab facts (don't re-derive) + +- PVE node `pv1-1`, API `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0). +- VM network: `vmbr0` is VLAN-aware, VMs in **VLAN 4** → always `bridge=vmbr0` + tag/`vlan: 4`, + never `vmbr0.4@vmbr0` (that is an `ip link` name, not a bridge). +- Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2. +- Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds). +- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55). Login only via cloud-init. +- Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`, + PVESDNUser on `/sdn/zones/localnetwork`. + +## Version rules + +- `docs/versions.md` is the source of truth. Any version change updates it in the same change. +- Before proposing any deployment or upgrade, verify versions are current (use the + `k8s-version-check` skill if available). Go module proxy / git ls-remote / Helm index.yaml work + even when the GitHub API is unavailable. +- CAPI core is pinned to **v1.12.x** because CAPMOX v0.9 is only tested on CAPI 1.11/1.12. This caps + workload Kubernetes at 1.35 and cluster-autoscaler at 1.35.x. Don't bump CAPI unless the CAPMOX + compatibility table allows it. +- Always pass explicit versions to `clusterctl init` / `helm install`; never rely on "latest". +- Check a provider's `metadata.yaml` contract and CAPI version (go.mod) before adding it to the + management cluster — all providers share one CAPI core (known skew: Kamaji CP provider v0.21.0 is + on CAPI 1.13). + +## Facts about providers: verify in source + +READMEs lag. For "does provider X support Y" read the API types and templates of the exact tagged +release (Go proxy zip: `https://proxy.golang.org//@v/.zip`). Delegate to the +`upstream-researcher` agent for anything non-trivial. + +## Safety + +- Never commit secrets: PVE token secret, kubeconfigs, `clusterctl.yaml` (see .gitignore). +- Cluster inspection is read-only via the `cluster-inspector` agent and read-only kubeconfigs + (`.claude/cluster-inspector/`, see its README). Mutations (`kubectl apply/delete/scale`, `qm`, `pveum`) + are proposed as commands for the owner to run unless explicitly asked to execute them. +- Don't delete Proxmox VMs/templates or CAPI objects on your own initiative. + +## Conventions + +- Docs in English, Markdown, Mermaid for diagrams (Gitea renders it). +- Troubleshooting findings go into the relevant doc's troubleshooting/fixes table: + symptom → root cause → fix. +- Example configs live in `configs/` as `*.example` or non-secret values files. + +## Project history + +Research repo: plans, conversations and decisions are kept in the repo, dated. Timestamps always +come from `date` (`+%Y-%m-%d-%H%M` for file names, `+%Y-%m-%d %H:%M %Z` for headings), never guessed. +Summaries only, never raw transcripts; mask secrets and check the staged diff before committing. + +| What | Where | When | +| --- | --- | --- | +| Plans | `docs/plans/YYYY-MM-DD-HHMM-.md` | every plan-mode plan | +| Execution logs | `docs/plans-executions/` | after each step of an approved plan | +| Session notes | `docs/sessions/YYYY-MM-DD-HHMM-.md` | research/design/debugging conversations without a plan | +| Decisions | `docs/decisions.md` (one table) | a direction chosen, rejected or reversed | +| Prompts | `docs/prompts/YYYY-MM-DD-HHMM-.md` | long crafted prompts/specs that kick off work (verbatim) | +| Reviews | `docs/reviews/YYYY-MM-DD-HHMM-.md` | review findings, most severe first, `- [ ]` ticked as processed | +| Changelog | `CHANGELOG.md`, newest on top | lab milestone, version change, design direction, new runbook/config — after the owner confirms it works | + +- **Plans:** header `# Plan: ` + `**Created:** <timestamp>`. Plan mode can only write to + `~/.claude/plans/`, so right after `ExitPlanMode` the first action is copying the plan into + `docs/plans/` and committing it on its own, before any work starts. +- **Execution logs:** a `- [ ] Step N — <title>` checklist on top (the authoritative status; a new + session resumes from this file alone), then one section per step in chronological order, + committed with that step's work. Light prose; exact commands as fenced blocks (`clusterctl`, + `helm`, `kubectl apply`, `qm`, Packer), each with one line on why it was run that way, including + commands the owner reports running on the lab and outputs that mattered (VM IDs, IPs, resolved + versions, errors). Call out deviations from the plan; end with "Worth noting". Problems also go + into the relevant fixes table, linked from the log. +- **Session notes:** write one unprompted when a conversation produced a finding worth keeping + (option compared or rejected, compatibility check, root cause, open question) that no plan + covers; mention it in the reply. Sections: Question, What was checked (sources, versions with + date checked), Findings, Open questions, Follow-ups. Extend the note if the topic continues. +- **Decisions:** never rewrite a row; a reversal is a new row and the old one becomes + `superseded by #N`. Design-doc suggestions are `proposed` until lab work confirms them. Link + the plan/session/doc with the reasoning. README "Key decisions so far" stays the short summary. + +## Git + +- Remote: Gitea `gitea.home.hrajfrisbee.cz/kacerr/cluster-autoscaler-with-capi`. +- Docs, configs and history files go straight to `main`. Phase-3 Go code or anything the owner + calls a feature: `feat/<slug>` branch, `git push -u`, MR via + `tea pr create --title … --description … --base main --head <branch>` (fallback: print + `…/compare/main...<branch>`). Never merge or delete branches from the CLI. +- Commit messages end with a `Co-Authored-By:` trailer for Claude. `/ship` commits and pushes. diff --git a/README.md b/README.md new file mode 100644 index 0000000..263fddf --- /dev/null +++ b/README.md @@ -0,0 +1,115 @@ +# cluster-autoscaler-with-capi + +Node autoscaling for Kubernetes clusters running as VMs on a home-lab hypervisor fleet +(Proxmox VE + vanilla KVM/libvirt), driven by **Cluster API** and **cluster-autoscaler** — +growing, in later phases, into a "poor man's Kubernetes-as-a-Service". + +> Status: **Phase 1 in progress** — VM template built and smoke-tested; management cluster +> and first workload cluster next. Versions verified 2026-09-24/26, see [docs/versions.md](docs/versions.md). + +## Goals + +- Workload clusters whose worker pool scales **0 → N → 0** on demand, with VMs created and + destroyed on the hypervisors automatically. +- Everything declarative: clusters are CAPI objects in a management cluster; no hand-made VMs. +- Learn Cluster API internals (contracts, providers, reconcile flow) by assembling, then extending it. +- End state: self-service tenant clusters (hosted control planes + autoscaled VM workers). + +## Non-goals (for now) + +- HA management cluster, production SLAs, multi-site. +- GitOps / fleet management (may come later on top). +- Writing our own autoscaler — cluster-autoscaler's `clusterapi` provider does the job. + +## Lab inventory + +| Hypervisor type | Management API | CAPI infrastructure provider | +| --- | --- | --- | +| Proxmox VE (e.g. `pv1-1`, 192.168.0.8) | PVE REST API :8006 | CAPMOX (phase 1) | +| Ubuntu 24.04 + KVM/libvirt | libvirt RPC (`qemu+ssh`/`qemu+tls`) | none mature upstream → phase 2 | + +VM network: `vmbr0` (VLAN-aware), VLAN 4, build host 192.168.4.181. + +## Architecture (phase 1) + +```mermaid +flowchart LR + subgraph MGMT[mgmt cluster: k3s VM] + CAPI[CAPI core + kubeadm bootstrap/CP] + CAPMOX[CAPMOX] + IPAM[IPAM in-cluster] + CA[cluster-autoscaler] + end + PVE[Proxmox VE API] + subgraph LAB1[workload cluster lab1] + CP[control plane VM + kube-vip VIP] + W[worker VMs - MachineDeployment 0..N] + end + CA -- scales MachineDeployment --> CAPI + CAPI --> CAPMOX + CAPMOX -- clone template, cloud-init ISO --> PVE + PVE --> CP + PVE --> W + CA -- watches pending pods via lab1-kubeconfig --> LAB1 +``` + +cluster-autoscaler runs in the management cluster in `kubeconfig-incluster` mode: it reads +pending pods/nodes from the workload cluster through the CAPI-generated kubeconfig secret and +changes `replicas` on MachineDeployments via the in-cluster management API. + +## Roadmap + +| Phase | Scope | Exit criteria | Doc | +| --- | --- | --- | --- | +| **1** | k3s mgmt cluster (manual) → CAPI + CAPMOX → workload cluster on Proxmox VMs → cluster-autoscaler | scale-up 1→3, scale-down to 0, scale-from-zero all work | [phase-1-proxmox-capi.md](docs/phase-1-proxmox-capi.md) | +| **2** | CAPI provider for vanilla KVM/libvirt hosts (evaluate CAPLV / KubeVirt / Metal3 / own provider) | worker MachineDeployment on Ubuntu KVM hosts, autoscaled | [phase-2-kvm-provider.md](docs/phase-2-kvm-provider.md) | +| **3** | Hosted control planes with Kamaji, tenant clusters on demand ("poor man's KaaS") | new tenant cluster from one manifest/ClusterClass, workers autoscaled | [phase-3-kamaji-kaas.md](docs/phase-3-kamaji-kaas.md) | + +Phases 2 and 3 are independent and can be done in either order. + +## Repository layout + +``` +. +├── README.md # this file +├── CLAUDE.md # context + rules for Claude Code sessions +├── CHANGELOG.md # milestones and significant changes, newest first +├── .claude/ +│ ├── agents/ # upstream-researcher, cluster-inspector (read-only) +│ ├── commands/ # /ship +│ └── cluster-inspector/ # RBAC, kubeconfig minting script, gitignored kubeconfigs/ +├── docs/ +│ ├── decisions.md # decision log: alternatives, why, status +│ ├── plans/ # plans (plan mode), one file per plan +│ ├── plans-executions/ # per-plan execution logs with exact commands +│ ├── sessions/ # research/design conversation summaries +│ ├── prompts/ # prompts/specs that kicked off work +│ ├── reviews/ # review findings +│ ├── versions.md # pinned versions + compatibility reasoning +│ ├── phase-1-proxmox-capi.md # runbook for phase 1 +│ ├── image-builder-proxmox.md # building the node VM template + fixes log +│ ├── phase-2-kvm-provider.md # KVM/libvirt provider options +│ └── phase-3-kamaji-kaas.md # hosted control planes / KaaS design notes +└── configs/ + ├── image-builder/ # packer var file for the Proxmox template + ├── capi/ # clusterctl config example + ├── cluster-autoscaler/ # Helm values + └── test/ # workloads for scale tests +``` + +Secrets (PVE token secret, kubeconfigs) never go into the repo — see `.gitignore`. + +## Key decisions so far + +Short summary; the full log with alternatives and status is [docs/decisions.md](docs/decisions.md). + +1. **CAPI v1.12.x, not the latest 1.14.x** — CAPMOX v0.9.x is only tested against CAPI 1.11/1.12. + This caps workload Kubernetes at **1.35** and cluster-autoscaler at **1.35.x**. +2. **Management cluster = k3s on a Proxmox VM**, not kind — CAPI controllers and the autoscaler + must run permanently. +3. **Node images via kubernetes-sigs/image-builder** (Ubuntu 24.04, Proxmox builder), overrides in + a `PACKER_VAR_FILES` JSON instead of `PACKER_FLAGS`. +4. **CNI installed by Helm** (Cilium, `ipam.mode=kubernetes`), not via ClusterResourceSet flavors, + to keep its version under our control. +5. **Scale-from-zero via capacity annotations** on the MachineDeployment — CAPMOX does not publish + `status.capacity` on `ProxmoxMachineTemplate`. diff --git a/configs/capi/clusterctl.yaml.example b/configs/capi/clusterctl.yaml.example new file mode 100644 index 0000000..8ea2818 --- /dev/null +++ b/configs/capi/clusterctl.yaml.example @@ -0,0 +1,26 @@ +# copy to ~/.cluster-api/clusterctl.yaml — never commit the real file +## controller +PROXMOX_URL: "https://192.168.0.8:8006" +PROXMOX_TOKEN: "capmox@pve!capi" +PROXMOX_SECRET: "<secret>" + +## workload defaults +PROXMOX_SOURCENODE: "pv1-1" # node holding the template +TEMPLATE_VMID: "<vmid of ubuntu-2404-kube-v1.35.9>" +ALLOWED_NODES: "[pv1-1]" # more nodes only with shared storage +VM_SSH_KEYS: "ssh-ed25519 AAAA..." + +## network (VLAN 4 on vmbr0 — add `vlan: 4` to the network device in ProxmoxMachineTemplates) +CONTROL_PLANE_ENDPOINT_IP: "<free IP in VLAN 4>" +NODE_IP_RANGES: "[<first>-<last>]" +GATEWAY: "<gw>" +IP_PREFIX: "24" +DNS_SERVERS: "[<dns>]" +BRIDGE: "vmbr0" + +## sizing defaults (workers overridden in lab1.yaml) +BOOT_VOLUME_DEVICE: "scsi0" +BOOT_VOLUME_SIZE: "40" +NUM_SOCKETS: "1" +NUM_CORES: "2" +MEMORY_MIB: "4096" diff --git a/configs/cluster-autoscaler/values.yaml b/configs/cluster-autoscaler/values.yaml new file mode 100644 index 0000000..ce94720 --- /dev/null +++ b/configs/cluster-autoscaler/values.yaml @@ -0,0 +1,17 @@ +# helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default -f values.yaml +# must be installed in the same namespace as the <cluster>-kubeconfig secret +cloudProvider: clusterapi +clusterAPIMode: kubeconfig-incluster # workload via kubeconfig, management in-cluster +clusterAPIKubeconfigSecret: lab1-kubeconfig +autoDiscovery: + clusterName: lab1 + namespace: default +image: + tag: v1.35.2 # chart default appVersion is 1.35.0 +extraArgs: + expander: least-waste + scale-down-unneeded-time: 5m + scale-down-delay-after-add: 5m + scale-down-utilization-threshold: "0.5" + balance-similar-node-groups: true + v: 4 diff --git a/configs/image-builder/proxmox-lab.json.example b/configs/image-builder/proxmox-lab.json.example new file mode 100644 index 0000000..896c82f --- /dev/null +++ b/configs/image-builder/proxmox-lab.json.example @@ -0,0 +1,8 @@ +{ + "disk_format": "raw", + "kubernetes_semver": "v1.35.9", + "kubernetes_series": "v1.35", + "kubernetes_deb_version": "1.35.9-1.1", + "iso_file": "local:iso/ubuntu-24.04.4-live-server-amd64.iso", + "iso_url": "" +} diff --git a/configs/test/inflate.yaml b/configs/test/inflate.yaml new file mode 100644 index 0000000..985dfba --- /dev/null +++ b/configs/test/inflate.yaml @@ -0,0 +1,18 @@ +# apply to the workload cluster; scale replicas to trigger scale-up (≈2 pods per 4 vCPU / 8 GiB worker) +apiVersion: apps/v1 +kind: Deployment +metadata: + name: inflate +spec: + replicas: 0 + selector: + matchLabels: { app: inflate } + template: + metadata: + labels: { app: inflate } + spec: + containers: + - name: pause + image: registry.k8s.io/pause:3.10 + resources: + requests: { cpu: "1500m", memory: "2Gi" } diff --git a/docs/decisions.md b/docs/decisions.md new file mode 100644 index 0000000..da4eecc --- /dev/null +++ b/docs/decisions.md @@ -0,0 +1,22 @@ +# Decisions + +Every direction chosen, rejected or reversed, newest at the bottom. Rows are never rewritten: +a reversal is a new row, and the old row's status becomes `superseded by #N`. Status is +`active`, `proposed` (not yet confirmed by lab work) or `superseded by #N`. + +Rows 1–9 were back-filled on 2026-09-26 from the README, `versions.md`, the phase docs and the +image-builder fixes log; their dates are when the decision was recorded, not necessarily made. + +| # | Date | Decision | Alternatives considered | Why | Status | +| --- | --- | --- | --- | --- | --- | +| 1 | 2026-09-24 | Pin CAPI core to 1.12.x ([versions.md](versions.md)) | latest CAPI 1.14.x | CAPMOX v0.9.x is only tested against CAPI 1.11/1.12; caps workload Kubernetes and cluster-autoscaler at 1.35 | active | +| 2 | 2026-09-24 | Management cluster is k3s on a Proxmox VM | kind on a workstation | CAPI controllers and the autoscaler must run permanently | active | +| 3 | 2026-09-24 | Node images built with kubernetes-sigs/image-builder (Ubuntu 24.04, Proxmox builder) | — | upstream-maintained CAPI images | active | +| 4 | 2026-09-24 | image-builder overrides go in a `PACKER_VAR_FILES` JSON | `PACKER_FLAGS` | flags were silently lost when `PACKER_FLAGS` was re-exported ([fixes log #4](image-builder-proxmox.md#fixes-log)) | active | +| 5 | 2026-09-24 | Cilium installed by Helm with `ipam.mode=kubernetes`, pod CIDR `10.244.0.0/16` | ClusterResourceSet CNI flavors; pod CIDR `192.168.0.0/16` | keeps the CNI version under our control; `192.168.0.0/16` overlaps the lab LAN | active | +| 6 | 2026-09-24 | Scale-from-zero via `capacity.cluster-autoscaler.kubernetes.io/*` annotations on the MachineDeployment | infrastructure provider publishing `status.capacity` | CAPMOX does not publish `status.capacity` on `ProxmoxMachineTemplate` | active | +| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | active | +| 8 | 2026-09-26 | Phase 2: spike CAPLV first; own provider if insufficient; KubeVirt as fallback ([phase-2](phase-2-kvm-provider.md#suggested-approach)) | CAPK + KubeVirt, Metal3, own provider from the start | CAPLV shares CAPI 1.12 with CAPMOX and fits plain libvirt hosts directly | proposed | +| 9 | 2026-09-26 | Phases 2 and 3 are independent and can be done in either order | strict sequence | no technical dependency between the KVM provider and hosted control planes | active | +| 10 | 2026-09-26 | Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog ([session](sessions/2026-09-26-2355-history-conventions.md)) | chat history only; raw transcripts committed | research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets | active | +| 11 | 2026-09-27 | Cluster inspection by agents only via read-only ServiceAccount kubeconfigs minted by `.claude/cluster-inspector/make-kubeconfig.sh` (short-lived TokenRequest tokens, gitignored) | admin kubeconfig + permission allowlist; long-lived SA token Secret; RBAC under `configs/` | RBAC (`view`, no Secrets) enforces read-only regardless of which kubectl verb an agent runs — also closes the `kubectl get secret` gap from the history-conventions session; short-lived tokens limit leak impact | active | diff --git a/docs/image-builder-proxmox.md b/docs/image-builder-proxmox.md new file mode 100644 index 0000000..2c49dc7 --- /dev/null +++ b/docs/image-builder-proxmox.md @@ -0,0 +1,85 @@ +# Node VM template with image-builder (Proxmox) + +Result: Proxmox template `ubuntu-2404-kube-v1.35.9` on node `pv1-1` — Ubuntu 24.04.4, containerd, +kubeadm/kubelet/kubectl v1.35.9, pre-pulled control-plane images, cloud-init, qemu-guest-agent. +Its VMID is `TEMPLATE_VMID` for CAPMOX. + +## Build host prerequisites (Ubuntu 24.04) + +```bash +sudo apt install -y make git curl unzip ca-certificates openssl openssh-client \ + python3 python3-venv python3-pip +python3 -m venv ~/ib-venv && source ~/ib-venv/bin/activate +pip install ansible-core==2.18.18 # avoids PEP 668 / --break-system-packages +git clone https://github.com/kubernetes-sigs/image-builder.git ~/image-builder +cd ~/image-builder/images/capi && git checkout v0.1.55 +make deps-proxmox # python/ansible/packer 1.9.5 checks + packer init +``` + +## Build + +Overrides live in a var file ([configs/image-builder/proxmox-lab.json.example](../configs/image-builder/proxmox-lab.json.example)), +passed via `PACKER_VAR_FILES` (last on the packer command line → highest precedence). + +```bash +export PROXMOX_URL="https://192.168.0.8:8006/api2/json" +export PROXMOX_USERNAME='capmox@pve!capi' +export PROXMOX_TOKEN="<secret>" +export PROXMOX_NODE="pv1-1" # PVE node *name* (pvesh get /nodes), not IP; template lands here +export PROXMOX_ISO_POOL="local" # storage with content 'iso' (pvesm status --content iso) +export PROXMOX_STORAGE_POOL="local-lvm" # storage with content 'images' (pvesm status --content images) +export PROXMOX_BRIDGE="vmbr0" +export PROXMOX_VLAN="4" +export PACKER_VAR_FILES=~/proxmox-lab.json +unset PACKER_FLAGS ISO_FILE + +export SSH_PASSWORD=labpass # optional: ssh builder@<vm> while provisioning runs +ON_ERROR_ASK=1 PACKER_LOG=1 PACKER_LOG_PATH=packer.log make build-proxmox-ubuntu-2404 +``` + +The build VM (VLAN 4) needs DHCP and must reach the Packer HTTP server on the build host +(ports 8000–9000) to fetch the autoinstall `user-data`. + +## What the build does + +1. `set-ssh-password` renders the autoinstall `user-data` with a hashed password for user `builder`. +2. Packer creates VM `packer-<uuid>` (2 vCPU, 2 GB, 20 GB), attaches the ISO, serves `http/24.04/`. +3. After `boot_wait` it types the GRUB boot command over VNC: `autoinstall ds=nocloud-net;s=http://<host>:<port>/24.04/`. +4. Subiquity installs Ubuntu and reboots; Packer waits for SSH (up to 2h), IP via guest agent. +5. Provisioners: wait for cloud-init → ansible `firstboot.yml` → reboot → ansible `node.yml` + (containerd, k8s binaries, images, sysctl, cloud-init config) → goss tests → lock `builder`, shutdown. +6. VM converted to template `ubuntu-2404-kube-v1.35.9`. + +Observe it in the PVE noVNC console; `DEBUG=1` steps through each Packer stage; `make -n` prints the +exact `packer build` command. Typical duration 20–40 min. + +## Fixes log + +| # | Symptom | Root cause | Fix | +| --- | --- | --- | --- | +| 1 | `POST .../storage/local/upload: write: broken pipe` | pveproxy dropped the 3.17 GiB upload mid-stream — missing `Datastore.AllocateTemplate` (token had only `PVEVMAdmin`) or no spool space in `/var/tmp` (exact cause not pinned down) | `PVEDatastoreAdmin` on `/storage`, check `df -h /var/tmp`; then avoid re-uploads with `iso_file` | +| 2 | `net0.bridge: invalid bridge ID 'vmbr0.4@vmbr0'` | `PROXMOX_BRIDGE` set to the `ip link` name of a VLAN sub-interface; PVE wants a bridge name + separate tag | `PROXMOX_BRIDGE=vmbr0`, `PROXMOX_VLAN=4` (→ `bridge=vmbr0,tag=4`) | +| 3 | `403 Permission check failed (/sdn/zones/localnetwork/vmbr0.4, SDN.Use)` | since PVE 8 every Linux bridge is an object in SDN zone `localnetwork`; `PVEVMAdmin` lacks `SDN.Use` | `PVESDNUser` on `/sdn/zones/localnetwork` | +| 4 | `unsupported format 'qcow2' at .../LvmThinPlugin.pm line 102` | image-builder defaults `disk_format` to qcow2; LVM-thin is raw-only. `--var disk_format=raw` in `PACKER_FLAGS` was lost when `PACKER_FLAGS` was re-exported | `"disk_format": "raw"` in the `PACKER_VAR_FILES` JSON | +| 5 | `one of iso_file, iso_url, or ... cd_content must be specified for boot_iso` | packer-plugin-proxmox v1.2.4 requires **exactly one** ISO source; `ubuntu-2404.json` sets `iso_file` from `$ISO_FILE` *and* a hardcoded `iso_url` → count 2. Misleading error; README tip is broken (worth an upstream issue) | set `iso_file` and `"iso_url": ""` in the var file, unset `ISO_FILE` | + +Notes carried into CAPMOX (phase 1): same ACL set, `vlan: 4` on the network device, no +`format: qcow2` in `ProxmoxMachineTemplate`s. + +## Smoke test of the template + +The template has no usable login (`builder` is locked) — access only via cloud-init (NoCloud, +which the PVE Cloud-Init drive provides): + +```bash +TPL=<template vmid>; ID=9001 +qm clone $TPL $ID --name smoke-k8s --full 1 --storage local-lvm +qm set $ID --ide2 local-lvm:cloudinit +qm set $ID --ciuser jan --sshkeys /root/jan.pub --ipconfig0 ip=dhcp +qm set $ID --net0 virtio,bridge=vmbr0,tag=4 +qm start $ID && qm guest cmd $ID network-get-interfaces +# inside: cloud-init status --long; kubeadm version -o short; crictl images +qm stop $ID && qm destroy $ID --purge +``` + +kubelet crash-looping before `kubeadm init/join` is expected. diff --git a/docs/phase-1-proxmox-capi.md b/docs/phase-1-proxmox-capi.md new file mode 100644 index 0000000..293ad00 --- /dev/null +++ b/docs/phase-1-proxmox-capi.md @@ -0,0 +1,173 @@ +# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler + +Goal: a manually installed k3s management cluster runs CAPI + CAPMOX and creates workload +cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero. +No custom code in this phase — only assembling upstream components. + +Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md). + +## Topology + +| Role | Where | Size | +| --- | --- | --- | +| Management cluster | 1 VM on Proxmox, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB | +| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM | +| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 8 GB RAM | + +## Checklist + +- [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges) +- [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested +- [ ] Step 2 — management cluster + `clusterctl init` +- [ ] Step 3 — workload cluster `lab1` +- [ ] Step 4 — CNI (Cilium) +- [ ] Step 5 — MachineDeployment autoscaling annotations +- [ ] Step 6 — cluster-autoscaler +- [ ] Step 7 — scale tests + +## Step 0 — Proxmox prep + +```bash +pveum user add capmox@pve +pveum user token add capmox@pve capi -privsep 0 # token inherits user ACLs + +pveum aclmod / -user capmox@pve -role PVEVMAdmin +pveum aclmod / -user capmox@pve -role PVEAuditor # Sys.Audit etc. +pveum aclmod /storage -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks +pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser # NIC on any local bridge (PVE 8+) +``` + +Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`. + +Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP +for the control plane endpoint in the same subnet. + +Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node +requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template. + +## Step 2 — management cluster + +```bash +curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.8+k3s1 sh -s - --disable traefik +mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config + +curl -L -o clusterctl https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-linux-amd64 +sudo install -m 0755 clusterctl /usr/local/bin/clusterctl +``` + +`~/.cluster-api/clusterctl.yaml` — start from [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example). + +```bash +clusterctl init \ + --core cluster-api:v1.12.11 \ + --bootstrap kubeadm:v1.12.11 \ + --control-plane kubeadm:v1.12.11 \ + --infrastructure proxmox:v0.9.1 \ + --ipam in-cluster:v1.1.1 +``` + +Always pin versions explicitly — without them clusterctl installs the newest core (1.14.x) and +breaks the tested CAPMOX combination. + +## Step 3 — workload cluster `lab1` + +```bash +clusterctl generate cluster lab1 \ + --infrastructure proxmox:v0.9.1 \ + --kubernetes-version v1.35.9 \ + --control-plane-machine-count 1 \ + --worker-machine-count 1 > lab1.yaml +``` + +Edit `lab1.yaml` before applying: + +1. **Pod CIDR** — `Cluster.spec.clusterNetwork.pods.cidrBlocks` defaults to `192.168.0.0/16`, which + overlaps the lab network → use e.g. `10.244.0.0/16`. +2. **kube-vip** — bump `ghcr.io/kube-vip/kube-vip:v0.7.1` → `v1.2.4`; keep `kube-vip-prepare.sh` + (handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes. +3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin). +4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device. +5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 8192`. +6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it. + +```bash +kubectl apply -f lab1.yaml +clusterctl describe cluster lab1 --show-conditions all +kubectl get proxmoxmachines,machines -w +clusterctl get kubeconfig lab1 > lab1.kubeconfig +``` + +Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses`). providerID is set by +the CAPMOX template as `proxmox://{{ ds.meta_data.instance_id }}` — don't touch it, the autoscaler +matches Nodes to Machines by it. + +## Step 4 — Cilium + +```bash +export KUBECONFIG=$PWD/lab1.kubeconfig +helm repo add cilium https://helm.cilium.io && helm repo update +helm install cilium cilium/cilium --version 1.20.2 -n kube-system --set ipam.mode=kubernetes +``` + +`ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default. + +## Step 5 — autoscaling annotations + +```yaml +metadata: + name: lab1-workers + annotations: + cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0" + cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5" + capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores + capacity.cluster-autoscaler.kubernetes.io/memory: "8Gi" + capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" + capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110" +``` + +Capacity annotations are required for scale-from-zero because CAPMOX does not publish +`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints` +annotations if pods target the group by selector/toleration. + +## Step 6 — cluster-autoscaler + +Install into the **same namespace as the `lab1-kubeconfig` secret** (here `default`), values in +[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml): + +```bash +helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update +helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \ + -f configs/cluster-autoscaler/values.yaml +``` + +The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5. + +## Step 7 — scale tests + +Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 2Gi each). + +- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time. +- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear. +- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting). +- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations. + +`kubectl describe pod <pending>` shows `TriggeredScaleUp` / `NotTriggerScaleUp` events — fastest diagnostic. + +## Troubleshooting + +| Symptom | Cause | Fix | +| --- | --- | --- | +| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values | +| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` | +| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` | +| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` | +| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods | +| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` | +| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range | +| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` | + +```bash +kubectl -n capmox-system logs deploy/capmox-controller-manager -f +kubectl -n capi-system logs deploy/capi-controller-manager -f +ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100' +``` diff --git a/docs/phase-2-kvm-provider.md b/docs/phase-2-kvm-provider.md new file mode 100644 index 0000000..119961d --- /dev/null +++ b/docs/phase-2-kvm-provider.md @@ -0,0 +1,62 @@ +# Phase 2 — CAPI provider for vanilla KVM/libvirt hosts + +Goal: autoscaled worker VMs on the Ubuntu 24.04 KVM/libvirt hypervisors, managed by the same +management cluster as phase 1. Status: **design / evaluation**. + +## Problem + +There is no mature, SIG-maintained CAPI infrastructure provider for plain libvirt. Proxmox does not +use libvirt (it drives QEMU via `qm`), so one libvirt provider cannot cover both hypervisor types. + +## Options + +| Option | How it works | Pros | Cons | +| --- | --- | --- | --- | +| **A. CAPLV** ([joshuagrisham/cluster-api-provider-libvirt](https://github.com/joshuagrisham/cluster-api-provider-libvirt), v0.2.1, CAPI v1.12.1) | talks libvirt RPC via pure-Go `digitalocean/go-libvirt` | same CAPI core as CAPMOX; direct fit for our hosts | young, single maintainer; feature gaps unknown (IPAM, placement, capacity) | +| **B. CAPK + KubeVirt** (CAPK v0.11.2 on CAPI v1.11.4, KubeVirt v1.9.0) | Ubuntu hosts become a Kubernetes "infra cluster" running KubeVirt; CAPK creates `VirtualMachine`s there | kubernetes-sigs project, mature, live migration, CDI for images | replaces plain libvirt management with a whole k8s + KubeVirt stack on the hypervisors; nested networking design (bridge/Multus) | +| **C. Metal3** (CAPM3 v1.14.1) + sushy-tools | VMs pre-defined in libvirt, exposed as "bare metal" via Redfish emulator, provisioned by Ironic | production-grade bare-metal flow | Ironic is heavy; VM pool is fixed → not real elasticity | +| **D. Own provider** | kubebuilder project, `LibvirtCluster/LibvirtMachine/LibvirtMachineTemplate` | full control, best learning value; can add multi-backend and `status.capacity` | we own maintenance | + +## Suggested approach + +1. **Spike A (≈ 1 week):** deploy CAPLV next to CAPMOX (both on CAPI 1.12), one MachineDeployment + on one Ubuntu host. Check: cloud-init/bootstrap handling, IP assignment (DHCP vs CAPI IPAM), + `providerID` format, delete/cleanup of volumes, behaviour under cluster-autoscaler. +2. If A is insufficient → **D**, reusing CAPLV's patterns, or contribute upstream. +3. Keep **B** as the alternative if we're willing to turn the Ubuntu hosts into a KubeVirt cluster + (it may pay off together with phase 3). + +## Own provider sketch (option D) + +Reconcile of `LibvirtMachine`: + +1. Wait for `Machine.spec.bootstrap.dataSecretName`. +2. Pick a host (placement: one MachineDeployment per host, or least-loaded by free memory). +3. Create a qcow2 overlay volume on the base image (backing file) in the host's storage pool. +4. Build a NoCloud ISO from the bootstrap secret (+ network-config from CAPI IPAM `IPAddressClaim`). +5. `virDomainDefineXML` + start; wait for IP (guest agent / DHCP leases). +6. Set `spec.providerID = libvirt://<host>/<domain-uuid>`, `status.addresses`, `ready`. +7. Finalizer: destroy domain, delete overlay + ISO. + +`LibvirtMachineTemplate.status.capacity` (cpu/memory) → scale-from-zero without annotations. +Connection: `qemu+ssh://` or `qemu+tls://` per host, credentials in a Secret. + +## Mixing providers in one workload cluster + +`Cluster.spec.infrastructureRef` points to **one** InfraCluster kind, and most providers' machine +reconcilers look up *their own* InfraCluster kind through it — a `ProxmoxMachine` in a cluster whose +InfraCluster is a `LibvirtCluster` won't reconcile. Choices: + +- **Separate workload clusters per provider** — simplest, no hacks. +- **Tolerant own provider** — our `LibvirtMachine` works regardless of the InfraCluster kind, so a + CAPMOX-based cluster can also get libvirt worker MachineDeployments. +- **Multi-backend provider** — one `HypervisorCluster` kind with Proxmox API + libvirt backends. + +cluster-autoscaler is indifferent: it scales any MachineDeployment carrying the min/max annotations, +whatever its infrastructure kind. The priority expander can prefer one hypervisor pool over another. + +## Exit criteria + +- [ ] Worker MachineDeployment on an Ubuntu KVM host joins a cluster managed from the phase-1 mgmt cluster. +- [ ] cluster-autoscaler scales it 0 → N → 0. +- [ ] VM and volumes fully cleaned up on scale-down. diff --git a/docs/phase-3-kamaji-kaas.md b/docs/phase-3-kamaji-kaas.md new file mode 100644 index 0000000..c4a7098 --- /dev/null +++ b/docs/phase-3-kamaji-kaas.md @@ -0,0 +1,97 @@ +# Phase 3 — Hosted control planes with Kamaji ("poor man's KaaS") + +Goal: create a tenant Kubernetes cluster from one manifest. Its control plane runs as pods in the +management cluster (Kamaji), and its workers are autoscaled Proxmox (later also KVM) VMs. +Status: **design notes**. + +## Why Kamaji + +With kubeadm control planes every tenant costs ≥ 1 VM (3 for HA) just for apiserver/etcd. Kamaji +runs each tenant control plane as a Deployment in the management cluster, backed by a shared +datastore. That cuts the control plane to a few hundred MB of RAM, gives it seconds to start, and +makes upgrades a rolling Deployment update. + +## Architecture + +```mermaid +flowchart LR + subgraph MGMT[management cluster - 3 nodes] + CAPI[CAPI core + CAPMOX] + KCP[Kamaji CP provider] + KAM[Kamaji controller] + DS[(kamaji-etcd datastore)] + LB[MetalLB pool on VLAN 4] + subgraph T1[ns tenant-a] + TCP1[TenantControlPlane pods<br/>apiserver, ctrl-mgr, scheduler] + CA1[cluster-autoscaler tenant-a] + end + end + PVE[Proxmox VE] + W1[tenant-a worker VMs] + KCP --> KAM --> TCP1 + TCP1 --> DS + LB -. VIP :6443 .- TCP1 + CAPI -- MachineDeployment --> PVE --> W1 + W1 -- kubelet join via VIP --> TCP1 + CA1 -- scales MD --> CAPI +``` + +CAPI object wiring per tenant: + +- `Cluster.spec.controlPlaneRef` → `KamajiControlPlane`, which creates the Kamaji `TenantControlPlane`. +- `Cluster.spec.infrastructureRef` → `ProxmoxCluster` with **`externalManagedControlPlane: true`**, + so the Kamaji provider patches the control plane endpoint. The field exists in CAPMOX v1alpha2. +- Workers: a regular `MachineDeployment` + `KubeadmConfigTemplate` + `ProxmoxMachineTemplate`, with + autoscaler annotations as in phase 1. +- Konnectivity handles apiserver → kubelet traffic, because the control plane isn't on the worker network. + +References: [Kamaji + Proxmox infra provider docs](https://kamaji.clastix.io/cluster-api/proxmox-infra-provider/), +CLASTIX Helm chart `capi-kamaji-proxmox` (0.0.4) as a wiring example. + +## Management cluster changes vs. phase 1 + +| Area | Phase 1 | Phase 3 | +| --- | --- | --- | +| Nodes | 1× k3s | 3 nodes: tenant control planes and their datastore live here | +| LoadBalancer | none needed (kube-vip in workload CP) | MetalLB (or kube-vip cloud provider) with an IP pool on VLAN 4, one VIP per tenant | +| Datastore | — | `kamaji-etcd` chart 0.18.1 (etcd 3.6.12); alternatives: PostgreSQL/MySQL via kine | +| Storage | — | CSI for etcd PVCs | + +k3s' built-in servicelb (klipper) binds host ports, so it can't hand out many `:6443` endpoints. Use MetalLB. + +## Autoscaling per tenant + +A cluster-autoscaler instance in `clusterapi` mode serves **one** workload cluster (one kubeconfig). +So each tenant gets its own CA release in its namespace in the management cluster +(`kubeconfig-incluster`, secret `<tenant>-kubeconfig`). Automating that is our first real Go code: + +- **tenant-operator** (small controller in this repo): watches `Cluster` objects labelled + `kaas.lab/autoscaling=enabled`, renders a per-tenant cluster-autoscaler Deployment + RBAC, and + cleans up on delete. The alternative is CAAPH (cluster-api-addon-provider-helm), but that installs + into the *workload* cluster, and we want CA in the management cluster. + +## Self-service UX + +- `ClusterClass` `lab-tenant` with variables: Kubernetes version, worker size, min/max workers, VLAN. +- A tenant cluster is one `Cluster` with `spec.topology` referencing the class. +- A namespace per tenant with RBAC limited to its own `Cluster`; the kubeconfig is handed out from the + `<tenant>-kubeconfig` secret. + +## Open questions and risks + +- **CAPI version skew:** Kamaji CP provider v0.21.0 is built against CAPI **v1.13.4**, while CAPMOX + v0.9.x is tested on **v1.12**. All providers share one CAPI core, so either find a Kamaji CP + provider release built on 1.12 or wait for CAPMOX to support 1.13+. Check each provider's + `metadata.yaml` contract before installing. +- **Kamaji OSS release model:** OSS publishes edge releases only (latest tag `edge-26.2.5`, Helm chart + `0.0.0+latest`). Stable versioned builds are commercial (CLASTIX), so pin by edge tag and image digest. +- Tenant Kubernetes version range is bounded by the Kamaji release and by our node images. We need + one image-builder template per supported minor. +- Backup and restore of the shared datastore (etcd snapshots) becomes critical: it holds every tenant. + +## Exit criteria + +- [ ] Tenant cluster created from a single `Cluster` + `ClusterClass`, control plane as pods in mgmt. +- [ ] Workers join via Konnectivity, and CNI works. +- [ ] Per-tenant cluster-autoscaler deployed automatically, scale 0 → N → 0 works. +- [ ] Deleting the `Cluster` removes the control plane, VMs, VIP and autoscaler. diff --git a/docs/plans-executions/.gitkeep b/docs/plans-executions/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/plans/.gitkeep b/docs/plans/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/prompts/.gitkeep b/docs/prompts/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/reviews/.gitkeep b/docs/reviews/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/sessions/2026-09-26-2355-history-conventions.md b/docs/sessions/2026-09-26-2355-history-conventions.md new file mode 100644 index 0000000..b298175 --- /dev/null +++ b/docs/sessions/2026-09-26-2355-history-conventions.md @@ -0,0 +1,51 @@ +# Session: which conventions to adopt from egress-proxies-operator + +**Created:** 2026-09-26 23:55 + +## Question + +> look into project in /Users/jan.novak/srv/go/egress-proxies-operator and suggest which rules i +> should copy over into this project +> +> this project is sort of research/experimentation project, so i want to keep history of +> conversations, decisions, plans, etc. + +## What was checked + +`egress-proxies-operator`: `CLAUDE.md`, `.claude/settings.json` + `settings.local.json`, +`.claude/commands/{ship,release}.md`, `.claude/agents/*`, `AGENTS.md`, and samples from +`docs/plans/`, `docs/plans-executions/`, `docs/prompts/`, `docs/reviews/`, `CHANGELOG.md`. + +## Findings + +Adopted, adapted to a docs/lab repo: + +- Plans in `docs/plans/` with the "commit the plan first, on its own" ordering rule. +- Execution logs in `docs/plans-executions/`, especially "exact commands with one line of + context, deviations called out", which suits lab runbook work better than Go code. +- `docs/prompts/`, `docs/reviews/`, top-inserted `CHANGELOG.md` with real `date` timestamps. +- Gitea + `tea pr create` MR flow, but only for phase-3 Go code; docs go straight to `main`. +- `/ship` command; a small read-only permission allowlist (`kubectl get/describe/logs`, + `clusterctl describe`, `helm list`, git read commands). + +Not adopted: Go and testing conventions, CI/CD, kubebuilder gotchas, `AGENTS.md`, the two +operator reviewer agents (candidates for phase 3), `/release` (no tags or images here), +`.junie/.codex/.agents` duplicates, and the broad `kubectl *` / `curl *` permissions. + +New, not in the operator repo, because decisions there are buried in plan context sections: + +- `docs/decisions.md`: one table with alternatives and status, back-filled with 9 rows. +- `docs/sessions/`: summaries of conversations that did not produce a plan (this file is the first). + +## Open questions + +- ~~Whether `kubectl get *` in the allowlist is too broad: `kubectl get secret -o yaml` would + print the `lab1-kubeconfig` secret without a prompt.~~ Resolved by + [decision #11](../decisions.md): a parallel session introduced read-only `cluster-inspector` + kubeconfigs, so the `kubectl` / `clusterctl` / `helm` allowlist entries were dropped from + `.claude/settings.json` again. + +## Follow-ups + +- [Decision #10](../decisions.md) +- `CLAUDE.md` sections "Project history" and "Git" diff --git a/docs/versions.md b/docs/versions.md new file mode 100644 index 0000000..e865fbf --- /dev/null +++ b/docs/versions.md @@ -0,0 +1,44 @@ +# Versions + +Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go module proxy, +`dl.k8s.io`, `registry.k8s.io`, upstream Helm indexes and git tags. Re-verify before every deploy. + +## Phase 1 — pinned + +| Component | Pinned | Latest available | Why pinned there | +| --- | --- | --- | --- | +| Cluster API core, kubeadm bootstrap + control plane, clusterctl | **v1.12.11** | v1.14.2 | CAPMOX v0.9 compatibility table lists CAPI v1.11 / v1.12 only; built against v1.12.11 | +| CAPMOX (ionos-cloud/cluster-api-provider-proxmox) | **v0.9.1** | v0.9.1 | API `v1alpha2`, contract `v1beta2` | +| IPAM provider in-cluster | **v1.1.1** | v1.1.1 | CAPMOX v0.9 requires ≥ 1.1.0 | +| Workload Kubernetes | **v1.35.9** | v1.37.1 | CAPI 1.12 supports workload clusters up to 1.35 | +| Management cluster (k3s) | **v1.35.8+k3s1** | v1.37.0+k3s1 | CAPI 1.12 supports management clusters 1.31–1.35 | +| cluster-autoscaler | **v1.35.2** | v1.36.1 | CA minor must match workload Kubernetes minor | +| cluster-autoscaler Helm chart | **9.59.0** | 9.59.0 | default appVersion 1.35.0 → override `image.tag` | +| Cilium | **1.20.2** | 1.20.2 | | +| kube-vip | **v1.2.4** | v1.2.4 | CAPMOX template still ships v0.7.1 | +| image-builder | **v0.1.55** | v0.1.55 | | +| Packer (installed by image-builder) | **1.9.5** | — | last MPL-licensed release, pinned by image-builder | +| packer-plugin-proxmox | **v1.2.4** | v1.2.4 | pulled by `packer init` (`>= 1.1.2`) | +| ansible-core (image-builder) | **2.18.18** | — | pinned by image-builder `hack/utils.sh` | +| Node OS | Ubuntu 24.04.4 | — | ISO sha256 `e907d92e…a138433` | + +### Upgrade path + +When CAPMOX adds CAPI 1.13/1.14 to its compatibility table: bump CAPI core → k3s mgmt to 1.36+ → +rebuild template with Kubernetes 1.36/1.37 → cluster-autoscaler to the matching minor. + +## Phase 2 / 3 — candidates (not pinned yet) + +| Component | Latest | Built against CAPI | Notes | +| --- | --- | --- | --- | +| CAPLV (joshuagrisham/cluster-api-provider-libvirt) | v0.2.1 (2026-02) | v1.12.1 | pure-Go `digitalocean/go-libvirt`, single maintainer | +| CAPK (cluster-api-provider-kubevirt) | v0.11.2 (2026-02) | v1.11.4 | needs KubeVirt on the hypervisors | +| KubeVirt | v1.9.0 (2026-07) | — | | +| CAPM3 (Metal3) | v1.14.1 (2026-09) | — | needs Ironic + BMO + sushy-tools for VMs | +| cluster-api-control-plane-provider-kamaji | v0.21.0 (2026-09-16) | **v1.13.4** | ⚠ newer CAPI than CAPMOX v0.9 is tested with | +| Kamaji (OSS) | `edge-26.2.5`, Helm chart `0.0.0+latest` | — | OSS ships edge releases only; last semver tag v1.0.0 (2024) | +| kamaji-etcd Helm chart | 0.18.1 (etcd 3.6.12) | — | datastore for hosted control planes | +| CLASTIX `capi-kamaji-proxmox` chart | 0.0.4 (2025-10) | — | reference for Kamaji + CAPMOX wiring | + +The CAPI version skew between CAPMOX (1.12) and the Kamaji CP provider (1.13) is the main open +compatibility question for phase 3 — all providers in one management cluster share one CAPI core.