Initial import: phase docs, configs, Claude conventions
Phase 1-3 runbooks and design notes, version pins, config templates, project-history conventions, cluster-inspector read-only credentials setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -0,0 +1,36 @@
|
|||||||
|
---
|
||||||
|
name: cluster-inspector
|
||||||
|
description: Read-only inspection and triage of the lab management cluster (CAPI, CAPMOX, IPAM, cluster-autoscaler) and workload clusters. Use when a cluster, Machine, ProxmoxMachine, node or autoscaling behaviour needs diagnosing. Never mutates anything.
|
||||||
|
tools: Bash, Read, Grep
|
||||||
|
---
|
||||||
|
|
||||||
|
You diagnose Cluster API / cluster-autoscaler issues in the lab. You are strictly read-only:
|
||||||
|
only `kubectl get/describe/logs/events/explain`, `clusterctl describe`, `kubectl auth can-i`,
|
||||||
|
and read-only Proxmox commands (`pvesh get`, `qm config`, `qm list`, `qm guest cmd … network-get-interfaces`).
|
||||||
|
Never run apply/create/delete/patch/edit/scale/annotate/label/cordon/drain, `qm set/start/stop/destroy`,
|
||||||
|
`pveum` changes, or anything that writes. If a fix needs a mutation, output the exact command
|
||||||
|
for the owner to run.
|
||||||
|
|
||||||
|
Credentials — use ONLY the read-only kubeconfigs, never ~/.kube/config or any admin kubeconfig:
|
||||||
|
- Files: `$(git rev-parse --show-toplevel)/.claude/cluster-inspector/kubeconfigs/<cluster>.kubeconfig`,
|
||||||
|
context `<cluster>-ro` (e.g. `lab-mgmt`, `lab1`). List the directory to see which clusters exist.
|
||||||
|
- Always pass it explicitly: `KUBECONFIG=<file> kubectl ...` (or `--kubeconfig <file>`).
|
||||||
|
- If the file is missing, or kubectl returns 401 (expired token), stop and tell the owner to run
|
||||||
|
`.claude/cluster-inspector/make-kubeconfig.sh <cluster> ...` (see .claude/cluster-inspector/README.md).
|
||||||
|
- Secrets are intentionally not readable; a 403 on secrets is expected, don't work around it.
|
||||||
|
|
||||||
|
Triage order:
|
||||||
|
1. `clusterctl describe cluster <name> --show-conditions all`
|
||||||
|
2. Objects + conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`,
|
||||||
|
`kubectl describe` the first non-Ready object.
|
||||||
|
3. Controller logs: `capi-system/capi-controller-manager`, `capmox-system/capmox-controller-manager`,
|
||||||
|
`capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses -A`).
|
||||||
|
4. Node side: Node vs Machine providerID match
|
||||||
|
(`kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID`), kubelet/cloud-init
|
||||||
|
status if SSH access is given.
|
||||||
|
5. Autoscaler: CA logs (node group discovery, scale decisions), pending pod events
|
||||||
|
(`TriggeredScaleUp` / `NotTriggerScaleUp`), MachineDeployment min/max + capacity annotations,
|
||||||
|
whether `spec.replicas` is being fought by a re-apply.
|
||||||
|
|
||||||
|
Known failure → fix mappings live in docs/phase-1-proxmox-capi.md and docs/image-builder-proxmox.md;
|
||||||
|
check them first. Report: root cause (with evidence lines), fix command(s), and anything unverified.
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
---
|
||||||
|
name: upstream-researcher
|
||||||
|
description: Answers "does upstream project X (CAPI providers, cluster-autoscaler, image-builder, Kamaji, KubeVirt, packer plugins) support Y / how does it behave / which versions are compatible" from the tagged release's source code, not READMEs. Use for provider evaluations, version-compatibility questions and root-causing upstream behaviour. Read-only.
|
||||||
|
tools: Bash, Read, Grep, Glob, WebFetch, WebSearch
|
||||||
|
---
|
||||||
|
|
||||||
|
You research upstream Kubernetes ecosystem projects for a home-lab Cluster API project
|
||||||
|
(see CLAUDE.md and docs/versions.md for the pinned stack). You never modify the repository.
|
||||||
|
|
||||||
|
Method:
|
||||||
|
1. Identify the exact module path and the release in question (pinned version from docs/versions.md,
|
||||||
|
or latest if asked). Latest: `curl -sS https://proxy.golang.org/<module>/@latest`;
|
||||||
|
all tags: `git ls-remote --tags <repo-url>` (works when the GitHub API is blocked).
|
||||||
|
2. Fetch the source of that exact tag into a temp dir, e.g.
|
||||||
|
`curl -sSL -o src.zip https://proxy.golang.org/<module>/@v/<ver>.zip && unzip -q src.zip -d src`
|
||||||
|
(non-Go projects: `git clone --depth 1 --branch <tag>`).
|
||||||
|
3. Answer from code: API types (`api/*/…_types.go`), webhooks, controllers, `templates/`,
|
||||||
|
`metadata.yaml` (contract), `go.mod` (CAPI / k8s versions), Makefile / hack scripts.
|
||||||
|
4. For compatibility: compare the CAPI version in go.mod and the contract in metadata.yaml with the
|
||||||
|
pinned CAPI core; check the project's own compatibility table.
|
||||||
|
|
||||||
|
Report format (concise, SRE audience):
|
||||||
|
- Answer in 1–3 sentences.
|
||||||
|
- Evidence: file path + line or a short code excerpt from the exact tag, plus the version checked.
|
||||||
|
- Caveats / version constraints relevant to our pinned stack.
|
||||||
|
- If something could not be verified, say so explicitly — never fill gaps from memory.
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# cluster-inspector: credentials setup
|
||||||
|
|
||||||
|
The `cluster-inspector` agent (`.claude/agents/cluster-inspector.md`) only ever talks to clusters
|
||||||
|
through the read-only kubeconfigs in `kubeconfigs/` here. They're gitignored and minted by
|
||||||
|
`make-kubeconfig.sh` from a short-lived ServiceAccount token.
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
| --- | --- |
|
||||||
|
| `rbac.yaml` | ServiceAccount `kube-system/readonly-inspector` + `view` + read on CAPI CRDs and pod logs (no Secrets) |
|
||||||
|
| `make-kubeconfig.sh` | applies RBAC (once), mints a TokenRequest token, writes `kubeconfigs/<name>.kubeconfig`, runs `can-i` checks |
|
||||||
|
| `kubeconfigs/` | output, **gitignored**; one file per cluster, context `<name>-ro` |
|
||||||
|
|
||||||
|
## 1. Management cluster (after phase 1 step 2)
|
||||||
|
|
||||||
|
k3s writes `server: https://127.0.0.1:6443`, so pass the VM address. k3s puts node IPs into the
|
||||||
|
API cert SANs, so the IP works; add `--tls-san` for a DNS name.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scp ubuntu@<mgmt-vm>:/etc/rancher/k3s/k3s.yaml /tmp/lab-mgmt-admin.yaml # or use your existing admin kubeconfig
|
||||||
|
.claude/cluster-inspector/make-kubeconfig.sh lab-mgmt \
|
||||||
|
--admin-kubeconfig /tmp/lab-mgmt-admin.yaml \
|
||||||
|
--server https://<mgmt-vm-ip>:6443 \
|
||||||
|
--apply-rbac
|
||||||
|
rm /tmp/lab-mgmt-admin.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2. Workload clusters (after phase 1 step 3)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
clusterctl get kubeconfig lab1 > /tmp/lab1-admin.yaml # run against the mgmt cluster (admin)
|
||||||
|
.claude/cluster-inspector/make-kubeconfig.sh lab1 --admin-kubeconfig /tmp/lab1-admin.yaml --apply-rbac
|
||||||
|
rm /tmp/lab1-admin.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Expected verification output
|
||||||
|
|
||||||
|
```
|
||||||
|
list pods yes
|
||||||
|
get machines.cluster.x-k8s.io yes # "no" on workload clusters (no CAPI CRDs), fine
|
||||||
|
delete machines.cluster.x-k8s.io no
|
||||||
|
patch machinedeployments.cluster.x-k8s.io no
|
||||||
|
get secrets no
|
||||||
|
```
|
||||||
|
|
||||||
|
Anything else means the RBAC is wrong. Don't use the agent until it matches.
|
||||||
|
|
||||||
|
## Refresh / revoke
|
||||||
|
|
||||||
|
- **Refresh:** re-run the same command without `--apply-rbac`. The default lifetime is 7 days
|
||||||
|
(`--duration`); the API server may cap it (`--service-account-max-token-expiration`).
|
||||||
|
- **Revoke everything:** `kubectl delete sa -n kube-system readonly-inspector` (as admin). That
|
||||||
|
invalidates every minted token; re-apply `rbac.yaml` to start over.
|
||||||
|
|
||||||
|
## Making read-only the default for this repo (recommended)
|
||||||
|
|
||||||
|
RBAC only protects you when the read-only kubeconfig is actually used. Your admin
|
||||||
|
`~/.kube/config` is still reachable from any shell. To make Claude Code sessions in this repo
|
||||||
|
default to the read-only files, add this to `.claude/settings.local.json` (personal, not committed):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"KUBECONFIG": "/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig:/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab1.kubeconfig"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Then plain `kubectl --context lab-mgmt-ro …` / `--context lab1-ro …` works everywhere in the
|
||||||
|
session, and mutations fail on RBAC even if something tries them.
|
||||||
Executable
+66
@@ -0,0 +1,66 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Mint a read-only kubeconfig for the cluster-inspector agent.
|
||||||
|
#
|
||||||
|
# Usage: make-kubeconfig.sh <name> [options]
|
||||||
|
# <name> cluster name, e.g. lab-mgmt or lab1 -> kubeconfigs/<name>.kubeconfig
|
||||||
|
# --admin-kubeconfig PATH admin kubeconfig used to apply RBAC / mint the token (default: $KUBECONFIG or ~/.kube/config)
|
||||||
|
# --context CTX context inside the admin kubeconfig
|
||||||
|
# --server URL override API server URL (k3s.yaml says https://127.0.0.1:6443)
|
||||||
|
# --duration DUR token lifetime (default 168h); re-run the script to refresh
|
||||||
|
# --apply-rbac kubectl apply rbac.yaml first (needed once per cluster)
|
||||||
|
set -euo pipefail
|
||||||
|
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; server=""; duration="168h"; apply=0
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--admin-kubeconfig) admin="$2"; shift 2 ;;
|
||||||
|
--context) ctx="$2"; shift 2 ;;
|
||||||
|
--server) server="$2"; shift 2 ;;
|
||||||
|
--duration) duration="$2"; shift 2 ;;
|
||||||
|
--apply-rbac) apply=1; shift ;;
|
||||||
|
-h|--help) sed -n '2,10p' "$0"; exit 0 ;;
|
||||||
|
-*) echo "unknown option $1" >&2; exit 1 ;;
|
||||||
|
*) name="$1"; shift ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
[[ -n "$name" ]] || { sed -n '2,10p' "$0" >&2; exit 1; }
|
||||||
|
|
||||||
|
k=(kubectl --kubeconfig "$admin")
|
||||||
|
[[ -n "$ctx" ]] && k+=(--context "$ctx")
|
||||||
|
|
||||||
|
if (( apply )); then "${k[@]}" apply -f "$here/rbac.yaml"; fi
|
||||||
|
|
||||||
|
token="$("${k[@]}" -n kube-system create token readonly-inspector --duration "$duration")"
|
||||||
|
[[ -n "$server" ]] || server="$("${k[@]}" config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.server}')"
|
||||||
|
ca="$("${k[@]}" config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')"
|
||||||
|
[[ -n "$ca" ]] || { echo "no certificate-authority-data in admin kubeconfig" >&2; exit 1; }
|
||||||
|
|
||||||
|
mkdir -p "$here/kubeconfigs"
|
||||||
|
out="$here/kubeconfigs/$name.kubeconfig"
|
||||||
|
umask 077
|
||||||
|
cat > "$out" <<EOF
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Config
|
||||||
|
clusters:
|
||||||
|
- name: $name
|
||||||
|
cluster:
|
||||||
|
server: $server
|
||||||
|
certificate-authority-data: $ca
|
||||||
|
users:
|
||||||
|
- name: $name-readonly-inspector
|
||||||
|
user:
|
||||||
|
token: $token
|
||||||
|
contexts:
|
||||||
|
- name: $name-ro
|
||||||
|
context:
|
||||||
|
cluster: $name
|
||||||
|
user: $name-readonly-inspector
|
||||||
|
current-context: $name-ro
|
||||||
|
EOF
|
||||||
|
echo "wrote $out (context $name-ro, server $server, token valid $duration)"
|
||||||
|
|
||||||
|
echo "verification (expect yes, yes, no, no, no; CAPI rows say no on workload clusters):"
|
||||||
|
for check in "list pods" "get machines.cluster.x-k8s.io" "delete machines.cluster.x-k8s.io" \
|
||||||
|
"patch machinedeployments.cluster.x-k8s.io" "get secrets"; do
|
||||||
|
printf ' %-45s %s\n' "$check" "$(KUBECONFIG="$out" kubectl auth can-i $check -A 2>/dev/null || true)"
|
||||||
|
done
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Read-only identity for the cluster-inspector agent.
|
||||||
|
# Apply to the management cluster and to every workload cluster the agent should inspect
|
||||||
|
# (CAPI rules are simply unused on workload clusters).
|
||||||
|
# Built-in `view` excludes Secrets; capi-readonly adds read access to CAPI CRDs and pod logs.
|
||||||
|
# Tokens are short-lived TokenRequest tokens minted by make-kubeconfig.sh — no Secret-based token.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ServiceAccount
|
||||||
|
metadata:
|
||||||
|
name: readonly-inspector
|
||||||
|
namespace: kube-system
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: ClusterRole
|
||||||
|
metadata:
|
||||||
|
name: capi-readonly
|
||||||
|
rules:
|
||||||
|
- apiGroups:
|
||||||
|
- cluster.x-k8s.io
|
||||||
|
- infrastructure.cluster.x-k8s.io
|
||||||
|
- controlplane.cluster.x-k8s.io
|
||||||
|
- bootstrap.cluster.x-k8s.io
|
||||||
|
- ipam.cluster.x-k8s.io
|
||||||
|
- addons.cluster.x-k8s.io
|
||||||
|
- clusterctl.cluster.x-k8s.io
|
||||||
|
resources: ["*"]
|
||||||
|
verbs: ["get", "list", "watch"]
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["pods/log"]
|
||||||
|
verbs: ["get", "list"]
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: ClusterRoleBinding
|
||||||
|
metadata:
|
||||||
|
name: readonly-inspector-view
|
||||||
|
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: view }
|
||||||
|
subjects: [{ kind: ServiceAccount, name: readonly-inspector, namespace: kube-system }]
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: ClusterRoleBinding
|
||||||
|
metadata:
|
||||||
|
name: readonly-inspector-capi
|
||||||
|
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: capi-readonly }
|
||||||
|
subjects: [{ kind: ServiceAccount, name: readonly-inspector, namespace: kube-system }]
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
---
|
||||||
|
description: Commit current changes and push to the remote
|
||||||
|
---
|
||||||
|
|
||||||
|
Commit the current working-tree changes and push them.
|
||||||
|
|
||||||
|
1. Run `git status` and `git diff` to see what will be committed.
|
||||||
|
2. Check the staged content for secrets: Proxmox token secrets, kubeconfig contents, anything
|
||||||
|
`.gitignore` would have caught by name but that was pasted into a doc. Stop and report if found.
|
||||||
|
3. Check the current branch (`git branch --show-current`). Docs, configs and history files may go
|
||||||
|
straight to `main`. If on `main` and the diff contains phase-3 Go code or something the user
|
||||||
|
called a feature, stop and tell the user to branch first (CLAUDE.md "Git").
|
||||||
|
4. If an approved plan is in progress, make sure its execution log in `docs/plans-executions/` is
|
||||||
|
updated and its checklist ticked for the steps included in this commit.
|
||||||
|
5. Stage the changed files (`git add -A` is fine here).
|
||||||
|
6. Commit with a short message inferred from the diff, ending with the `Co-Authored-By:` trailer
|
||||||
|
for Claude. Pass the message via a HEREDOC.
|
||||||
|
7. Push with `git push`; if the branch has no upstream, `git push -u origin HEAD`.
|
||||||
|
8. If the branch isn't `main`, remind the user to open an MR with `tea pr create` rather than
|
||||||
|
opening one automatically.
|
||||||
|
9. Report the commit hash and push result.
|
||||||
|
|
||||||
|
If there are no changes, stop and say so; do not create an empty commit.
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"permissions": {
|
||||||
|
"allow": [
|
||||||
|
"Bash(git status:*)",
|
||||||
|
"Bash(git diff:*)",
|
||||||
|
"Bash(git log:*)",
|
||||||
|
"Bash(git branch:*)",
|
||||||
|
"Bash(git add:*)",
|
||||||
|
"Bash(date:*)",
|
||||||
|
"Bash(tea pr create:*)",
|
||||||
|
"Read(//private/tmp/**)"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
+12
@@ -0,0 +1,12 @@
|
|||||||
|
# credentials & generated kubeconfigs
|
||||||
|
.claude/cluster-inspector/kubeconfigs/
|
||||||
|
.claude/settings.local.json
|
||||||
|
*.kubeconfig
|
||||||
|
kubeconfig*
|
||||||
|
*secret*
|
||||||
|
clusterctl.yaml
|
||||||
|
!configs/capi/clusterctl.yaml.example
|
||||||
|
.env
|
||||||
|
# packer / image-builder output
|
||||||
|
packer.log
|
||||||
|
*.iso
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 2026-09-27 00:02 CEST — Read-only cluster-inspector credentials next to the agent
|
||||||
|
|
||||||
|
- Moved inspector RBAC from `configs/rbac/` to `.claude/cluster-inspector/` with `make-kubeconfig.sh`
|
||||||
|
(applies RBAC once, mints a short-lived TokenRequest token, writes gitignored
|
||||||
|
`kubeconfigs/<name>.kubeconfig`, runs `can-i` checks) and a setup README.
|
||||||
|
- Dropped the long-lived `service-account-token` Secret; tokens default to 7 days, refresh by re-running.
|
||||||
|
- `cluster-inspector` agent now reads only `.claude/cluster-inspector/kubeconfigs/*`; `.gitignore`
|
||||||
|
covers that dir and `.claude/settings.local.json`.
|
||||||
|
- Files: `.claude/cluster-inspector/`, `.claude/agents/cluster-inspector.md`, `CLAUDE.md`, `README.md`,
|
||||||
|
`.gitignore`, `docs/decisions.md`.
|
||||||
|
|
||||||
|
## 2026-09-26 23:55 CEST — Project history conventions
|
||||||
|
|
||||||
|
- Adopted plan / execution-log / prompt / review / changelog conventions from
|
||||||
|
egress-proxies-operator, plus new `docs/decisions.md` (back-filled) and `docs/sessions/`.
|
||||||
|
- Git flow: docs straight to `main`, phase-3 code via `feat/` branch + `tea` MR; `/ship` command.
|
||||||
|
- Files: `CLAUDE.md`, `docs/decisions.md`, `docs/sessions/`, `.claude/`.
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
Home-lab research project: cluster-autoscaler + Cluster API on Proxmox VE (phase 1), then a
|
||||||
|
KVM/libvirt provider (phase 2) and Kamaji hosted control planes / "poor man's KaaS" (phase 3).
|
||||||
|
Start with README.md; details live in docs/.
|
||||||
|
|
||||||
|
## Owner context
|
||||||
|
|
||||||
|
Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux, distributed systems).
|
||||||
|
|
||||||
|
## Lab facts (don't re-derive)
|
||||||
|
|
||||||
|
- PVE node `pv1-1`, API `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0).
|
||||||
|
- VM network: `vmbr0` is VLAN-aware, VMs in **VLAN 4** → always `bridge=vmbr0` + tag/`vlan: 4`,
|
||||||
|
never `vmbr0.4@vmbr0` (that is an `ip link` name, not a bridge).
|
||||||
|
- Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2.
|
||||||
|
- Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds).
|
||||||
|
- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55). Login only via cloud-init.
|
||||||
|
- Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`,
|
||||||
|
PVESDNUser on `/sdn/zones/localnetwork`.
|
||||||
|
|
||||||
|
## Version rules
|
||||||
|
|
||||||
|
- `docs/versions.md` is the source of truth. Any version change updates it in the same change.
|
||||||
|
- Before proposing any deployment or upgrade, verify versions are current (use the
|
||||||
|
`k8s-version-check` skill if available). Go module proxy / git ls-remote / Helm index.yaml work
|
||||||
|
even when the GitHub API is unavailable.
|
||||||
|
- CAPI core is pinned to **v1.12.x** because CAPMOX v0.9 is only tested on CAPI 1.11/1.12. This caps
|
||||||
|
workload Kubernetes at 1.35 and cluster-autoscaler at 1.35.x. Don't bump CAPI unless the CAPMOX
|
||||||
|
compatibility table allows it.
|
||||||
|
- Always pass explicit versions to `clusterctl init` / `helm install`; never rely on "latest".
|
||||||
|
- Check a provider's `metadata.yaml` contract and CAPI version (go.mod) before adding it to the
|
||||||
|
management cluster — all providers share one CAPI core (known skew: Kamaji CP provider v0.21.0 is
|
||||||
|
on CAPI 1.13).
|
||||||
|
|
||||||
|
## Facts about providers: verify in source
|
||||||
|
|
||||||
|
READMEs lag. For "does provider X support Y" read the API types and templates of the exact tagged
|
||||||
|
release (Go proxy zip: `https://proxy.golang.org/<module>/@v/<ver>.zip`). Delegate to the
|
||||||
|
`upstream-researcher` agent for anything non-trivial.
|
||||||
|
|
||||||
|
## Safety
|
||||||
|
|
||||||
|
- Never commit secrets: PVE token secret, kubeconfigs, `clusterctl.yaml` (see .gitignore).
|
||||||
|
- Cluster inspection is read-only via the `cluster-inspector` agent and read-only kubeconfigs
|
||||||
|
(`.claude/cluster-inspector/`, see its README). Mutations (`kubectl apply/delete/scale`, `qm`, `pveum`)
|
||||||
|
are proposed as commands for the owner to run unless explicitly asked to execute them.
|
||||||
|
- Don't delete Proxmox VMs/templates or CAPI objects on your own initiative.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- Docs in English, Markdown, Mermaid for diagrams (Gitea renders it).
|
||||||
|
- Troubleshooting findings go into the relevant doc's troubleshooting/fixes table:
|
||||||
|
symptom → root cause → fix.
|
||||||
|
- Example configs live in `configs/` as `*.example` or non-secret values files.
|
||||||
|
|
||||||
|
## Project history
|
||||||
|
|
||||||
|
Research repo: plans, conversations and decisions are kept in the repo, dated. Timestamps always
|
||||||
|
come from `date` (`+%Y-%m-%d-%H%M` for file names, `+%Y-%m-%d %H:%M %Z` for headings), never guessed.
|
||||||
|
Summaries only, never raw transcripts; mask secrets and check the staged diff before committing.
|
||||||
|
|
||||||
|
| What | Where | When |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Plans | `docs/plans/YYYY-MM-DD-HHMM-<slug>.md` | every plan-mode plan |
|
||||||
|
| Execution logs | `docs/plans-executions/<plan's file name>` | after each step of an approved plan |
|
||||||
|
| Session notes | `docs/sessions/YYYY-MM-DD-HHMM-<slug>.md` | research/design/debugging conversations without a plan |
|
||||||
|
| Decisions | `docs/decisions.md` (one table) | a direction chosen, rejected or reversed |
|
||||||
|
| Prompts | `docs/prompts/YYYY-MM-DD-HHMM-<slug>.md` | long crafted prompts/specs that kick off work (verbatim) |
|
||||||
|
| Reviews | `docs/reviews/YYYY-MM-DD-HHMM-<slug>.md` | review findings, most severe first, `- [ ]` ticked as processed |
|
||||||
|
| Changelog | `CHANGELOG.md`, newest on top | lab milestone, version change, design direction, new runbook/config — after the owner confirms it works |
|
||||||
|
|
||||||
|
- **Plans:** header `# Plan: <title>` + `**Created:** <timestamp>`. Plan mode can only write to
|
||||||
|
`~/.claude/plans/`, so right after `ExitPlanMode` the first action is copying the plan into
|
||||||
|
`docs/plans/` and committing it on its own, before any work starts.
|
||||||
|
- **Execution logs:** a `- [ ] Step N — <title>` checklist on top (the authoritative status; a new
|
||||||
|
session resumes from this file alone), then one section per step in chronological order,
|
||||||
|
committed with that step's work. Light prose; exact commands as fenced blocks (`clusterctl`,
|
||||||
|
`helm`, `kubectl apply`, `qm`, Packer), each with one line on why it was run that way, including
|
||||||
|
commands the owner reports running on the lab and outputs that mattered (VM IDs, IPs, resolved
|
||||||
|
versions, errors). Call out deviations from the plan; end with "Worth noting". Problems also go
|
||||||
|
into the relevant fixes table, linked from the log.
|
||||||
|
- **Session notes:** write one unprompted when a conversation produced a finding worth keeping
|
||||||
|
(option compared or rejected, compatibility check, root cause, open question) that no plan
|
||||||
|
covers; mention it in the reply. Sections: Question, What was checked (sources, versions with
|
||||||
|
date checked), Findings, Open questions, Follow-ups. Extend the note if the topic continues.
|
||||||
|
- **Decisions:** never rewrite a row; a reversal is a new row and the old one becomes
|
||||||
|
`superseded by #N`. Design-doc suggestions are `proposed` until lab work confirms them. Link
|
||||||
|
the plan/session/doc with the reasoning. README "Key decisions so far" stays the short summary.
|
||||||
|
|
||||||
|
## Git
|
||||||
|
|
||||||
|
- Remote: Gitea `gitea.home.hrajfrisbee.cz/kacerr/cluster-autoscaler-with-capi`.
|
||||||
|
- Docs, configs and history files go straight to `main`. Phase-3 Go code or anything the owner
|
||||||
|
calls a feature: `feat/<slug>` branch, `git push -u`, MR via
|
||||||
|
`tea pr create --title … --description … --base main --head <branch>` (fallback: print
|
||||||
|
`…/compare/main...<branch>`). Never merge or delete branches from the CLI.
|
||||||
|
- Commit messages end with a `Co-Authored-By:` trailer for Claude. `/ship` commits and pushes.
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
# cluster-autoscaler-with-capi
|
||||||
|
|
||||||
|
Node autoscaling for Kubernetes clusters running as VMs on a home-lab hypervisor fleet
|
||||||
|
(Proxmox VE + vanilla KVM/libvirt), driven by **Cluster API** and **cluster-autoscaler** —
|
||||||
|
growing, in later phases, into a "poor man's Kubernetes-as-a-Service".
|
||||||
|
|
||||||
|
> Status: **Phase 1 in progress** — VM template built and smoke-tested; management cluster
|
||||||
|
> and first workload cluster next. Versions verified 2026-09-24/26, see [docs/versions.md](docs/versions.md).
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- Workload clusters whose worker pool scales **0 → N → 0** on demand, with VMs created and
|
||||||
|
destroyed on the hypervisors automatically.
|
||||||
|
- Everything declarative: clusters are CAPI objects in a management cluster; no hand-made VMs.
|
||||||
|
- Learn Cluster API internals (contracts, providers, reconcile flow) by assembling, then extending it.
|
||||||
|
- End state: self-service tenant clusters (hosted control planes + autoscaled VM workers).
|
||||||
|
|
||||||
|
## Non-goals (for now)
|
||||||
|
|
||||||
|
- HA management cluster, production SLAs, multi-site.
|
||||||
|
- GitOps / fleet management (may come later on top).
|
||||||
|
- Writing our own autoscaler — cluster-autoscaler's `clusterapi` provider does the job.
|
||||||
|
|
||||||
|
## Lab inventory
|
||||||
|
|
||||||
|
| Hypervisor type | Management API | CAPI infrastructure provider |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Proxmox VE (e.g. `pv1-1`, 192.168.0.8) | PVE REST API :8006 | CAPMOX (phase 1) |
|
||||||
|
| Ubuntu 24.04 + KVM/libvirt | libvirt RPC (`qemu+ssh`/`qemu+tls`) | none mature upstream → phase 2 |
|
||||||
|
|
||||||
|
VM network: `vmbr0` (VLAN-aware), VLAN 4, build host 192.168.4.181.
|
||||||
|
|
||||||
|
## Architecture (phase 1)
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
subgraph MGMT[mgmt cluster: k3s VM]
|
||||||
|
CAPI[CAPI core + kubeadm bootstrap/CP]
|
||||||
|
CAPMOX[CAPMOX]
|
||||||
|
IPAM[IPAM in-cluster]
|
||||||
|
CA[cluster-autoscaler]
|
||||||
|
end
|
||||||
|
PVE[Proxmox VE API]
|
||||||
|
subgraph LAB1[workload cluster lab1]
|
||||||
|
CP[control plane VM + kube-vip VIP]
|
||||||
|
W[worker VMs - MachineDeployment 0..N]
|
||||||
|
end
|
||||||
|
CA -- scales MachineDeployment --> CAPI
|
||||||
|
CAPI --> CAPMOX
|
||||||
|
CAPMOX -- clone template, cloud-init ISO --> PVE
|
||||||
|
PVE --> CP
|
||||||
|
PVE --> W
|
||||||
|
CA -- watches pending pods via lab1-kubeconfig --> LAB1
|
||||||
|
```
|
||||||
|
|
||||||
|
cluster-autoscaler runs in the management cluster in `kubeconfig-incluster` mode: it reads
|
||||||
|
pending pods/nodes from the workload cluster through the CAPI-generated kubeconfig secret and
|
||||||
|
changes `replicas` on MachineDeployments via the in-cluster management API.
|
||||||
|
|
||||||
|
## Roadmap
|
||||||
|
|
||||||
|
| Phase | Scope | Exit criteria | Doc |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| **1** | k3s mgmt cluster (manual) → CAPI + CAPMOX → workload cluster on Proxmox VMs → cluster-autoscaler | scale-up 1→3, scale-down to 0, scale-from-zero all work | [phase-1-proxmox-capi.md](docs/phase-1-proxmox-capi.md) |
|
||||||
|
| **2** | CAPI provider for vanilla KVM/libvirt hosts (evaluate CAPLV / KubeVirt / Metal3 / own provider) | worker MachineDeployment on Ubuntu KVM hosts, autoscaled | [phase-2-kvm-provider.md](docs/phase-2-kvm-provider.md) |
|
||||||
|
| **3** | Hosted control planes with Kamaji, tenant clusters on demand ("poor man's KaaS") | new tenant cluster from one manifest/ClusterClass, workers autoscaled | [phase-3-kamaji-kaas.md](docs/phase-3-kamaji-kaas.md) |
|
||||||
|
|
||||||
|
Phases 2 and 3 are independent and can be done in either order.
|
||||||
|
|
||||||
|
## Repository layout
|
||||||
|
|
||||||
|
```
|
||||||
|
.
|
||||||
|
├── README.md # this file
|
||||||
|
├── CLAUDE.md # context + rules for Claude Code sessions
|
||||||
|
├── CHANGELOG.md # milestones and significant changes, newest first
|
||||||
|
├── .claude/
|
||||||
|
│ ├── agents/ # upstream-researcher, cluster-inspector (read-only)
|
||||||
|
│ ├── commands/ # /ship
|
||||||
|
│ └── cluster-inspector/ # RBAC, kubeconfig minting script, gitignored kubeconfigs/
|
||||||
|
├── docs/
|
||||||
|
│ ├── decisions.md # decision log: alternatives, why, status
|
||||||
|
│ ├── plans/ # plans (plan mode), one file per plan
|
||||||
|
│ ├── plans-executions/ # per-plan execution logs with exact commands
|
||||||
|
│ ├── sessions/ # research/design conversation summaries
|
||||||
|
│ ├── prompts/ # prompts/specs that kicked off work
|
||||||
|
│ ├── reviews/ # review findings
|
||||||
|
│ ├── versions.md # pinned versions + compatibility reasoning
|
||||||
|
│ ├── phase-1-proxmox-capi.md # runbook for phase 1
|
||||||
|
│ ├── image-builder-proxmox.md # building the node VM template + fixes log
|
||||||
|
│ ├── phase-2-kvm-provider.md # KVM/libvirt provider options
|
||||||
|
│ └── phase-3-kamaji-kaas.md # hosted control planes / KaaS design notes
|
||||||
|
└── configs/
|
||||||
|
├── image-builder/ # packer var file for the Proxmox template
|
||||||
|
├── capi/ # clusterctl config example
|
||||||
|
├── cluster-autoscaler/ # Helm values
|
||||||
|
└── test/ # workloads for scale tests
|
||||||
|
```
|
||||||
|
|
||||||
|
Secrets (PVE token secret, kubeconfigs) never go into the repo — see `.gitignore`.
|
||||||
|
|
||||||
|
## Key decisions so far
|
||||||
|
|
||||||
|
Short summary; the full log with alternatives and status is [docs/decisions.md](docs/decisions.md).
|
||||||
|
|
||||||
|
1. **CAPI v1.12.x, not the latest 1.14.x** — CAPMOX v0.9.x is only tested against CAPI 1.11/1.12.
|
||||||
|
This caps workload Kubernetes at **1.35** and cluster-autoscaler at **1.35.x**.
|
||||||
|
2. **Management cluster = k3s on a Proxmox VM**, not kind — CAPI controllers and the autoscaler
|
||||||
|
must run permanently.
|
||||||
|
3. **Node images via kubernetes-sigs/image-builder** (Ubuntu 24.04, Proxmox builder), overrides in
|
||||||
|
a `PACKER_VAR_FILES` JSON instead of `PACKER_FLAGS`.
|
||||||
|
4. **CNI installed by Helm** (Cilium, `ipam.mode=kubernetes`), not via ClusterResourceSet flavors,
|
||||||
|
to keep its version under our control.
|
||||||
|
5. **Scale-from-zero via capacity annotations** on the MachineDeployment — CAPMOX does not publish
|
||||||
|
`status.capacity` on `ProxmoxMachineTemplate`.
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# copy to ~/.cluster-api/clusterctl.yaml — never commit the real file
|
||||||
|
## controller
|
||||||
|
PROXMOX_URL: "https://192.168.0.8:8006"
|
||||||
|
PROXMOX_TOKEN: "capmox@pve!capi"
|
||||||
|
PROXMOX_SECRET: "<secret>"
|
||||||
|
|
||||||
|
## workload defaults
|
||||||
|
PROXMOX_SOURCENODE: "pv1-1" # node holding the template
|
||||||
|
TEMPLATE_VMID: "<vmid of ubuntu-2404-kube-v1.35.9>"
|
||||||
|
ALLOWED_NODES: "[pv1-1]" # more nodes only with shared storage
|
||||||
|
VM_SSH_KEYS: "ssh-ed25519 AAAA..."
|
||||||
|
|
||||||
|
## network (VLAN 4 on vmbr0 — add `vlan: 4` to the network device in ProxmoxMachineTemplates)
|
||||||
|
CONTROL_PLANE_ENDPOINT_IP: "<free IP in VLAN 4>"
|
||||||
|
NODE_IP_RANGES: "[<first>-<last>]"
|
||||||
|
GATEWAY: "<gw>"
|
||||||
|
IP_PREFIX: "24"
|
||||||
|
DNS_SERVERS: "[<dns>]"
|
||||||
|
BRIDGE: "vmbr0"
|
||||||
|
|
||||||
|
## sizing defaults (workers overridden in lab1.yaml)
|
||||||
|
BOOT_VOLUME_DEVICE: "scsi0"
|
||||||
|
BOOT_VOLUME_SIZE: "40"
|
||||||
|
NUM_SOCKETS: "1"
|
||||||
|
NUM_CORES: "2"
|
||||||
|
MEMORY_MIB: "4096"
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default -f values.yaml
|
||||||
|
# must be installed in the same namespace as the <cluster>-kubeconfig secret
|
||||||
|
cloudProvider: clusterapi
|
||||||
|
clusterAPIMode: kubeconfig-incluster # workload via kubeconfig, management in-cluster
|
||||||
|
clusterAPIKubeconfigSecret: lab1-kubeconfig
|
||||||
|
autoDiscovery:
|
||||||
|
clusterName: lab1
|
||||||
|
namespace: default
|
||||||
|
image:
|
||||||
|
tag: v1.35.2 # chart default appVersion is 1.35.0
|
||||||
|
extraArgs:
|
||||||
|
expander: least-waste
|
||||||
|
scale-down-unneeded-time: 5m
|
||||||
|
scale-down-delay-after-add: 5m
|
||||||
|
scale-down-utilization-threshold: "0.5"
|
||||||
|
balance-similar-node-groups: true
|
||||||
|
v: 4
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
{
|
||||||
|
"disk_format": "raw",
|
||||||
|
"kubernetes_semver": "v1.35.9",
|
||||||
|
"kubernetes_series": "v1.35",
|
||||||
|
"kubernetes_deb_version": "1.35.9-1.1",
|
||||||
|
"iso_file": "local:iso/ubuntu-24.04.4-live-server-amd64.iso",
|
||||||
|
"iso_url": ""
|
||||||
|
}
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
# apply to the workload cluster; scale replicas to trigger scale-up (≈2 pods per 4 vCPU / 8 GiB worker)
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: inflate
|
||||||
|
spec:
|
||||||
|
replicas: 0
|
||||||
|
selector:
|
||||||
|
matchLabels: { app: inflate }
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels: { app: inflate }
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: pause
|
||||||
|
image: registry.k8s.io/pause:3.10
|
||||||
|
resources:
|
||||||
|
requests: { cpu: "1500m", memory: "2Gi" }
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Decisions
|
||||||
|
|
||||||
|
Every direction chosen, rejected or reversed, newest at the bottom. Rows are never rewritten:
|
||||||
|
a reversal is a new row, and the old row's status becomes `superseded by #N`. Status is
|
||||||
|
`active`, `proposed` (not yet confirmed by lab work) or `superseded by #N`.
|
||||||
|
|
||||||
|
Rows 1–9 were back-filled on 2026-09-26 from the README, `versions.md`, the phase docs and the
|
||||||
|
image-builder fixes log; their dates are when the decision was recorded, not necessarily made.
|
||||||
|
|
||||||
|
| # | Date | Decision | Alternatives considered | Why | Status |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| 1 | 2026-09-24 | Pin CAPI core to 1.12.x ([versions.md](versions.md)) | latest CAPI 1.14.x | CAPMOX v0.9.x is only tested against CAPI 1.11/1.12; caps workload Kubernetes and cluster-autoscaler at 1.35 | active |
|
||||||
|
| 2 | 2026-09-24 | Management cluster is k3s on a Proxmox VM | kind on a workstation | CAPI controllers and the autoscaler must run permanently | active |
|
||||||
|
| 3 | 2026-09-24 | Node images built with kubernetes-sigs/image-builder (Ubuntu 24.04, Proxmox builder) | — | upstream-maintained CAPI images | active |
|
||||||
|
| 4 | 2026-09-24 | image-builder overrides go in a `PACKER_VAR_FILES` JSON | `PACKER_FLAGS` | flags were silently lost when `PACKER_FLAGS` was re-exported ([fixes log #4](image-builder-proxmox.md#fixes-log)) | active |
|
||||||
|
| 5 | 2026-09-24 | Cilium installed by Helm with `ipam.mode=kubernetes`, pod CIDR `10.244.0.0/16` | ClusterResourceSet CNI flavors; pod CIDR `192.168.0.0/16` | keeps the CNI version under our control; `192.168.0.0/16` overlaps the lab LAN | active |
|
||||||
|
| 6 | 2026-09-24 | Scale-from-zero via `capacity.cluster-autoscaler.kubernetes.io/*` annotations on the MachineDeployment | infrastructure provider publishing `status.capacity` | CAPMOX does not publish `status.capacity` on `ProxmoxMachineTemplate` | active |
|
||||||
|
| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | active |
|
||||||
|
| 8 | 2026-09-26 | Phase 2: spike CAPLV first; own provider if insufficient; KubeVirt as fallback ([phase-2](phase-2-kvm-provider.md#suggested-approach)) | CAPK + KubeVirt, Metal3, own provider from the start | CAPLV shares CAPI 1.12 with CAPMOX and fits plain libvirt hosts directly | proposed |
|
||||||
|
| 9 | 2026-09-26 | Phases 2 and 3 are independent and can be done in either order | strict sequence | no technical dependency between the KVM provider and hosted control planes | active |
|
||||||
|
| 10 | 2026-09-26 | Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog ([session](sessions/2026-09-26-2355-history-conventions.md)) | chat history only; raw transcripts committed | research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets | active |
|
||||||
|
| 11 | 2026-09-27 | Cluster inspection by agents only via read-only ServiceAccount kubeconfigs minted by `.claude/cluster-inspector/make-kubeconfig.sh` (short-lived TokenRequest tokens, gitignored) | admin kubeconfig + permission allowlist; long-lived SA token Secret; RBAC under `configs/` | RBAC (`view`, no Secrets) enforces read-only regardless of which kubectl verb an agent runs — also closes the `kubectl get secret` gap from the history-conventions session; short-lived tokens limit leak impact | active |
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
# Node VM template with image-builder (Proxmox)
|
||||||
|
|
||||||
|
Result: Proxmox template `ubuntu-2404-kube-v1.35.9` on node `pv1-1` — Ubuntu 24.04.4, containerd,
|
||||||
|
kubeadm/kubelet/kubectl v1.35.9, pre-pulled control-plane images, cloud-init, qemu-guest-agent.
|
||||||
|
Its VMID is `TEMPLATE_VMID` for CAPMOX.
|
||||||
|
|
||||||
|
## Build host prerequisites (Ubuntu 24.04)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt install -y make git curl unzip ca-certificates openssl openssh-client \
|
||||||
|
python3 python3-venv python3-pip
|
||||||
|
python3 -m venv ~/ib-venv && source ~/ib-venv/bin/activate
|
||||||
|
pip install ansible-core==2.18.18 # avoids PEP 668 / --break-system-packages
|
||||||
|
git clone https://github.com/kubernetes-sigs/image-builder.git ~/image-builder
|
||||||
|
cd ~/image-builder/images/capi && git checkout v0.1.55
|
||||||
|
make deps-proxmox # python/ansible/packer 1.9.5 checks + packer init
|
||||||
|
```
|
||||||
|
|
||||||
|
## Build
|
||||||
|
|
||||||
|
Overrides live in a var file ([configs/image-builder/proxmox-lab.json.example](../configs/image-builder/proxmox-lab.json.example)),
|
||||||
|
passed via `PACKER_VAR_FILES` (last on the packer command line → highest precedence).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PROXMOX_URL="https://192.168.0.8:8006/api2/json"
|
||||||
|
export PROXMOX_USERNAME='capmox@pve!capi'
|
||||||
|
export PROXMOX_TOKEN="<secret>"
|
||||||
|
export PROXMOX_NODE="pv1-1" # PVE node *name* (pvesh get /nodes), not IP; template lands here
|
||||||
|
export PROXMOX_ISO_POOL="local" # storage with content 'iso' (pvesm status --content iso)
|
||||||
|
export PROXMOX_STORAGE_POOL="local-lvm" # storage with content 'images' (pvesm status --content images)
|
||||||
|
export PROXMOX_BRIDGE="vmbr0"
|
||||||
|
export PROXMOX_VLAN="4"
|
||||||
|
export PACKER_VAR_FILES=~/proxmox-lab.json
|
||||||
|
unset PACKER_FLAGS ISO_FILE
|
||||||
|
|
||||||
|
export SSH_PASSWORD=labpass # optional: ssh builder@<vm> while provisioning runs
|
||||||
|
ON_ERROR_ASK=1 PACKER_LOG=1 PACKER_LOG_PATH=packer.log make build-proxmox-ubuntu-2404
|
||||||
|
```
|
||||||
|
|
||||||
|
The build VM (VLAN 4) needs DHCP and must reach the Packer HTTP server on the build host
|
||||||
|
(ports 8000–9000) to fetch the autoinstall `user-data`.
|
||||||
|
|
||||||
|
## What the build does
|
||||||
|
|
||||||
|
1. `set-ssh-password` renders the autoinstall `user-data` with a hashed password for user `builder`.
|
||||||
|
2. Packer creates VM `packer-<uuid>` (2 vCPU, 2 GB, 20 GB), attaches the ISO, serves `http/24.04/`.
|
||||||
|
3. After `boot_wait` it types the GRUB boot command over VNC: `autoinstall ds=nocloud-net;s=http://<host>:<port>/24.04/`.
|
||||||
|
4. Subiquity installs Ubuntu and reboots; Packer waits for SSH (up to 2h), IP via guest agent.
|
||||||
|
5. Provisioners: wait for cloud-init → ansible `firstboot.yml` → reboot → ansible `node.yml`
|
||||||
|
(containerd, k8s binaries, images, sysctl, cloud-init config) → goss tests → lock `builder`, shutdown.
|
||||||
|
6. VM converted to template `ubuntu-2404-kube-v1.35.9`.
|
||||||
|
|
||||||
|
Observe it in the PVE noVNC console; `DEBUG=1` steps through each Packer stage; `make -n` prints the
|
||||||
|
exact `packer build` command. Typical duration 20–40 min.
|
||||||
|
|
||||||
|
## Fixes log
|
||||||
|
|
||||||
|
| # | Symptom | Root cause | Fix |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 1 | `POST .../storage/local/upload: write: broken pipe` | pveproxy dropped the 3.17 GiB upload mid-stream — missing `Datastore.AllocateTemplate` (token had only `PVEVMAdmin`) or no spool space in `/var/tmp` (exact cause not pinned down) | `PVEDatastoreAdmin` on `/storage`, check `df -h /var/tmp`; then avoid re-uploads with `iso_file` |
|
||||||
|
| 2 | `net0.bridge: invalid bridge ID 'vmbr0.4@vmbr0'` | `PROXMOX_BRIDGE` set to the `ip link` name of a VLAN sub-interface; PVE wants a bridge name + separate tag | `PROXMOX_BRIDGE=vmbr0`, `PROXMOX_VLAN=4` (→ `bridge=vmbr0,tag=4`) |
|
||||||
|
| 3 | `403 Permission check failed (/sdn/zones/localnetwork/vmbr0.4, SDN.Use)` | since PVE 8 every Linux bridge is an object in SDN zone `localnetwork`; `PVEVMAdmin` lacks `SDN.Use` | `PVESDNUser` on `/sdn/zones/localnetwork` |
|
||||||
|
| 4 | `unsupported format 'qcow2' at .../LvmThinPlugin.pm line 102` | image-builder defaults `disk_format` to qcow2; LVM-thin is raw-only. `--var disk_format=raw` in `PACKER_FLAGS` was lost when `PACKER_FLAGS` was re-exported | `"disk_format": "raw"` in the `PACKER_VAR_FILES` JSON |
|
||||||
|
| 5 | `one of iso_file, iso_url, or ... cd_content must be specified for boot_iso` | packer-plugin-proxmox v1.2.4 requires **exactly one** ISO source; `ubuntu-2404.json` sets `iso_file` from `$ISO_FILE` *and* a hardcoded `iso_url` → count 2. Misleading error; README tip is broken (worth an upstream issue) | set `iso_file` and `"iso_url": ""` in the var file, unset `ISO_FILE` |
|
||||||
|
|
||||||
|
Notes carried into CAPMOX (phase 1): same ACL set, `vlan: 4` on the network device, no
|
||||||
|
`format: qcow2` in `ProxmoxMachineTemplate`s.
|
||||||
|
|
||||||
|
## Smoke test of the template
|
||||||
|
|
||||||
|
The template has no usable login (`builder` is locked) — access only via cloud-init (NoCloud,
|
||||||
|
which the PVE Cloud-Init drive provides):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
TPL=<template vmid>; ID=9001
|
||||||
|
qm clone $TPL $ID --name smoke-k8s --full 1 --storage local-lvm
|
||||||
|
qm set $ID --ide2 local-lvm:cloudinit
|
||||||
|
qm set $ID --ciuser jan --sshkeys /root/jan.pub --ipconfig0 ip=dhcp
|
||||||
|
qm set $ID --net0 virtio,bridge=vmbr0,tag=4
|
||||||
|
qm start $ID && qm guest cmd $ID network-get-interfaces
|
||||||
|
# inside: cloud-init status --long; kubeadm version -o short; crictl images
|
||||||
|
qm stop $ID && qm destroy $ID --purge
|
||||||
|
```
|
||||||
|
|
||||||
|
kubelet crash-looping before `kubeadm init/join` is expected.
|
||||||
@@ -0,0 +1,173 @@
|
|||||||
|
# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler
|
||||||
|
|
||||||
|
Goal: a manually installed k3s management cluster runs CAPI + CAPMOX and creates workload
|
||||||
|
cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
|
||||||
|
No custom code in this phase — only assembling upstream components.
|
||||||
|
|
||||||
|
Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md).
|
||||||
|
|
||||||
|
## Topology
|
||||||
|
|
||||||
|
| Role | Where | Size |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Management cluster | 1 VM on Proxmox, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
|
||||||
|
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
|
||||||
|
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 8 GB RAM |
|
||||||
|
|
||||||
|
## Checklist
|
||||||
|
|
||||||
|
- [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
|
||||||
|
- [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
|
||||||
|
- [ ] Step 2 — management cluster + `clusterctl init`
|
||||||
|
- [ ] Step 3 — workload cluster `lab1`
|
||||||
|
- [ ] Step 4 — CNI (Cilium)
|
||||||
|
- [ ] Step 5 — MachineDeployment autoscaling annotations
|
||||||
|
- [ ] Step 6 — cluster-autoscaler
|
||||||
|
- [ ] Step 7 — scale tests
|
||||||
|
|
||||||
|
## Step 0 — Proxmox prep
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pveum user add capmox@pve
|
||||||
|
pveum user token add capmox@pve capi -privsep 0 # token inherits user ACLs
|
||||||
|
|
||||||
|
pveum aclmod / -user capmox@pve -role PVEVMAdmin
|
||||||
|
pveum aclmod / -user capmox@pve -role PVEAuditor # Sys.Audit etc.
|
||||||
|
pveum aclmod /storage -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks
|
||||||
|
pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser # NIC on any local bridge (PVE 8+)
|
||||||
|
```
|
||||||
|
|
||||||
|
Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`.
|
||||||
|
|
||||||
|
Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP
|
||||||
|
for the control plane endpoint in the same subnet.
|
||||||
|
|
||||||
|
Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node
|
||||||
|
requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template.
|
||||||
|
|
||||||
|
## Step 2 — management cluster
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.8+k3s1 sh -s - --disable traefik
|
||||||
|
mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config
|
||||||
|
|
||||||
|
curl -L -o clusterctl https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-linux-amd64
|
||||||
|
sudo install -m 0755 clusterctl /usr/local/bin/clusterctl
|
||||||
|
```
|
||||||
|
|
||||||
|
`~/.cluster-api/clusterctl.yaml` — start from [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
clusterctl init \
|
||||||
|
--core cluster-api:v1.12.11 \
|
||||||
|
--bootstrap kubeadm:v1.12.11 \
|
||||||
|
--control-plane kubeadm:v1.12.11 \
|
||||||
|
--infrastructure proxmox:v0.9.1 \
|
||||||
|
--ipam in-cluster:v1.1.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Always pin versions explicitly — without them clusterctl installs the newest core (1.14.x) and
|
||||||
|
breaks the tested CAPMOX combination.
|
||||||
|
|
||||||
|
## Step 3 — workload cluster `lab1`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
clusterctl generate cluster lab1 \
|
||||||
|
--infrastructure proxmox:v0.9.1 \
|
||||||
|
--kubernetes-version v1.35.9 \
|
||||||
|
--control-plane-machine-count 1 \
|
||||||
|
--worker-machine-count 1 > lab1.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Edit `lab1.yaml` before applying:
|
||||||
|
|
||||||
|
1. **Pod CIDR** — `Cluster.spec.clusterNetwork.pods.cidrBlocks` defaults to `192.168.0.0/16`, which
|
||||||
|
overlaps the lab network → use e.g. `10.244.0.0/16`.
|
||||||
|
2. **kube-vip** — bump `ghcr.io/kube-vip/kube-vip:v0.7.1` → `v1.2.4`; keep `kube-vip-prepare.sh`
|
||||||
|
(handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
|
||||||
|
3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
|
||||||
|
4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
|
||||||
|
5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 8192`.
|
||||||
|
6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl apply -f lab1.yaml
|
||||||
|
clusterctl describe cluster lab1 --show-conditions all
|
||||||
|
kubectl get proxmoxmachines,machines -w
|
||||||
|
clusterctl get kubeconfig lab1 > lab1.kubeconfig
|
||||||
|
```
|
||||||
|
|
||||||
|
Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses`). providerID is set by
|
||||||
|
the CAPMOX template as `proxmox://{{ ds.meta_data.instance_id }}` — don't touch it, the autoscaler
|
||||||
|
matches Nodes to Machines by it.
|
||||||
|
|
||||||
|
## Step 4 — Cilium
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export KUBECONFIG=$PWD/lab1.kubeconfig
|
||||||
|
helm repo add cilium https://helm.cilium.io && helm repo update
|
||||||
|
helm install cilium cilium/cilium --version 1.20.2 -n kube-system --set ipam.mode=kubernetes
|
||||||
|
```
|
||||||
|
|
||||||
|
`ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default.
|
||||||
|
|
||||||
|
## Step 5 — autoscaling annotations
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
metadata:
|
||||||
|
name: lab1-workers
|
||||||
|
annotations:
|
||||||
|
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
|
||||||
|
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
|
||||||
|
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores
|
||||||
|
capacity.cluster-autoscaler.kubernetes.io/memory: "8Gi"
|
||||||
|
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
|
||||||
|
capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
|
||||||
|
```
|
||||||
|
|
||||||
|
Capacity annotations are required for scale-from-zero because CAPMOX does not publish
|
||||||
|
`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints`
|
||||||
|
annotations if pods target the group by selector/toleration.
|
||||||
|
|
||||||
|
## Step 6 — cluster-autoscaler
|
||||||
|
|
||||||
|
Install into the **same namespace as the `lab1-kubeconfig` secret** (here `default`), values in
|
||||||
|
[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update
|
||||||
|
helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \
|
||||||
|
-f configs/cluster-autoscaler/values.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5.
|
||||||
|
|
||||||
|
## Step 7 — scale tests
|
||||||
|
|
||||||
|
Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 2Gi each).
|
||||||
|
|
||||||
|
- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
|
||||||
|
- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear.
|
||||||
|
- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
|
||||||
|
- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.
|
||||||
|
|
||||||
|
`kubectl describe pod <pending>` shows `TriggeredScaleUp` / `NotTriggerScaleUp` events — fastest diagnostic.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
| Symptom | Cause | Fix |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
|
||||||
|
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
|
||||||
|
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
|
||||||
|
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
|
||||||
|
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
|
||||||
|
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
|
||||||
|
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
|
||||||
|
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n capmox-system logs deploy/capmox-controller-manager -f
|
||||||
|
kubectl -n capi-system logs deploy/capi-controller-manager -f
|
||||||
|
ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100'
|
||||||
|
```
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# Phase 2 — CAPI provider for vanilla KVM/libvirt hosts
|
||||||
|
|
||||||
|
Goal: autoscaled worker VMs on the Ubuntu 24.04 KVM/libvirt hypervisors, managed by the same
|
||||||
|
management cluster as phase 1. Status: **design / evaluation**.
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
There is no mature, SIG-maintained CAPI infrastructure provider for plain libvirt. Proxmox does not
|
||||||
|
use libvirt (it drives QEMU via `qm`), so one libvirt provider cannot cover both hypervisor types.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
| Option | How it works | Pros | Cons |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| **A. CAPLV** ([joshuagrisham/cluster-api-provider-libvirt](https://github.com/joshuagrisham/cluster-api-provider-libvirt), v0.2.1, CAPI v1.12.1) | talks libvirt RPC via pure-Go `digitalocean/go-libvirt` | same CAPI core as CAPMOX; direct fit for our hosts | young, single maintainer; feature gaps unknown (IPAM, placement, capacity) |
|
||||||
|
| **B. CAPK + KubeVirt** (CAPK v0.11.2 on CAPI v1.11.4, KubeVirt v1.9.0) | Ubuntu hosts become a Kubernetes "infra cluster" running KubeVirt; CAPK creates `VirtualMachine`s there | kubernetes-sigs project, mature, live migration, CDI for images | replaces plain libvirt management with a whole k8s + KubeVirt stack on the hypervisors; nested networking design (bridge/Multus) |
|
||||||
|
| **C. Metal3** (CAPM3 v1.14.1) + sushy-tools | VMs pre-defined in libvirt, exposed as "bare metal" via Redfish emulator, provisioned by Ironic | production-grade bare-metal flow | Ironic is heavy; VM pool is fixed → not real elasticity |
|
||||||
|
| **D. Own provider** | kubebuilder project, `LibvirtCluster/LibvirtMachine/LibvirtMachineTemplate` | full control, best learning value; can add multi-backend and `status.capacity` | we own maintenance |
|
||||||
|
|
||||||
|
## Suggested approach
|
||||||
|
|
||||||
|
1. **Spike A (≈ 1 week):** deploy CAPLV next to CAPMOX (both on CAPI 1.12), one MachineDeployment
|
||||||
|
on one Ubuntu host. Check: cloud-init/bootstrap handling, IP assignment (DHCP vs CAPI IPAM),
|
||||||
|
`providerID` format, delete/cleanup of volumes, behaviour under cluster-autoscaler.
|
||||||
|
2. If A is insufficient → **D**, reusing CAPLV's patterns, or contribute upstream.
|
||||||
|
3. Keep **B** as the alternative if we're willing to turn the Ubuntu hosts into a KubeVirt cluster
|
||||||
|
(it may pay off together with phase 3).
|
||||||
|
|
||||||
|
## Own provider sketch (option D)
|
||||||
|
|
||||||
|
Reconcile of `LibvirtMachine`:
|
||||||
|
|
||||||
|
1. Wait for `Machine.spec.bootstrap.dataSecretName`.
|
||||||
|
2. Pick a host (placement: one MachineDeployment per host, or least-loaded by free memory).
|
||||||
|
3. Create a qcow2 overlay volume on the base image (backing file) in the host's storage pool.
|
||||||
|
4. Build a NoCloud ISO from the bootstrap secret (+ network-config from CAPI IPAM `IPAddressClaim`).
|
||||||
|
5. `virDomainDefineXML` + start; wait for IP (guest agent / DHCP leases).
|
||||||
|
6. Set `spec.providerID = libvirt://<host>/<domain-uuid>`, `status.addresses`, `ready`.
|
||||||
|
7. Finalizer: destroy domain, delete overlay + ISO.
|
||||||
|
|
||||||
|
`LibvirtMachineTemplate.status.capacity` (cpu/memory) → scale-from-zero without annotations.
|
||||||
|
Connection: `qemu+ssh://` or `qemu+tls://` per host, credentials in a Secret.
|
||||||
|
|
||||||
|
## Mixing providers in one workload cluster
|
||||||
|
|
||||||
|
`Cluster.spec.infrastructureRef` points to **one** InfraCluster kind, and most providers' machine
|
||||||
|
reconcilers look up *their own* InfraCluster kind through it — a `ProxmoxMachine` in a cluster whose
|
||||||
|
InfraCluster is a `LibvirtCluster` won't reconcile. Choices:
|
||||||
|
|
||||||
|
- **Separate workload clusters per provider** — simplest, no hacks.
|
||||||
|
- **Tolerant own provider** — our `LibvirtMachine` works regardless of the InfraCluster kind, so a
|
||||||
|
CAPMOX-based cluster can also get libvirt worker MachineDeployments.
|
||||||
|
- **Multi-backend provider** — one `HypervisorCluster` kind with Proxmox API + libvirt backends.
|
||||||
|
|
||||||
|
cluster-autoscaler is indifferent: it scales any MachineDeployment carrying the min/max annotations,
|
||||||
|
whatever its infrastructure kind. The priority expander can prefer one hypervisor pool over another.
|
||||||
|
|
||||||
|
## Exit criteria
|
||||||
|
|
||||||
|
- [ ] Worker MachineDeployment on an Ubuntu KVM host joins a cluster managed from the phase-1 mgmt cluster.
|
||||||
|
- [ ] cluster-autoscaler scales it 0 → N → 0.
|
||||||
|
- [ ] VM and volumes fully cleaned up on scale-down.
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
# Phase 3 — Hosted control planes with Kamaji ("poor man's KaaS")
|
||||||
|
|
||||||
|
Goal: create a tenant Kubernetes cluster from one manifest. Its control plane runs as pods in the
|
||||||
|
management cluster (Kamaji), and its workers are autoscaled Proxmox (later also KVM) VMs.
|
||||||
|
Status: **design notes**.
|
||||||
|
|
||||||
|
## Why Kamaji
|
||||||
|
|
||||||
|
With kubeadm control planes every tenant costs ≥ 1 VM (3 for HA) just for apiserver/etcd. Kamaji
|
||||||
|
runs each tenant control plane as a Deployment in the management cluster, backed by a shared
|
||||||
|
datastore. That cuts the control plane to a few hundred MB of RAM, gives it seconds to start, and
|
||||||
|
makes upgrades a rolling Deployment update.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
subgraph MGMT[management cluster - 3 nodes]
|
||||||
|
CAPI[CAPI core + CAPMOX]
|
||||||
|
KCP[Kamaji CP provider]
|
||||||
|
KAM[Kamaji controller]
|
||||||
|
DS[(kamaji-etcd datastore)]
|
||||||
|
LB[MetalLB pool on VLAN 4]
|
||||||
|
subgraph T1[ns tenant-a]
|
||||||
|
TCP1[TenantControlPlane pods<br/>apiserver, ctrl-mgr, scheduler]
|
||||||
|
CA1[cluster-autoscaler tenant-a]
|
||||||
|
end
|
||||||
|
end
|
||||||
|
PVE[Proxmox VE]
|
||||||
|
W1[tenant-a worker VMs]
|
||||||
|
KCP --> KAM --> TCP1
|
||||||
|
TCP1 --> DS
|
||||||
|
LB -. VIP :6443 .- TCP1
|
||||||
|
CAPI -- MachineDeployment --> PVE --> W1
|
||||||
|
W1 -- kubelet join via VIP --> TCP1
|
||||||
|
CA1 -- scales MD --> CAPI
|
||||||
|
```
|
||||||
|
|
||||||
|
CAPI object wiring per tenant:
|
||||||
|
|
||||||
|
- `Cluster.spec.controlPlaneRef` → `KamajiControlPlane`, which creates the Kamaji `TenantControlPlane`.
|
||||||
|
- `Cluster.spec.infrastructureRef` → `ProxmoxCluster` with **`externalManagedControlPlane: true`**,
|
||||||
|
so the Kamaji provider patches the control plane endpoint. The field exists in CAPMOX v1alpha2.
|
||||||
|
- Workers: a regular `MachineDeployment` + `KubeadmConfigTemplate` + `ProxmoxMachineTemplate`, with
|
||||||
|
autoscaler annotations as in phase 1.
|
||||||
|
- Konnectivity handles apiserver → kubelet traffic, because the control plane isn't on the worker network.
|
||||||
|
|
||||||
|
References: [Kamaji + Proxmox infra provider docs](https://kamaji.clastix.io/cluster-api/proxmox-infra-provider/),
|
||||||
|
CLASTIX Helm chart `capi-kamaji-proxmox` (0.0.4) as a wiring example.
|
||||||
|
|
||||||
|
## Management cluster changes vs. phase 1
|
||||||
|
|
||||||
|
| Area | Phase 1 | Phase 3 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Nodes | 1× k3s | 3 nodes: tenant control planes and their datastore live here |
|
||||||
|
| LoadBalancer | none needed (kube-vip in workload CP) | MetalLB (or kube-vip cloud provider) with an IP pool on VLAN 4, one VIP per tenant |
|
||||||
|
| Datastore | — | `kamaji-etcd` chart 0.18.1 (etcd 3.6.12); alternatives: PostgreSQL/MySQL via kine |
|
||||||
|
| Storage | — | CSI for etcd PVCs |
|
||||||
|
|
||||||
|
k3s' built-in servicelb (klipper) binds host ports, so it can't hand out many `:6443` endpoints. Use MetalLB.
|
||||||
|
|
||||||
|
## Autoscaling per tenant
|
||||||
|
|
||||||
|
A cluster-autoscaler instance in `clusterapi` mode serves **one** workload cluster (one kubeconfig).
|
||||||
|
So each tenant gets its own CA release in its namespace in the management cluster
|
||||||
|
(`kubeconfig-incluster`, secret `<tenant>-kubeconfig`). Automating that is our first real Go code:
|
||||||
|
|
||||||
|
- **tenant-operator** (small controller in this repo): watches `Cluster` objects labelled
|
||||||
|
`kaas.lab/autoscaling=enabled`, renders a per-tenant cluster-autoscaler Deployment + RBAC, and
|
||||||
|
cleans up on delete. The alternative is CAAPH (cluster-api-addon-provider-helm), but that installs
|
||||||
|
into the *workload* cluster, and we want CA in the management cluster.
|
||||||
|
|
||||||
|
## Self-service UX
|
||||||
|
|
||||||
|
- `ClusterClass` `lab-tenant` with variables: Kubernetes version, worker size, min/max workers, VLAN.
|
||||||
|
- A tenant cluster is one `Cluster` with `spec.topology` referencing the class.
|
||||||
|
- A namespace per tenant with RBAC limited to its own `Cluster`; the kubeconfig is handed out from the
|
||||||
|
`<tenant>-kubeconfig` secret.
|
||||||
|
|
||||||
|
## Open questions and risks
|
||||||
|
|
||||||
|
- **CAPI version skew:** Kamaji CP provider v0.21.0 is built against CAPI **v1.13.4**, while CAPMOX
|
||||||
|
v0.9.x is tested on **v1.12**. All providers share one CAPI core, so either find a Kamaji CP
|
||||||
|
provider release built on 1.12 or wait for CAPMOX to support 1.13+. Check each provider's
|
||||||
|
`metadata.yaml` contract before installing.
|
||||||
|
- **Kamaji OSS release model:** OSS publishes edge releases only (latest tag `edge-26.2.5`, Helm chart
|
||||||
|
`0.0.0+latest`). Stable versioned builds are commercial (CLASTIX), so pin by edge tag and image digest.
|
||||||
|
- Tenant Kubernetes version range is bounded by the Kamaji release and by our node images. We need
|
||||||
|
one image-builder template per supported minor.
|
||||||
|
- Backup and restore of the shared datastore (etcd snapshots) becomes critical: it holds every tenant.
|
||||||
|
|
||||||
|
## Exit criteria
|
||||||
|
|
||||||
|
- [ ] Tenant cluster created from a single `Cluster` + `ClusterClass`, control plane as pods in mgmt.
|
||||||
|
- [ ] Workers join via Konnectivity, and CNI works.
|
||||||
|
- [ ] Per-tenant cluster-autoscaler deployed automatically, scale 0 → N → 0 works.
|
||||||
|
- [ ] Deleting the `Cluster` removes the control plane, VMs, VIP and autoscaler.
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# Session: which conventions to adopt from egress-proxies-operator
|
||||||
|
|
||||||
|
**Created:** 2026-09-26 23:55
|
||||||
|
|
||||||
|
## Question
|
||||||
|
|
||||||
|
> look into project in /Users/jan.novak/srv/go/egress-proxies-operator and suggest which rules i
|
||||||
|
> should copy over into this project
|
||||||
|
>
|
||||||
|
> this project is sort of research/experimentation project, so i want to keep history of
|
||||||
|
> conversations, decisions, plans, etc.
|
||||||
|
|
||||||
|
## What was checked
|
||||||
|
|
||||||
|
`egress-proxies-operator`: `CLAUDE.md`, `.claude/settings.json` + `settings.local.json`,
|
||||||
|
`.claude/commands/{ship,release}.md`, `.claude/agents/*`, `AGENTS.md`, and samples from
|
||||||
|
`docs/plans/`, `docs/plans-executions/`, `docs/prompts/`, `docs/reviews/`, `CHANGELOG.md`.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
Adopted, adapted to a docs/lab repo:
|
||||||
|
|
||||||
|
- Plans in `docs/plans/` with the "commit the plan first, on its own" ordering rule.
|
||||||
|
- Execution logs in `docs/plans-executions/`, especially "exact commands with one line of
|
||||||
|
context, deviations called out", which suits lab runbook work better than Go code.
|
||||||
|
- `docs/prompts/`, `docs/reviews/`, top-inserted `CHANGELOG.md` with real `date` timestamps.
|
||||||
|
- Gitea + `tea pr create` MR flow, but only for phase-3 Go code; docs go straight to `main`.
|
||||||
|
- `/ship` command; a small read-only permission allowlist (`kubectl get/describe/logs`,
|
||||||
|
`clusterctl describe`, `helm list`, git read commands).
|
||||||
|
|
||||||
|
Not adopted: Go and testing conventions, CI/CD, kubebuilder gotchas, `AGENTS.md`, the two
|
||||||
|
operator reviewer agents (candidates for phase 3), `/release` (no tags or images here),
|
||||||
|
`.junie/.codex/.agents` duplicates, and the broad `kubectl *` / `curl *` permissions.
|
||||||
|
|
||||||
|
New, not in the operator repo, because decisions there are buried in plan context sections:
|
||||||
|
|
||||||
|
- `docs/decisions.md`: one table with alternatives and status, back-filled with 9 rows.
|
||||||
|
- `docs/sessions/`: summaries of conversations that did not produce a plan (this file is the first).
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- ~~Whether `kubectl get *` in the allowlist is too broad: `kubectl get secret -o yaml` would
|
||||||
|
print the `lab1-kubeconfig` secret without a prompt.~~ Resolved by
|
||||||
|
[decision #11](../decisions.md): a parallel session introduced read-only `cluster-inspector`
|
||||||
|
kubeconfigs, so the `kubectl` / `clusterctl` / `helm` allowlist entries were dropped from
|
||||||
|
`.claude/settings.json` again.
|
||||||
|
|
||||||
|
## Follow-ups
|
||||||
|
|
||||||
|
- [Decision #10](../decisions.md)
|
||||||
|
- `CLAUDE.md` sections "Project history" and "Git"
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Versions
|
||||||
|
|
||||||
|
Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go module proxy,
|
||||||
|
`dl.k8s.io`, `registry.k8s.io`, upstream Helm indexes and git tags. Re-verify before every deploy.
|
||||||
|
|
||||||
|
## Phase 1 — pinned
|
||||||
|
|
||||||
|
| Component | Pinned | Latest available | Why pinned there |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| Cluster API core, kubeadm bootstrap + control plane, clusterctl | **v1.12.11** | v1.14.2 | CAPMOX v0.9 compatibility table lists CAPI v1.11 / v1.12 only; built against v1.12.11 |
|
||||||
|
| CAPMOX (ionos-cloud/cluster-api-provider-proxmox) | **v0.9.1** | v0.9.1 | API `v1alpha2`, contract `v1beta2` |
|
||||||
|
| IPAM provider in-cluster | **v1.1.1** | v1.1.1 | CAPMOX v0.9 requires ≥ 1.1.0 |
|
||||||
|
| Workload Kubernetes | **v1.35.9** | v1.37.1 | CAPI 1.12 supports workload clusters up to 1.35 |
|
||||||
|
| Management cluster (k3s) | **v1.35.8+k3s1** | v1.37.0+k3s1 | CAPI 1.12 supports management clusters 1.31–1.35 |
|
||||||
|
| cluster-autoscaler | **v1.35.2** | v1.36.1 | CA minor must match workload Kubernetes minor |
|
||||||
|
| cluster-autoscaler Helm chart | **9.59.0** | 9.59.0 | default appVersion 1.35.0 → override `image.tag` |
|
||||||
|
| Cilium | **1.20.2** | 1.20.2 | |
|
||||||
|
| kube-vip | **v1.2.4** | v1.2.4 | CAPMOX template still ships v0.7.1 |
|
||||||
|
| image-builder | **v0.1.55** | v0.1.55 | |
|
||||||
|
| Packer (installed by image-builder) | **1.9.5** | — | last MPL-licensed release, pinned by image-builder |
|
||||||
|
| packer-plugin-proxmox | **v1.2.4** | v1.2.4 | pulled by `packer init` (`>= 1.1.2`) |
|
||||||
|
| ansible-core (image-builder) | **2.18.18** | — | pinned by image-builder `hack/utils.sh` |
|
||||||
|
| Node OS | Ubuntu 24.04.4 | — | ISO sha256 `e907d92e…a138433` |
|
||||||
|
|
||||||
|
### Upgrade path
|
||||||
|
|
||||||
|
When CAPMOX adds CAPI 1.13/1.14 to its compatibility table: bump CAPI core → k3s mgmt to 1.36+ →
|
||||||
|
rebuild template with Kubernetes 1.36/1.37 → cluster-autoscaler to the matching minor.
|
||||||
|
|
||||||
|
## Phase 2 / 3 — candidates (not pinned yet)
|
||||||
|
|
||||||
|
| Component | Latest | Built against CAPI | Notes |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| CAPLV (joshuagrisham/cluster-api-provider-libvirt) | v0.2.1 (2026-02) | v1.12.1 | pure-Go `digitalocean/go-libvirt`, single maintainer |
|
||||||
|
| CAPK (cluster-api-provider-kubevirt) | v0.11.2 (2026-02) | v1.11.4 | needs KubeVirt on the hypervisors |
|
||||||
|
| KubeVirt | v1.9.0 (2026-07) | — | |
|
||||||
|
| CAPM3 (Metal3) | v1.14.1 (2026-09) | — | needs Ironic + BMO + sushy-tools for VMs |
|
||||||
|
| cluster-api-control-plane-provider-kamaji | v0.21.0 (2026-09-16) | **v1.13.4** | ⚠ newer CAPI than CAPMOX v0.9 is tested with |
|
||||||
|
| Kamaji (OSS) | `edge-26.2.5`, Helm chart `0.0.0+latest` | — | OSS ships edge releases only; last semver tag v1.0.0 (2024) |
|
||||||
|
| kamaji-etcd Helm chart | 0.18.1 (etcd 3.6.12) | — | datastore for hosted control planes |
|
||||||
|
| CLASTIX `capi-kamaji-proxmox` chart | 0.0.4 (2025-10) | — | reference for Kamaji + CAPMOX wiring |
|
||||||
|
|
||||||
|
The CAPI version skew between CAPMOX (1.12) and the Kamaji CP provider (1.13) is the main open
|
||||||
|
compatibility question for phase 3 — all providers in one management cluster share one CAPI core.
|
||||||
Reference in New Issue
Block a user