kacerrandClaude Opus 5.5
49c548e296
The agent could only reach lab-mgmt, so it couldn't verify nodes, CNI or system pods on lab1. It correctly refused the admin ~/.kube/lab1.kubeconfig when the owner's permission was relayed through the main session. Workload kubeconfigs: - make-kubeconfig.sh: new --capi-cluster [NS/]NAME. --admin-kubeconfig is then the management cluster. The workload admin kubeconfig is read from Secret NS/NAME-kubeconfig into a mktemp file, removed on exit, so no admin file for the workload cluster lands on disk. - ansible/mgmt.yml --tags inspector now also mints kubeconfigs for every cluster in the new inspector_workload_clusters (group_vars: lab1). Clusters without a <name>-kubeconfig Secret are skipped. The tag is now the single weekly token refresh; tasks stay changed_when: false, so the playbook's changed=0 rule still holds. - Owner ran --tags inspector: kubeconfigs/lab1.kubeconfig (lab1-ro) minted, can-i checks as expected. Agent prompt: - Cluster map: what to inspect on lab-mgmt vs the workload cluster. Machine -> Node -> VM via nodeRef, providerID and ProxmoxMachine.spec.virtualMachineID (checked on live objects). - New triage step: scale-down blockers (non-DaemonSet kube-system pods without PDB, local storage, bare pods, PDBs at 0, safe-to-evict / scale-down-disabled). - Refuses admin kubeconfigs even when a caller relays permission; points the owner to --tags inspector on missing file or 401. - Verb before --kubeconfig so commands match the kubectl get/describe/logs allow rules; single-quote jsonpath/custom-columns for zsh. - Report format: abnormal first, then health summary, then unverified. settings.json: allow kubectl describe/logs/events and clusterctl describe. All four are read-only, and the RO kubeconfigs enforce read-only anyway. Checked: bash -n on the script; ansible-playbook --syntax-check and ansible-lint --profile production pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cluster-autoscaler-with-capi
Node autoscaling for Kubernetes clusters running as VMs on a home-lab hypervisor fleet (Proxmox VE + vanilla KVM/libvirt), driven by Cluster API and cluster-autoscaler — growing, in later phases, into a "poor man's Kubernetes-as-a-Service".
Status: Phase 1 in progress — VM template built and smoke-tested; management cluster running (k3s + CAPI via cluster-api-operator, built by ansible/mgmt.yml); first workload cluster
lab1next. Versions verified 2026-09-24/26/27, see docs/versions.md.
Goals
- Workload clusters whose worker pool scales 0 → N → 0 on demand, with VMs created and destroyed on the hypervisors automatically.
- Everything declarative: clusters are CAPI objects in a management cluster; no hand-made VMs.
- Learn Cluster API internals (contracts, providers, reconcile flow) by assembling, then extending it.
- End state: self-service tenant clusters (hosted control planes + autoscaled VM workers).
Non-goals (for now)
- HA management cluster, production SLAs, multi-site.
- GitOps / fleet management (may come later on top).
- Writing our own autoscaler — cluster-autoscaler's
clusterapiprovider does the job.
Lab inventory
| Hypervisor type | Management API | CAPI infrastructure provider |
|---|---|---|
Proxmox VE (e.g. pv1-1, 192.168.0.8) |
PVE REST API :8006 | CAPMOX (phase 1) |
| Ubuntu 24.04 + KVM/libvirt | libvirt RPC (qemu+ssh/qemu+tls) |
none mature upstream → phase 2 |
VM network: vmbr0 (VLAN-aware), VLAN 4, build host 192.168.4.181.
Architecture (phase 1)
flowchart LR
subgraph MGMT[mgmt cluster: k3s VM]
CAPI[CAPI core + kubeadm bootstrap/CP]
CAPMOX[CAPMOX]
IPAM[IPAM in-cluster]
CA[cluster-autoscaler]
end
PVE[Proxmox VE API]
subgraph LAB1[workload cluster lab1]
CP[control plane VM + kube-vip VIP]
W[worker VMs - MachineDeployment 0..N]
end
CA -- scales MachineDeployment --> CAPI
CAPI --> CAPMOX
CAPMOX -- clone template, cloud-init ISO --> PVE
PVE --> CP
PVE --> W
CA -- watches pending pods via lab1-kubeconfig --> LAB1
cluster-autoscaler runs in the management cluster in kubeconfig-incluster mode: it reads
pending pods/nodes from the workload cluster through the CAPI-generated kubeconfig secret and
changes replicas on MachineDeployments via the in-cluster management API.
Roadmap
| Phase | Scope | Exit criteria | Doc |
|---|---|---|---|
| 1 | k3s mgmt cluster (manual) → CAPI + CAPMOX → workload cluster on Proxmox VMs → cluster-autoscaler | scale-up 1→3, scale-down to 0, scale-from-zero all work | phase-1-proxmox-capi.md |
| 2 | CAPI provider for vanilla KVM/libvirt hosts (evaluate CAPLV / KubeVirt / Metal3 / own provider) | worker MachineDeployment on Ubuntu KVM hosts, autoscaled | phase-2-kvm-provider.md |
| 3 | Hosted control planes with Kamaji, tenant clusters on demand ("poor man's KaaS") | new tenant cluster from one manifest/ClusterClass, workers autoscaled | phase-3-kamaji-kaas.md |
Phases 2 and 3 are independent and can be done in either order.
Repository layout
.
├── README.md # this file
├── CLAUDE.md # context + rules for Claude Code sessions
├── CHANGELOG.md # milestones and significant changes, newest first
├── ansible/ # mgmt.yml: management VM + k3s + cert-manager + CAPI operator/providers
├── .claude/
│ ├── agents/ # upstream-researcher, cluster-inspector (read-only)
│ ├── commands/ # /ship
│ └── cluster-inspector/ # RBAC, kubeconfig minting script, gitignored kubeconfigs/
├── docs/
│ ├── decisions.md # decision log: alternatives, why, status
│ ├── plans/ # plans (plan mode), one file per plan
│ ├── plans-executions/ # per-plan execution logs with exact commands
│ ├── sessions/ # research/design conversation summaries
│ ├── prompts/ # prompts/specs that kicked off work
│ ├── reviews/ # review findings
│ ├── versions.md # pinned versions + compatibility reasoning
│ ├── phase-1-proxmox-capi.md # runbook for phase 1
│ ├── image-builder-proxmox.md # building the node VM template + fixes log
│ ├── phase-2-kvm-provider.md # KVM/libvirt provider options
│ └── phase-3-kamaji-kaas.md # hosted control planes / KaaS design notes
└── configs/
├── image-builder/ # packer var file for the Proxmox template
├── capi/ # clusterctl config example
├── cluster-autoscaler/ # Helm values
└── test/ # workloads for scale tests
Secrets (PVE token secret, kubeconfigs) never go into the repo — see .gitignore.
Key decisions so far
Short summary; the full log with alternatives and status is docs/decisions.md.
- CAPI v1.12.x, not the latest 1.14.x — CAPMOX v0.9.x is only tested against CAPI 1.11/1.12. This caps workload Kubernetes at 1.35 and cluster-autoscaler at 1.35.x.
- Management cluster = k3s on a Proxmox VM, not kind — CAPI controllers and the autoscaler must run permanently.
- Node images via kubernetes-sigs/image-builder (Ubuntu 24.04, Proxmox builder), overrides in
a
PACKER_VAR_FILESJSON instead ofPACKER_FLAGS. - CNI installed by Helm (Cilium,
ipam.mode=kubernetes), not via ClusterResourceSet flavors, to keep its version under our control. - Scale-from-zero via capacity annotations on the MachineDeployment — CAPMOX does not publish
status.capacityonProxmoxMachineTemplate. - Management cluster from one idempotent Ansible playbook, CAPI providers installed by
cluster-api-operator instead of
clusterctl init— re-runs are no-ops, upgrades are version bumps.