kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document Ansible-built management cluster (plan step 4)

Browse Source 
Phase-1 Step 2 rewritten around ansible/mgmt.yml and cluster-api-operator,
version pins for operator/cert-manager/cloud image, decisions #12-14,
execution log and changelog. Phase-1 Step 2 checked off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-27 10:41:03 +02:00
co-authored by Claude Opus 5.5
parent aaa1ed64e2
commit d8efa3b961
9 changed files with 121 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/settings.json
+3 -1
View File
@@ -16,7 +16,9 @@
"Bash(clusterctl version *)",
"Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get nodes)",
"Bash(clusterctl generate *)",
"Bash(ansible mgmt *)"
"Bash(ansible mgmt *)",
"Bash(kubectl get *)",
"Bash(kubectl auth *)"
]
}
}
CHANGELOG.md
+12
View File
@@ -1,5 +1,17 @@
# Changelog
## 2026-09-27 10:40 CEST — Management cluster running, built by an idempotent Ansible playbook
- Phase 1 Step 2 done: `ansible/mgmt.yml` creates VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`
from the pinned Ubuntu cloud image, installs k3s v1.35.8+k3s1, cert-manager v1.21.2 and
cluster-api-operator 0.28.0, and CAPI v1.12.11 / CAPMOX v0.9.1 / IPAM v1.1.1 as operator
Provider CRs (all READY). A second full run reports `changed=0`.
- `clusterctl init` is gone from the runbook; `clusterctl` (v1.12.11 in `~/bin`) is used only for
`generate`/`describe`/`get kubeconfig`. Proxmox credentials moved to `ansible/secrets.yml`.
- cluster-inspector role can now read Provider CRs, HelmCharts and nodes.
- Files: `ansible/`, `docs/phase-1-proxmox-capi.md`, `docs/versions.md`, `docs/decisions.md`
(#12–14), `configs/capi/clusterctl.yaml.example`, `CLAUDE.md`, `README.md`.
## 2026-09-27 00:02 CEST — Read-only cluster-inspector credentials next to the agent
- Moved inspector RBAC from `configs/rbac/` to `.claude/cluster-inspector/` with `make-kubeconfig.sh`
CLAUDE.md
+7 -1
View File
@@ -16,6 +16,8 @@ Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux
- Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2.
- Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds).
- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55). Login only via cloud-init.
- Management cluster: VM 201 `lab-mgmt`, 192.168.4.21 (`ssh ubuntu@`), built by the idempotent
`ansible/mgmt.yml` (re-run must be `changed=0`); admin kubeconfig `~/.kube/lab-mgmt.kubeconfig`.
- Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`,
PVESDNUser on `/sdn/zones/localnetwork`.
@@ -28,7 +30,11 @@ Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux
- CAPI core is pinned to **v1.12.x** because CAPMOX v0.9 is only tested on CAPI 1.11/1.12. This caps
workload Kubernetes at 1.35 and cluster-autoscaler at 1.35.x. Don't bump CAPI unless the CAPMOX
compatibility table allows it.
- Always pass explicit versions to `clusterctl init` / `helm install`; never rely on "latest".
- Always pin versions explicitly (management cluster: `ansible/group_vars/all.yml`; `helm install
--version`); never rely on "latest".
- CAPI providers are managed by cluster-api-operator (Provider CRs from `ansible/mgmt.yml`). Never
run `clusterctl init/upgrade/move/delete` against the management cluster; `clusterctl` is only for
`generate cluster` (always with `--infrastructure proxmox:v0.9.1`), `describe`, `get kubeconfig`.
- Check a provider's `metadata.yaml` contract and CAPI version (go.mod) before adding it to the
management cluster — all providers share one CAPI core (known skew: Kamaji CP provider v0.21.0 is
on CAPI 1.13).
README.md
+5 -1
View File
@@ -5,7 +5,8 @@ Node autoscaling for Kubernetes clusters running as VMs on a home-lab hypervisor
growing, in later phases, into a "poor man's Kubernetes-as-a-Service".
> Status: **Phase 1 in progress** — VM template built and smoke-tested; management cluster
> and first workload cluster next. Versions verified 2026-09-24/26, see [docs/versions.md](docs/versions.md).
> running (k3s + CAPI via cluster-api-operator, built by [ansible/mgmt.yml](ansible/mgmt.yml));
> first workload cluster `lab1` next. Versions verified 2026-09-24/26/27, see [docs/versions.md](docs/versions.md).
## Goals
@@ -74,6 +75,7 @@ Phases 2 and 3 are independent and can be done in either order.
├── README.md # this file
├── CLAUDE.md # context + rules for Claude Code sessions
├── CHANGELOG.md # milestones and significant changes, newest first
├── ansible/ # mgmt.yml: management VM + k3s + cert-manager + CAPI operator/providers
├── .claude/
│ ├── agents/ # upstream-researcher, cluster-inspector (read-only)
│ ├── commands/ # /ship
@@ -113,3 +115,5 @@ Short summary; the full log with alternatives and status is [docs/decisions.md](
to keep its version under our control.
5. **Scale-from-zero via capacity annotations** on the MachineDeployment — CAPMOX does not publish
`status.capacity` on `ProxmoxMachineTemplate`.
6. **Management cluster from one idempotent Ansible playbook**, CAPI providers installed by
**cluster-api-operator** instead of `clusterctl init` — re-runs are no-ops, upgrades are version bumps.
configs/capi/clusterctl.yaml.example
+2 -4
View File
@@ -1,8 +1,6 @@
# copy to ~/.cluster-api/clusterctl.yaml — never commit the real file
## controller
PROXMOX_URL: "https://192.168.0.8:8006"
PROXMOX_TOKEN: "capmox@pve!capi"
PROXMOX_SECRET: "<secret>"
# Only used by `clusterctl generate cluster`. No Proxmox credentials here: CAPMOX is installed by
# cluster-api-operator and gets them from ansible/secrets.yml (Secret capmox-system/capmox-variables).
## workload defaults
PROXMOX_SOURCENODE: "pv1-1" # node holding the template
docs/decisions.md
+3
View File
@@ -20,3 +20,6 @@ image-builder fixes log; their dates are when the decision was recorded, not nec
| 9 | 2026-09-26 | Phases 2 and 3 are independent and can be done in either order | strict sequence | no technical dependency between the KVM provider and hosted control planes | active |
| 10 | 2026-09-26 | Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog ([session](sessions/2026-09-26-2355-history-conventions.md)) | chat history only; raw transcripts committed | research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets | active |
| 11 | 2026-09-27 | Cluster inspection by agents only via read-only ServiceAccount kubeconfigs minted by `.claude/cluster-inspector/make-kubeconfig.sh` (short-lived TokenRequest tokens, gitignored) | admin kubeconfig + permission allowlist; long-lived SA token Secret; RBAC under `configs/` | RBAC (`view`, no Secrets) enforces read-only regardless of which kubectl verb an agent runs — also closes the `kubectl get secret` gap from the history-conventions session; short-lived tokens limit leak impact | active |
| 12 | 2026-09-27 | Management cluster built by one idempotent Ansible playbook, `ansible/mgmt.yml`, run from the laptop ([plan](plans/2026-09-27-0058-mgmt-cluster-ansible.md)) | cloud-init + k3s auto-deploy manifests; bash scripts with guards; OpenTofu + Helm | owner's choice; idempotent modules, one tool; only `ansible.builtin`, charts through k3s's helm-controller so nothing extra on the VM | active |
| 13 | 2026-09-27 | CAPI providers installed by cluster-api-operator 0.28.0 (Provider CRs), cert-manager v1.21.2 pre-installed | guarded `clusterctl init` | declarative; re-apply is a no-op; upgrade = `spec.version` bump; 0.28.0 built against CAPI 1.12.10. Consequence: no `clusterctl init/upgrade/move/delete` | active |
| 14 | 2026-09-27 | Management VM created by the playbook from the pinned Ubuntu 24.04 cloud image (VM 201, 192.168.4.21) | hand-made VM; the image-builder node template | reproducible from the repo; the image-builder template is a kubeadm node image, not meant for k3s | active |
docs/phase-1-proxmox-capi.md
+45 -18
View File
@@ -1,6 +1,6 @@
# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler
Goal: a manually installed k3s management cluster runs CAPI + CAPMOX and creates workload
Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload
cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
No custom code in this phase — only assembling upstream components.
@@ -10,7 +10,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
| Role | Where | Size |
| --- | --- | --- |
| Management cluster | 1 VM on Proxmox, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 8 GB RAM |
@@ -18,7 +18,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
- [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
- [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
- [ ] Step 2 — management cluster + `clusterctl init`
- [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers)
- [ ] Step 3 — workload cluster `lab1`
- [ ] Step 4 — CNI (Cilium)
- [ ] Step 5 — MachineDeployment autoscaling annotations
@@ -47,27 +47,50 @@ requires shared storage; with local storage keep `ALLOWED_NODES` to the node hol
## Step 2 — management cluster
```bash
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.8+k3s1 sh -s - --disable traefik
mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config
One idempotent playbook, run from the laptop: [ansible/mgmt.yml](../ansible/mgmt.yml). It creates
the VM on `pv1-1` from the pinned Ubuntu 24.04 cloud image, installs k3s, then cert-manager and
cluster-api-operator (as k3s `HelmChart`s), then the five CAPI providers as operator Provider CRs.
Versions and VM parameters are in `ansible/group_vars/all.yml`. A re-run must report `changed=0`.
curl -L -o clusterctl https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-linux-amd64
sudo install -m 0755 clusterctl /usr/local/bin/clusterctl
```
Laptop prerequisites:
`~/.cluster-api/clusterctl.yaml` — start from [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example).
- root SSH to `pv1-1` with `~/.ssh/id_ed25519` (`ssh-copy-id root@192.168.0.8`);
- ansible-core 2.18 (`brew install ansible`, or prefix commands with `uvx --from 'ansible-core>=2.18,<2.19'`);
- `cp ansible/secrets.yml.example ansible/secrets.yml` and fill in the `capmox@pve!capi` token secret;
- `clusterctl` v1.12.11 for Step 3 (not `brew install clusterctl`, which installs 1.14.x):
```bash
curl -fL -o /tmp/clusterctl \
https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-darwin-arm64
install -m 0755 /tmp/clusterctl ~/bin/clusterctl && rm /tmp/clusterctl
```
```bash
clusterctl init \
--core cluster-api:v1.12.11 \
--bootstrap kubeadm:v1.12.11 \
--control-plane kubeadm:v1.12.11 \
--infrastructure proxmox:v0.9.1 \
--ipam in-cluster:v1.1.1
cd ansible
ansible-playbook mgmt.yml --tags vm # VM 201 on pv1-1, waits for SSH
ansible-playbook mgmt.yml --tags k3s # qemu-guest-agent, k3s, node Ready
ansible-playbook mgmt.yml --tags addons # cert-manager v1.21.2, cluster-api-operator 0.28.0
ansible-playbook mgmt.yml --tags capi # capmox-variables Secret + 5 Provider CRs, waits for controllers
ansible-playbook mgmt.yml --tags kubeconfig # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfig
ansible-playbook mgmt.yml # everything; must be changed=0 on a built cluster
```
Always pin versions explicitly — without them clusterctl installs the newest core (1.14.x) and
breaks the tested CAPMOX combination.
```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A
# all READY True, INSTALLEDVERSION v1.12.11 / v0.9.1 / v1.1.1
```
cluster-api-operator replaces `clusterctl init`:
- Upgrade a provider by bumping its version in `group_vars/all.yml` and re-running `--tags capi`,
one provider per run.
- Use `clusterctl` only for `generate cluster` (always with `--infrastructure proxmox:v0.9.1`),
`describe` and `get kubeconfig`. Never `init`, `upgrade`, `move` or `delete`: they need the
clusterctl inventory, which the operator doesn't create.
- Proxmox credentials live only in `ansible/secrets.yml`; `~/.cluster-api/clusterctl.yaml` (from
[configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example)) holds just the
workload variables.
## Step 3 — workload cluster `lab1`
@@ -165,6 +188,10 @@ Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods,
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
| Provider CR not `READY` / `capi` play keeps retrying | operator preflight or component download failed | `kubectl -n capi-operator-system logs deploy/cluster-api-operator`, `kubectl -n <ns> describe <kind> <name>` |
| Provider keeps broken components after a failed fetch | stale per-version cache (cluster-api-operator #1075) | delete that provider's `…-cache` Secret, re-run `--tags capi` |
| CAPMOX still uses the old token after editing `secrets.yml` | operator doesn't watch `configSecret` | new Secret name in the template + `configSecret` reference |
| `ansible-playbook mgmt.yml` reports `changed` on a built cluster | a task isn't idempotent | `-v` for command output; on the VM `k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml` |
```bash
kubectl -n capmox-system logs deploy/capmox-controller-manager -f
docs/plans-executions/2026-09-27-0058-mgmt-cluster-ansible.md
+34 -4
View File
@@ -7,8 +7,8 @@ Pairs with [docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md](../plans/2026-09
- [x] Step 0 — Initial import commit + plan commit
- [x] Step 1 — Scaffold `ansible/`, `.gitignore` exception, syntax/lint checks
- [x] Step 2 — Run the playbook tag by tag against the lab (owner)
- [ ] Step 3 — Idempotency check (second full run `changed=0`)
- [ ] Step 4 — Docs, versions, decisions, changelog
- [x] Step 3 — Idempotency check (second full run `changed=0`)
- [x] Step 4 — Docs, versions, decisions, changelog
## Step 0 — Initial import + plan
@@ -56,7 +56,7 @@ uvx --from 'ansible-core>=2.18,<2.19' ansible pve -m ping -o
Play 1 asserts on them.
- `clusterctl` is not installed on the laptop either (needed for phase-1 Step 3, not for this playbook).
## Step 2 — Playbook runs (in progress)
## Step 2 — Playbook runs
### `--tags vm`, first attempt (2026-09-27 09:30 CEST, owner)
@@ -111,7 +111,7 @@ full playbook run re-applies it via the `inspector` task (`--apply-rbac`).
**Worth noting:** the `.claude/settings.json` allowlist picked up four one-off entries approved
during this check (`export KUBECONFIG=…`, three exact `kubectl … logs` commands).
## Step 3 — Idempotency check (in progress)
## Step 3 — Idempotency check
### Full run: `Apply provider manifests` always `changed` (2026-09-27 10:35 CEST)
@@ -135,3 +135,33 @@ Also from this check: the widened inspector RBAC was not applied yet (`get nodes
Forbidden), so no clean full run had completed before this fix. `clusterctl` v1.12.11
darwin/arm64 is installed in `~/bin`; `clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --target-namespace default --list-variables`
lists no `PROXMOX_*` credentials, confirming `clusterctl.yaml` only needs workload values now.
### Result (2026-09-27 ~10:38 CEST, owner)
After the `data` fix, one run with the expected `changed` on the Secret, then:
```text
PLAY RECAP
lab-mgmt : ok=19 changed=0 unreachable=0 failed=0 skipped=2
pv1-1 : ok=3 changed=0 unreachable=0 failed=0 skipped=6
```
Verified read-only with `lab-mgmt-ro`: node `lab-mgmt` Ready (v1.35.8+k3s1, Ubuntu 24.04.5);
all five Provider CRs `READY True` at v1.12.11 / v0.9.1 / v1.1.1; HelmCharts cert-manager v1.21.2
and cluster-api-operator 0.28.0 not failed; `auth can-i delete machines` / `get secrets` → `no`.
The widened inspector RBAC was applied by this run's `inspector` task.
## Step 4 — Docs (2026-09-27 10:40 CEST)
- `docs/phase-1-proxmox-capi.md`: Step 2 rewritten around the playbook (prereqs incl. `clusterctl`
darwin/arm64 install into `~/bin`, tag-by-tag commands, operator rules), checklist ticked,
topology row, 4 troubleshooting rows (provider not Ready, stale cache #1075, token rotation,
non-idempotent task).
- `docs/versions.md`: cluster-api-operator 0.28.0, cert-manager v1.21.2, cloud image serial;
provider upgrade procedure.
- `docs/decisions.md` #12–14; `CLAUDE.md` lab fact + version rules (no `clusterctl init/upgrade/move/delete`);
`configs/capi/clusterctl.yaml.example` without controller credentials; README status, layout, decision 6.
**Worth noting:** the mgmt VM runs Ubuntu 24.04.5 (cloud image), the node template 24.04.4
(ISO) — different on purpose, they're separate images. Not covered by the plan and still open:
token rotation, VM resize, re-creating the VM (SSH `known_hosts` would conflict with `accept-new`).
docs/versions.md
+10 -1
View File
@@ -1,6 +1,7 @@
# Versions
Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go module proxy,
Verified 2026-09-24 (phase 1), 2026-09-26 (phase 2/3 candidates) and 2026-09-27 (management
cluster add-ons) from the Go module proxy,
`dl.k8s.io`, `registry.k8s.io`, upstream Helm indexes and git tags. Re-verify before every deploy.
## Phase 1 — pinned
@@ -12,6 +13,9 @@ Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go
| IPAM provider in-cluster | **v1.1.1** | v1.1.1 | CAPMOX v0.9 requires ≥ 1.1.0 |
| Workload Kubernetes | **v1.35.9** | v1.37.1 | CAPI 1.12 supports workload clusters up to 1.35 |
| Management cluster (k3s) | **v1.35.8+k3s1** | v1.37.0+k3s1 | CAPI 1.12 supports management clusters 1.31–1.35 |
| cluster-api-operator (Helm chart) | **0.28.0** | 0.29.0 | installs the providers instead of `clusterctl init`; 0.28.0 is built against CAPI v1.12.10, 0.29.0 against v1.13.5 |
| cert-manager (Helm chart) | **v1.21.2** | v1.21.2 | required by the operator and all providers (webhook certs); supports k8s 1.33–1.36 |
| Management VM image | Ubuntu cloud image `noble/20260926` (24.04.5) | — | sha256 `6a81c375…5bb4d2`; `current/` is a rolling pointer, pin the dated serial |
| cluster-autoscaler | **v1.35.2** | v1.36.1 | CA minor must match workload Kubernetes minor |
| cluster-autoscaler Helm chart | **9.59.0** | 9.59.0 | default appVersion 1.35.0 → override `image.tag` |
| Cilium | **1.20.2** | 1.20.2 | |
@@ -27,6 +31,11 @@ Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go
When CAPMOX adds CAPI 1.13/1.14 to its compatibility table: bump CAPI core → k3s mgmt to 1.36+ →
rebuild template with Kubernetes 1.36/1.37 → cluster-autoscaler to the matching minor.
All management-cluster pins live in `ansible/group_vars/all.yml`. Provider upgrades are a
`spec.version` bump there + `ansible-playbook mgmt.yml --tags capi`, one provider per run (the
operator blocks minor downgrades). Bump cluster-api-operator only to a release built against the
target CAPI minor.
## Phase 2 / 3 — candidates (not pinned yet)
| Component | Latest | Built against CAPI | Notes |