Document Ansible-built management cluster (plan step 4)

Phase-1 Step 2 rewritten around ansible/mgmt.yml and cluster-api-operator,
version pins for operator/cert-manager/cloud image, decisions #12-14,
execution log and changelog. Phase-1 Step 2 checked off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.claude/settings.json

@@ -16,7 +16,9 @@
       "Bash(clusterctl version *)",
       "Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get nodes)",
       "Bash(clusterctl generate *)",
-      "Bash(ansible mgmt *)"
+      "Bash(ansible mgmt *)",
+      "Bash(kubectl get *)",
+      "Bash(kubectl auth *)"
     ]
   }
 }

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 2026-09-27 10:40 CEST — Management cluster running, built by an idempotent Ansible playbook
+
+- Phase 1 Step 2 done: `ansible/mgmt.yml` creates VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`
+  from the pinned Ubuntu cloud image, installs k3s v1.35.8+k3s1, cert-manager v1.21.2 and
+  cluster-api-operator 0.28.0, and CAPI v1.12.11 / CAPMOX v0.9.1 / IPAM v1.1.1 as operator
+  Provider CRs (all READY). A second full run reports `changed=0`.
+- `clusterctl init` is gone from the runbook; `clusterctl` (v1.12.11 in `~/bin`) is used only for
+  `generate`/`describe`/`get kubeconfig`. Proxmox credentials moved to `ansible/secrets.yml`.
+- cluster-inspector role can now read Provider CRs, HelmCharts and nodes.
+- Files: `ansible/`, `docs/phase-1-proxmox-capi.md`, `docs/versions.md`, `docs/decisions.md`
+  (#12–14), `configs/capi/clusterctl.yaml.example`, `CLAUDE.md`, `README.md`.
+
 ## 2026-09-27 00:02 CEST — Read-only cluster-inspector credentials next to the agent
 
 - Moved inspector RBAC from `configs/rbac/` to `.claude/cluster-inspector/` with `make-kubeconfig.sh`

CLAUDE.md

@@ -16,6 +16,8 @@ Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux
 - Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2.
 - Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds).
 - Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55). Login only via cloud-init.
+- Management cluster: VM 201 `lab-mgmt`, 192.168.4.21 (`ssh ubuntu@`), built by the idempotent
+  `ansible/mgmt.yml` (re-run must be `changed=0`); admin kubeconfig `~/.kube/lab-mgmt.kubeconfig`.
 - Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`,
   PVESDNUser on `/sdn/zones/localnetwork`.
 
@@ -28,7 +30,11 @@ Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux
 - CAPI core is pinned to **v1.12.x** because CAPMOX v0.9 is only tested on CAPI 1.11/1.12. This caps
   workload Kubernetes at 1.35 and cluster-autoscaler at 1.35.x. Don't bump CAPI unless the CAPMOX
   compatibility table allows it.
-- Always pass explicit versions to `clusterctl init` / `helm install`; never rely on "latest".
+- Always pin versions explicitly (management cluster: `ansible/group_vars/all.yml`; `helm install
+  --version`); never rely on "latest".
+- CAPI providers are managed by cluster-api-operator (Provider CRs from `ansible/mgmt.yml`). Never
+  run `clusterctl init/upgrade/move/delete` against the management cluster; `clusterctl` is only for
+  `generate cluster` (always with `--infrastructure proxmox:v0.9.1`), `describe`, `get kubeconfig`.
 - Check a provider's `metadata.yaml` contract and CAPI version (go.mod) before adding it to the
   management cluster — all providers share one CAPI core (known skew: Kamaji CP provider v0.21.0 is
   on CAPI 1.13).

README.md

@@ -5,7 +5,8 @@ Node autoscaling for Kubernetes clusters running as VMs on a home-lab hypervisor
 growing, in later phases, into a "poor man's Kubernetes-as-a-Service".
 
 > Status: **Phase 1 in progress** — VM template built and smoke-tested; management cluster
-> and first workload cluster next. Versions verified 2026-09-24/26, see [docs/versions.md](docs/versions.md).
+> running (k3s + CAPI via cluster-api-operator, built by [ansible/mgmt.yml](ansible/mgmt.yml));
+> first workload cluster `lab1` next. Versions verified 2026-09-24/26/27, see [docs/versions.md](docs/versions.md).
 
 ## Goals
 
@@ -74,6 +75,7 @@ Phases 2 and 3 are independent and can be done in either order.
 ├── README.md                     # this file
 ├── CLAUDE.md                     # context + rules for Claude Code sessions
 ├── CHANGELOG.md                  # milestones and significant changes, newest first
+├── ansible/                      # mgmt.yml: management VM + k3s + cert-manager + CAPI operator/providers
 ├── .claude/
 │   ├── agents/                   # upstream-researcher, cluster-inspector (read-only)
 │   ├── commands/                 # /ship
@@ -113,3 +115,5 @@ Short summary; the full log with alternatives and status is [docs/decisions.md](
    to keep its version under our control.
 5. **Scale-from-zero via capacity annotations** on the MachineDeployment — CAPMOX does not publish
    `status.capacity` on `ProxmoxMachineTemplate`.
+6. **Management cluster from one idempotent Ansible playbook**, CAPI providers installed by
+   **cluster-api-operator** instead of `clusterctl init` — re-runs are no-ops, upgrades are version bumps.

configs/capi/clusterctl.yaml.example

@@ -1,8 +1,6 @@
 # copy to ~/.cluster-api/clusterctl.yaml — never commit the real file
-## controller
+# Only used by `clusterctl generate cluster`. No Proxmox credentials here: CAPMOX is installed by
-PROXMOX_URL: "https://192.168.0.8:8006"
+# cluster-api-operator and gets them from ansible/secrets.yml (Secret capmox-system/capmox-variables).
-PROXMOX_TOKEN: "capmox@pve!capi"
-PROXMOX_SECRET: "<secret>"
 
 ## workload defaults
 PROXMOX_SOURCENODE: "pv1-1"               # node holding the template

docs/decisions.md

@@ -20,3 +20,6 @@ image-builder fixes log; their dates are when the decision was recorded, not nec
 | 9 | 2026-09-26 | Phases 2 and 3 are independent and can be done in either order | strict sequence | no technical dependency between the KVM provider and hosted control planes | active |
 | 10 | 2026-09-26 | Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog ([session](sessions/2026-09-26-2355-history-conventions.md)) | chat history only; raw transcripts committed | research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets | active |
 | 11 | 2026-09-27 | Cluster inspection by agents only via read-only ServiceAccount kubeconfigs minted by `.claude/cluster-inspector/make-kubeconfig.sh` (short-lived TokenRequest tokens, gitignored) | admin kubeconfig + permission allowlist; long-lived SA token Secret; RBAC under `configs/` | RBAC (`view`, no Secrets) enforces read-only regardless of which kubectl verb an agent runs — also closes the `kubectl get secret` gap from the history-conventions session; short-lived tokens limit leak impact | active |
+| 12 | 2026-09-27 | Management cluster built by one idempotent Ansible playbook, `ansible/mgmt.yml`, run from the laptop ([plan](plans/2026-09-27-0058-mgmt-cluster-ansible.md)) | cloud-init + k3s auto-deploy manifests; bash scripts with guards; OpenTofu + Helm | owner's choice; idempotent modules, one tool; only `ansible.builtin`, charts through k3s's helm-controller so nothing extra on the VM | active |
+| 13 | 2026-09-27 | CAPI providers installed by cluster-api-operator 0.28.0 (Provider CRs), cert-manager v1.21.2 pre-installed | guarded `clusterctl init` | declarative; re-apply is a no-op; upgrade = `spec.version` bump; 0.28.0 built against CAPI 1.12.10. Consequence: no `clusterctl init/upgrade/move/delete` | active |
+| 14 | 2026-09-27 | Management VM created by the playbook from the pinned Ubuntu 24.04 cloud image (VM 201, 192.168.4.21) | hand-made VM; the image-builder node template | reproducible from the repo; the image-builder template is a kubeadm node image, not meant for k3s | active |

docs/phase-1-proxmox-capi.md

@@ -1,6 +1,6 @@
 # Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler
 
-Goal: a manually installed k3s management cluster runs CAPI + CAPMOX and creates workload
+Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload
 cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
 No custom code in this phase — only assembling upstream components.
 
@@ -10,7 +10,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
 
 | Role | Where | Size |
 | --- | --- | --- |
-| Management cluster | 1 VM on Proxmox, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
+| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
 | `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
 | `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 8 GB RAM |
 
@@ -18,7 +18,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
 
 - [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
 - [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
-- [ ] Step 2 — management cluster + `clusterctl init`
+- [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers)
 - [ ] Step 3 — workload cluster `lab1`
 - [ ] Step 4 — CNI (Cilium)
 - [ ] Step 5 — MachineDeployment autoscaling annotations
@@ -47,27 +47,50 @@ requires shared storage; with local storage keep `ALLOWED_NODES` to the node hol
 
 ## Step 2 — management cluster
 
-```bash
+One idempotent playbook, run from the laptop: [ansible/mgmt.yml](../ansible/mgmt.yml). It creates
-curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.8+k3s1 sh -s - --disable traefik
+the VM on `pv1-1` from the pinned Ubuntu 24.04 cloud image, installs k3s, then cert-manager and
-mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config
+cluster-api-operator (as k3s `HelmChart`s), then the five CAPI providers as operator Provider CRs.
+Versions and VM parameters are in `ansible/group_vars/all.yml`. A re-run must report `changed=0`.
 
-curl -L -o clusterctl https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-linux-amd64
+Laptop prerequisites:
-sudo install -m 0755 clusterctl /usr/local/bin/clusterctl
-```
 
-`~/.cluster-api/clusterctl.yaml` — start from [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example).
+- root SSH to `pv1-1` with `~/.ssh/id_ed25519` (`ssh-copy-id root@192.168.0.8`);
+- ansible-core 2.18 (`brew install ansible`, or prefix commands with `uvx --from 'ansible-core>=2.18,<2.19'`);
+- `cp ansible/secrets.yml.example ansible/secrets.yml` and fill in the `capmox@pve!capi` token secret;
+- `clusterctl` v1.12.11 for Step 3 (not `brew install clusterctl`, which installs 1.14.x):
 
   ```bash
-clusterctl init \
+  curl -fL -o /tmp/clusterctl \
-  --core cluster-api:v1.12.11 \
+    https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-darwin-arm64
-  --bootstrap kubeadm:v1.12.11 \
+  install -m 0755 /tmp/clusterctl ~/bin/clusterctl && rm /tmp/clusterctl
-  --control-plane kubeadm:v1.12.11 \
-  --infrastructure proxmox:v0.9.1 \
-  --ipam in-cluster:v1.1.1
   ```
 
-Always pin versions explicitly — without them clusterctl installs the newest core (1.14.x) and
+```bash
-breaks the tested CAPMOX combination.
+cd ansible
+ansible-playbook mgmt.yml --tags vm           # VM 201 on pv1-1, waits for SSH
+ansible-playbook mgmt.yml --tags k3s          # qemu-guest-agent, k3s, node Ready
+ansible-playbook mgmt.yml --tags addons       # cert-manager v1.21.2, cluster-api-operator 0.28.0
+ansible-playbook mgmt.yml --tags capi         # capmox-variables Secret + 5 Provider CRs, waits for controllers
+ansible-playbook mgmt.yml --tags kubeconfig   # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfig
+ansible-playbook mgmt.yml                     # everything; must be changed=0 on a built cluster
+```
+
+```bash
+export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
+kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A
+# all READY True, INSTALLEDVERSION v1.12.11 / v0.9.1 / v1.1.1
+```
+
+cluster-api-operator replaces `clusterctl init`:
+
+- Upgrade a provider by bumping its version in `group_vars/all.yml` and re-running `--tags capi`,
+  one provider per run.
+- Use `clusterctl` only for `generate cluster` (always with `--infrastructure proxmox:v0.9.1`),
+  `describe` and `get kubeconfig`. Never `init`, `upgrade`, `move` or `delete`: they need the
+  clusterctl inventory, which the operator doesn't create.
+- Proxmox credentials live only in `ansible/secrets.yml`; `~/.cluster-api/clusterctl.yaml` (from
+  [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example)) holds just the
+  workload variables.
 
 ## Step 3 — workload cluster `lab1`
 
@@ -165,6 +188,10 @@ Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods,
 | clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
 | ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
 | `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
+| Provider CR not `READY` / `capi` play keeps retrying | operator preflight or component download failed | `kubectl -n capi-operator-system logs deploy/cluster-api-operator`, `kubectl -n <ns> describe <kind> <name>` |
+| Provider keeps broken components after a failed fetch | stale per-version cache (cluster-api-operator #1075) | delete that provider's `…-cache` Secret, re-run `--tags capi` |
+| CAPMOX still uses the old token after editing `secrets.yml` | operator doesn't watch `configSecret` | new Secret name in the template + `configSecret` reference |
+| `ansible-playbook mgmt.yml` reports `changed` on a built cluster | a task isn't idempotent | `-v` for command output; on the VM `k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml` |
 
 ```bash
 kubectl -n capmox-system logs deploy/capmox-controller-manager -f

docs/plans-executions/2026-09-27-0058-mgmt-cluster-ansible.md

@@ -7,8 +7,8 @@ Pairs with [docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md](../plans/2026-09
 - [x] Step 0 — Initial import commit + plan commit
 - [x] Step 1 — Scaffold `ansible/`, `.gitignore` exception, syntax/lint checks
 - [x] Step 2 — Run the playbook tag by tag against the lab (owner)
-- [ ] Step 3 — Idempotency check (second full run `changed=0`)
+- [x] Step 3 — Idempotency check (second full run `changed=0`)
-- [ ] Step 4 — Docs, versions, decisions, changelog
+- [x] Step 4 — Docs, versions, decisions, changelog
 
 ## Step 0 — Initial import + plan
 
@@ -56,7 +56,7 @@ uvx --from 'ansible-core>=2.18,<2.19' ansible pve -m ping -o
   Play 1 asserts on them.
 - `clusterctl` is not installed on the laptop either (needed for phase-1 Step 3, not for this playbook).
 
-## Step 2 — Playbook runs (in progress)
+## Step 2 — Playbook runs
 
 ### `--tags vm`, first attempt (2026-09-27 09:30 CEST, owner)
 
@@ -111,7 +111,7 @@ full playbook run re-applies it via the `inspector` task (`--apply-rbac`).
 **Worth noting:** the `.claude/settings.json` allowlist picked up four one-off entries approved
 during this check (`export KUBECONFIG=…`, three exact `kubectl … logs` commands).
 
-## Step 3 — Idempotency check (in progress)
+## Step 3 — Idempotency check
 
 ### Full run: `Apply provider manifests` always `changed` (2026-09-27 10:35 CEST)
 
@@ -135,3 +135,33 @@ Also from this check: the widened inspector RBAC was not applied yet (`get nodes
 Forbidden), so no clean full run had completed before this fix. `clusterctl` v1.12.11
 darwin/arm64 is installed in `~/bin`; `clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --target-namespace default --list-variables`
 lists no `PROXMOX_*` credentials, confirming `clusterctl.yaml` only needs workload values now.
+
+### Result (2026-09-27 ~10:38 CEST, owner)
+
+After the `data` fix, one run with the expected `changed` on the Secret, then:
+
+```text
+PLAY RECAP
+lab-mgmt : ok=19 changed=0 unreachable=0 failed=0 skipped=2
+pv1-1    : ok=3  changed=0 unreachable=0 failed=0 skipped=6
+```
+
+Verified read-only with `lab-mgmt-ro`: node `lab-mgmt` Ready (v1.35.8+k3s1, Ubuntu 24.04.5);
+all five Provider CRs `READY True` at v1.12.11 / v0.9.1 / v1.1.1; HelmCharts cert-manager v1.21.2
+and cluster-api-operator 0.28.0 not failed; `auth can-i delete machines` / `get secrets` → `no`.
+The widened inspector RBAC was applied by this run's `inspector` task.
+
+## Step 4 — Docs (2026-09-27 10:40 CEST)
+
+- `docs/phase-1-proxmox-capi.md`: Step 2 rewritten around the playbook (prereqs incl. `clusterctl`
+  darwin/arm64 install into `~/bin`, tag-by-tag commands, operator rules), checklist ticked,
+  topology row, 4 troubleshooting rows (provider not Ready, stale cache #1075, token rotation,
+  non-idempotent task).
+- `docs/versions.md`: cluster-api-operator 0.28.0, cert-manager v1.21.2, cloud image serial;
+  provider upgrade procedure.
+- `docs/decisions.md` #12–14; `CLAUDE.md` lab fact + version rules (no `clusterctl init/upgrade/move/delete`);
+  `configs/capi/clusterctl.yaml.example` without controller credentials; README status, layout, decision 6.
+
+**Worth noting:** the mgmt VM runs Ubuntu 24.04.5 (cloud image), the node template 24.04.4
+(ISO) — different on purpose, they're separate images. Not covered by the plan and still open:
+token rotation, VM resize, re-creating the VM (SSH `known_hosts` would conflict with `accept-new`).

docs/versions.md

@@ -1,6 +1,7 @@
 # Versions
 
-Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go module proxy,
+Verified 2026-09-24 (phase 1), 2026-09-26 (phase 2/3 candidates) and 2026-09-27 (management
+cluster add-ons) from the Go module proxy,
 `dl.k8s.io`, `registry.k8s.io`, upstream Helm indexes and git tags. Re-verify before every deploy.
 
 ## Phase 1 — pinned
@@ -12,6 +13,9 @@ Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go
 | IPAM provider in-cluster | **v1.1.1** | v1.1.1 | CAPMOX v0.9 requires ≥ 1.1.0 |
 | Workload Kubernetes | **v1.35.9** | v1.37.1 | CAPI 1.12 supports workload clusters up to 1.35 |
 | Management cluster (k3s) | **v1.35.8+k3s1** | v1.37.0+k3s1 | CAPI 1.12 supports management clusters 1.31–1.35 |
+| cluster-api-operator (Helm chart) | **0.28.0** | 0.29.0 | installs the providers instead of `clusterctl init`; 0.28.0 is built against CAPI v1.12.10, 0.29.0 against v1.13.5 |
+| cert-manager (Helm chart) | **v1.21.2** | v1.21.2 | required by the operator and all providers (webhook certs); supports k8s 1.33–1.36 |
+| Management VM image | Ubuntu cloud image `noble/20260926` (24.04.5) | — | sha256 `6a81c375…5bb4d2`; `current/` is a rolling pointer, pin the dated serial |
 | cluster-autoscaler | **v1.35.2** | v1.36.1 | CA minor must match workload Kubernetes minor |
 | cluster-autoscaler Helm chart | **9.59.0** | 9.59.0 | default appVersion 1.35.0 → override `image.tag` |
 | Cilium | **1.20.2** | 1.20.2 | |
@@ -27,6 +31,11 @@ Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go
 When CAPMOX adds CAPI 1.13/1.14 to its compatibility table: bump CAPI core → k3s mgmt to 1.36+ →
 rebuild template with Kubernetes 1.36/1.37 → cluster-autoscaler to the matching minor.
 
+All management-cluster pins live in `ansible/group_vars/all.yml`. Provider upgrades are a
+`spec.version` bump there + `ansible-playbook mgmt.yml --tags capi`, one provider per run (the
+operator blocks minor downgrades). Bump cluster-api-operator only to a release built against the
+target CAPI minor.
+
 ## Phase 2 / 3 — candidates (not pinned yet)
 
 | Component | Latest | Built against CAPI | Notes |