# Plan: Idempotent management cluster (Ansible + cluster-api-operator)

**Created:** 2026-09-27 00:58

## Context

Phase 1 step 2 (management cluster + CAPI providers) is next. The runbook today is a series of
hand-typed commands (`curl | sh` for k3s, `clusterctl init`), which is not re-runnable and leaves
no declarative record of what the management cluster contains. Goal: one Ansible playbook, run
from the laptop, that creates the management VM on Proxmox, installs pinned k3s, cert-manager,
cluster-api-operator and the five CAPI providers, and fetches kubeconfigs. Re-running it is a
no-op (`changed=0`). Keep it simple: happy path only, no VM resizing/reconciliation, no HA, no
token rotation automation.

Decisions made by the owner: Ansible playbook; providers via **cluster-api-operator** Provider CRs
(not `clusterctl init`); the playbook also **creates the VM from the Ubuntu 24.04 cloud image**.

Upstream facts verified 2026-09-27 (upstream-researcher, from tags/release assets):
- cluster-api-operator **v0.28.0** (chart 0.28.0, repo `https://kubernetes-sigs.github.io/cluster-api-operator`)
  is built against CAPI v1.12.10 → closest match to our v1.12.11 pin (v0.29.0 is on CAPI 1.13.5).
  Only gate: provider `metadata.yaml` contract v1beta1/v1beta2; CAPI v1.12.11, CAPMOX v0.9.1,
  IPAM in-cluster v1.1.1 all declare v1beta2.
- cert-manager must be pre-installed (operator doesn't manage it): **v1.21.2**, supports k8s 1.33–1.36.
- CRs are `operator.cluster.x-k8s.io/v1alpha2`; CR name = provider name, CR namespace = install
  namespace (must pre-exist); `proxmox` and `in-cluster` are built-in (no `fetchConfig`). Non-core
  providers wait for the CoreProvider on their own, so all five can be applied at once.
- CAPMOX v0.9.1 reads `PROXMOX_URL`, `PROXMOX_TOKEN`, `PROXMOX_SECRET` from the provider's
  `configSecret`. No feature gates needed for `lab1` (no ClusterClass).
- Components are downloaded from GitHub release URLs (not the rate-limited API) and cached per version.
- Re-applying unchanged CRs is a no-op; bumping `spec.version` upgrades in place (one provider at a time).
- Gotchas: don't put providers in the operator chart values (they become Helm hooks, #1052);
  stale cache secret `<type>-<name>-<ver>-cache` must be deleted by hand (#1075); `configSecret`
  is not watched, so treat it as immutable (rotate = new Secret name).
- `clusterctl describe cluster` and `clusterctl generate cluster --infrastructure proxmox:v0.9.1`
  keep working; **don't use `clusterctl upgrade/move/delete`** against an operator-managed cluster.
- Ubuntu cloud image pinned: `https://cloud-images.ubuntu.com/noble/20260926/noble-server-cloudimg-amd64.img`,
  sha256 `6a81c37564db9b1ee84e141922625e1d7c5b389b99bb3c572e0243607d5bb4d2`.

Assumptions (confirm at Step 1): root SSH from the laptop to `pv1-1` works; `ansible-core` is
installed on the laptop; the owner provides VMID, static IP, gateway, DNS for the mgmt VM (VLAN 4).

## Layout

```
ansible/
├── ansible.cfg                 # inventory=inventory.yml, interpreter_python=auto_silent
├── inventory.yml               # pve: pv1-1 (root) ; mgmt: lab-mgmt (ubuntu, become)
├── group_vars/all.yml          # versions + VM params (no secrets)
├── secrets.yml.example         # proxmox_token_secret: "<secret>"  → copy to secrets.yml (gitignored)
├── mgmt.yml                    # the playbook, plays tagged vm / k3s / addons / capi / kubeconfig
└── templates/
    ├── k3s-config.yaml.j2              # /etc/rancher/k3s/config.yaml
    ├── helmchart-cert-manager.yaml.j2  # k3s HelmChart CR
    ├── helmchart-capi-operator.yaml.j2 # k3s HelmChart CR
    └── capi-providers.yaml.j2          # namespaces + capmox-variables Secret + 5 provider CRs
```

Only `ansible.builtin` modules, no Galaxy collections, no helm/python-kubernetes on the VM: Helm
charts are installed through k3s's built-in helm-controller (`HelmChart` CRs written to
`/var/lib/rancher/k3s/server/manifests/`), which keeps them applied and upgrades on file change.

`.gitignore`: `ansible/secrets.yml` is already caught by `*secret*`; add `!ansible/secrets.yml.example`.

## Playbook (`ansible/mgmt.yml`)

**Play 1 — `vm` (hosts: pve).** Guard everything on `qm status {{ mgmt_vmid }}` (rc≠0 → create):
1. `get_url` the pinned cloud image to `/var/lib/vz/template/iso/` with `checksum: sha256:…`.
2. `copy` the SSH public key to a temp file; create + configure in one block:
   `qm create <vmid> --name lab-mgmt --cores 2 --memory 4096 --net0 virtio,bridge=vmbr0,tag=4
   --scsihw virtio-scsi-pci --scsi0 local-lvm:0,import-from=<img> --ide2 local-lvm:cloudinit
   --boot order=scsi0 --serial0 socket --vga serial0 --agent 1 --ciuser ubuntu --sshkeys <file>
   --ipconfig0 ip=<ip>/24,gw=<gw> --nameserver <dns>`, then `qm resize <vmid> scsi0 30G`, `qm start`.
   (Disk lands raw on LVM-thin; bridge + separate tag per lab rules.)
3. `wait_for` port 22 on the VM IP (delegated to localhost).

**Play 2 — `k3s` (hosts: mgmt, become).**
1. `apt` `qemu-guest-agent` (+ service started).
2. `template` `/etc/rancher/k3s/config.yaml`: `disable: [traefik]`, `tls-san: [<ip>]`,
   `write-kubeconfig-mode: "0600"`; notify restart handler.
3. `command: k3s --version` (failed_when false) → if it doesn't contain `k3s_version`,
   `get_url` https://get.k3s.io to `/usr/local/bin/k3s-install.sh` and run it with
   `INSTALL_K3S_VERSION={{ k3s_version }}` (so a version bump in `all.yml` = upgrade).
4. `k3s kubectl wait --for=condition=Ready node --all --timeout=180s` (changed_when false).

**Play 3 — `addons` (hosts: mgmt, become).**
1. `template` cert-manager HelmChart (repo `https://charts.jetstack.io`, `v1.21.2`,
   `targetNamespace: cert-manager`, `createNamespace: true`, `crds.enabled: true`).
2. Wait: `k3s kubectl -n cert-manager rollout status deploy/cert-manager-webhook` with
   `retries/until` (deployment appears only after the helm job ran).
3. `template` cluster-api-operator HelmChart (`0.28.0`, `targetNamespace: capi-operator-system`,
   no provider values), wait for its deployment rollout, then
   `k3s kubectl wait --for condition=established crd/coreproviders.operator.cluster.x-k8s.io`.

**Play 4 — `capi` (hosts: mgmt, become).**
1. `template` `/opt/lab-mgmt/capi-providers.yaml` (mode 0600, holds the token secret):
   namespaces `capi-system`, `capi-kubeadm-bootstrap-system`, `capi-kubeadm-control-plane-system`,
   `capmox-system`, `capi-ipam-in-cluster-system`; Secret `capmox-system/capmox-variables`
   (`PROXMOX_URL`, `PROXMOX_TOKEN`, `PROXMOX_SECRET`); the five CRs:
   `CoreProvider cluster-api v1.12.11`, `BootstrapProvider kubeadm v1.12.11`,
   `ControlPlaneProvider kubeadm v1.12.11`, `InfrastructureProvider proxmox v0.9.1`
   (`configSecret: {name: capmox-variables}`), `IPAMProvider in-cluster v1.1.1`.
2. `k3s kubectl apply -f` it; `changed_when` on `created|configured` in stdout.
3. `k3s kubectl wait --for=condition=Ready <kind>/<name> -n <ns> --timeout=10m` for all five
   (condition name to be confirmed on first run; fall back to deployment rollout status).

**Play 5 — `kubeconfig` (hosts: mgmt → localhost).**
1. `fetch` `/etc/rancher/k3s/k3s.yaml` to `~/.kube/lab-mgmt.kubeconfig` (outside the repo),
   `replace` `127.0.0.1` with the VM IP, context renamed `lab-mgmt`, mode 0600.
2. Run `.claude/cluster-inspector/make-kubeconfig.sh lab-mgmt --admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig --apply-rbac`
   (tagged `inspector`, `changed_when: false`; it re-mints the 7-day token each run).

## Execution steps

- **Step 0** — Repo has no commits yet: commit the existing tree as the initial import, then
  copy this plan to `docs/plans/<ts>-mgmt-cluster-ansible.md` and commit it alone.
- **Step 1** — Scaffold `ansible/` (all files above), `.gitignore` exception, owner fills VMID/IP/
  gw/DNS in `group_vars/all.yml` and `secrets.yml`. Check: `ansible-playbook mgmt.yml --syntax-check`,
  `ansible -m ping all` (reaches pv1-1; mgmt unreachable until Play 1 ran).
- **Step 2** — Owner runs `ansible-playbook mgmt.yml --tags vm`; then `--tags k3s`;
  then `--tags addons`; then `--tags capi`; then `--tags kubeconfig,inspector`. Per CLAUDE.md
  safety rules the owner runs lab mutations (or explicitly tells Claude to); Claude diagnoses
  failures via the `cluster-inspector` agent and records exact commands/outputs in the execution log.
- **Step 3** — Idempotency check: full `ansible-playbook mgmt.yml` twice; second run's recap must
  show `changed=0` (fix any task that reports changed spuriously).
- **Step 4** — Docs: rewrite phase-1 Step 2 around the playbook (prereqs: ansible-core, pinned
  clusterctl v1.12.11 **darwin** binary on the laptop for Step 3's `generate`/`describe`;
  remove `clusterctl init`; note "no `clusterctl upgrade/move/delete`"); add troubleshooting rows
  (provider CR not Ready → operator logs; stale cache secret #1075); `versions.md` rows for
  cluster-api-operator 0.28.0, cert-manager v1.21.2, Ubuntu cloud image serial 20260926 + upgrade
  order note (bump provider CRs one at a time); `configs/capi/clusterctl.yaml.example`: controller
  credentials section marked as unused (now in `ansible/secrets.yml`); CLAUDE.md version rule
  "explicit versions to `clusterctl init`" → "…in `group_vars/all.yml` / provider CRs";
  `docs/decisions.md` rows: Ansible for mgmt setup (alt: cloud-init + k3s manifests, bash,
  OpenTofu), cluster-api-operator (alt: guarded `clusterctl init`), mgmt VM from cloud image
  (alt: manual VM). Tick phase-1 checklist Step 2 and README status only after Step 3 passes;
  CHANGELOG entry after the owner confirms.

## Critical files

- New: `ansible/**` (above)
- Edit: `.gitignore`, `docs/phase-1-proxmox-capi.md` (Step 2, troubleshooting), `docs/versions.md`,
  `docs/decisions.md`, `configs/capi/clusterctl.yaml.example`, `CLAUDE.md` (version rules), `README.md` (status)
- Reuse: `.claude/cluster-inspector/make-kubeconfig.sh` + `rbac.yaml` (Play 5), existing Proxmox
  facts from `configs/capi/clusterctl.yaml.example` (URL, token id, node, bridge/VLAN)

## Verification

1. `ansible-playbook mgmt.yml --syntax-check` passes.
2. After the full run: `KUBECONFIG=~/.kube/lab-mgmt.kubeconfig kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A`
   → all Ready with the pinned versions; `kubectl get pods -A` → capi/capmox/ipam/cert-manager/operator Running.
3. `kubectl -n capmox-system logs deploy/capmox-controller-manager` shows no Proxmox auth errors.
4. `clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --kubernetes-version v1.35.9 ... --list-variables`
   works against the new cluster (readiness for phase-1 Step 3).
5. Second full playbook run: `changed=0`.
6. `cluster-inspector` agent can read the cluster with `lab-mgmt-ro` (`can-i` output as in its README).