Add plan: idempotent management cluster (Ansible + cluster-api-operator)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 00:58:57 +02:00
parent 5fd998c5f9
commit ac7d5a45b9
2 changed files with 155 additions and 0 deletions
@@ -0,0 +1,155 @@
+# Plan: Idempotent management cluster (Ansible + cluster-api-operator)
+
+**Created:** 2026-09-27 00:58
+
+## Context
+
+Phase 1 step 2 (management cluster + CAPI providers) is next. The runbook today is a series of
+hand-typed commands (`curl | sh` for k3s, `clusterctl init`), which is not re-runnable and leaves
+no declarative record of what the management cluster contains. Goal: one Ansible playbook, run
+from the laptop, that creates the management VM on Proxmox, installs pinned k3s, cert-manager,
+cluster-api-operator and the five CAPI providers, and fetches kubeconfigs. Re-running it is a
+no-op (`changed=0`). Keep it simple: happy path only, no VM resizing/reconciliation, no HA, no
+token rotation automation.
+
+Decisions made by the owner: Ansible playbook; providers via **cluster-api-operator** Provider CRs
+(not `clusterctl init`); the playbook also **creates the VM from the Ubuntu 24.04 cloud image**.
+
+Upstream facts verified 2026-09-27 (upstream-researcher, from tags/release assets):
+- cluster-api-operator **v0.28.0** (chart 0.28.0, repo `https://kubernetes-sigs.github.io/cluster-api-operator`)
+  is built against CAPI v1.12.10 → closest match to our v1.12.11 pin (v0.29.0 is on CAPI 1.13.5).
+  Only gate: provider `metadata.yaml` contract v1beta1/v1beta2; CAPI v1.12.11, CAPMOX v0.9.1,
+  IPAM in-cluster v1.1.1 all declare v1beta2.
+- cert-manager must be pre-installed (operator doesn't manage it): **v1.21.2**, supports k8s 1.33–1.36.
+- CRs are `operator.cluster.x-k8s.io/v1alpha2`; CR name = provider name, CR namespace = install
+  namespace (must pre-exist); `proxmox` and `in-cluster` are built-in (no `fetchConfig`). Non-core
+  providers wait for the CoreProvider on their own, so all five can be applied at once.
+- CAPMOX v0.9.1 reads `PROXMOX_URL`, `PROXMOX_TOKEN`, `PROXMOX_SECRET` from the provider's
+  `configSecret`. No feature gates needed for `lab1` (no ClusterClass).
+- Components are downloaded from GitHub release URLs (not the rate-limited API) and cached per version.
+- Re-applying unchanged CRs is a no-op; bumping `spec.version` upgrades in place (one provider at a time).
+- Gotchas: don't put providers in the operator chart values (they become Helm hooks, #1052);
+  stale cache secret `<type>-<name>-<ver>-cache` must be deleted by hand (#1075); `configSecret`
+  is not watched, so treat it as immutable (rotate = new Secret name).
+- `clusterctl describe cluster` and `clusterctl generate cluster --infrastructure proxmox:v0.9.1`
+  keep working; **don't use `clusterctl upgrade/move/delete`** against an operator-managed cluster.
+- Ubuntu cloud image pinned: `https://cloud-images.ubuntu.com/noble/20260926/noble-server-cloudimg-amd64.img`,
+  sha256 `6a81c37564db9b1ee84e141922625e1d7c5b389b99bb3c572e0243607d5bb4d2`.
+
+Assumptions (confirm at Step 1): root SSH from the laptop to `pv1-1` works; `ansible-core` is
+installed on the laptop; the owner provides VMID, static IP, gateway, DNS for the mgmt VM (VLAN 4).
+
+## Layout
+
+```
+ansible/
+├── ansible.cfg                 # inventory=inventory.yml, interpreter_python=auto_silent
+├── inventory.yml               # pve: pv1-1 (root) ; mgmt: lab-mgmt (ubuntu, become)
+├── group_vars/all.yml          # versions + VM params (no secrets)
+├── secrets.yml.example         # proxmox_token_secret: "<secret>"  → copy to secrets.yml (gitignored)
+├── mgmt.yml                    # the playbook, plays tagged vm / k3s / addons / capi / kubeconfig
+└── templates/
+    ├── k3s-config.yaml.j2              # /etc/rancher/k3s/config.yaml
+    ├── helmchart-cert-manager.yaml.j2  # k3s HelmChart CR
+    ├── helmchart-capi-operator.yaml.j2 # k3s HelmChart CR
+    └── capi-providers.yaml.j2          # namespaces + capmox-variables Secret + 5 provider CRs
+```
+
+Only `ansible.builtin` modules, no Galaxy collections, no helm/python-kubernetes on the VM: Helm
+charts are installed through k3s's built-in helm-controller (`HelmChart` CRs written to
+`/var/lib/rancher/k3s/server/manifests/`), which keeps them applied and upgrades on file change.
+
+`.gitignore`: `ansible/secrets.yml` is already caught by `*secret*`; add `!ansible/secrets.yml.example`.
+
+## Playbook (`ansible/mgmt.yml`)
+
+**Play 1 — `vm` (hosts: pve).** Guard everything on `qm status {{ mgmt_vmid }}` (rc≠0 → create):
+1. `get_url` the pinned cloud image to `/var/lib/vz/template/iso/` with `checksum: sha256:…`.
+2. `copy` the SSH public key to a temp file; create + configure in one block:
+   `qm create <vmid> --name lab-mgmt --cores 2 --memory 4096 --net0 virtio,bridge=vmbr0,tag=4
+   --scsihw virtio-scsi-pci --scsi0 local-lvm:0,import-from=<img> --ide2 local-lvm:cloudinit
+   --boot order=scsi0 --serial0 socket --vga serial0 --agent 1 --ciuser ubuntu --sshkeys <file>
+   --ipconfig0 ip=<ip>/24,gw=<gw> --nameserver <dns>`, then `qm resize <vmid> scsi0 30G`, `qm start`.
+   (Disk lands raw on LVM-thin; bridge + separate tag per lab rules.)
+3. `wait_for` port 22 on the VM IP (delegated to localhost).
+
+**Play 2 — `k3s` (hosts: mgmt, become).**
+1. `apt` `qemu-guest-agent` (+ service started).
+2. `template` `/etc/rancher/k3s/config.yaml`: `disable: [traefik]`, `tls-san: [<ip>]`,
+   `write-kubeconfig-mode: "0600"`; notify restart handler.
+3. `command: k3s --version` (failed_when false) → if it doesn't contain `k3s_version`,
+   `get_url` https://get.k3s.io to `/usr/local/bin/k3s-install.sh` and run it with
+   `INSTALL_K3S_VERSION={{ k3s_version }}` (so a version bump in `all.yml` = upgrade).
+4. `k3s kubectl wait --for=condition=Ready node --all --timeout=180s` (changed_when false).
+
+**Play 3 — `addons` (hosts: mgmt, become).**
+1. `template` cert-manager HelmChart (repo `https://charts.jetstack.io`, `v1.21.2`,
+   `targetNamespace: cert-manager`, `createNamespace: true`, `crds.enabled: true`).
+2. Wait: `k3s kubectl -n cert-manager rollout status deploy/cert-manager-webhook` with
+   `retries/until` (deployment appears only after the helm job ran).
+3. `template` cluster-api-operator HelmChart (`0.28.0`, `targetNamespace: capi-operator-system`,
+   no provider values), wait for its deployment rollout, then
+   `k3s kubectl wait --for condition=established crd/coreproviders.operator.cluster.x-k8s.io`.
+
+**Play 4 — `capi` (hosts: mgmt, become).**
+1. `template` `/opt/lab-mgmt/capi-providers.yaml` (mode 0600, holds the token secret):
+   namespaces `capi-system`, `capi-kubeadm-bootstrap-system`, `capi-kubeadm-control-plane-system`,
+   `capmox-system`, `capi-ipam-in-cluster-system`; Secret `capmox-system/capmox-variables`
+   (`PROXMOX_URL`, `PROXMOX_TOKEN`, `PROXMOX_SECRET`); the five CRs:
+   `CoreProvider cluster-api v1.12.11`, `BootstrapProvider kubeadm v1.12.11`,
+   `ControlPlaneProvider kubeadm v1.12.11`, `InfrastructureProvider proxmox v0.9.1`
+   (`configSecret: {name: capmox-variables}`), `IPAMProvider in-cluster v1.1.1`.
+2. `k3s kubectl apply -f` it; `changed_when` on `created|configured` in stdout.
+3. `k3s kubectl wait --for=condition=Ready <kind>/<name> -n <ns> --timeout=10m` for all five
+   (condition name to be confirmed on first run; fall back to deployment rollout status).
+
+**Play 5 — `kubeconfig` (hosts: mgmt → localhost).**
+1. `fetch` `/etc/rancher/k3s/k3s.yaml` to `~/.kube/lab-mgmt.kubeconfig` (outside the repo),
+   `replace` `127.0.0.1` with the VM IP, context renamed `lab-mgmt`, mode 0600.
+2. Run `.claude/cluster-inspector/make-kubeconfig.sh lab-mgmt --admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig --apply-rbac`
+   (tagged `inspector`, `changed_when: false`; it re-mints the 7-day token each run).
+
+## Execution steps
+
+- **Step 0** — Repo has no commits yet: commit the existing tree as the initial import, then
+  copy this plan to `docs/plans/<ts>-mgmt-cluster-ansible.md` and commit it alone.
+- **Step 1** — Scaffold `ansible/` (all files above), `.gitignore` exception, owner fills VMID/IP/
+  gw/DNS in `group_vars/all.yml` and `secrets.yml`. Check: `ansible-playbook mgmt.yml --syntax-check`,
+  `ansible -m ping all` (reaches pv1-1; mgmt unreachable until Play 1 ran).
+- **Step 2** — Owner runs `ansible-playbook mgmt.yml --tags vm`; then `--tags k3s`;
+  then `--tags addons`; then `--tags capi`; then `--tags kubeconfig,inspector`. Per CLAUDE.md
+  safety rules the owner runs lab mutations (or explicitly tells Claude to); Claude diagnoses
+  failures via the `cluster-inspector` agent and records exact commands/outputs in the execution log.
+- **Step 3** — Idempotency check: full `ansible-playbook mgmt.yml` twice; second run's recap must
+  show `changed=0` (fix any task that reports changed spuriously).
+- **Step 4** — Docs: rewrite phase-1 Step 2 around the playbook (prereqs: ansible-core, pinned
+  clusterctl v1.12.11 **darwin** binary on the laptop for Step 3's `generate`/`describe`;
+  remove `clusterctl init`; note "no `clusterctl upgrade/move/delete`"); add troubleshooting rows
+  (provider CR not Ready → operator logs; stale cache secret #1075); `versions.md` rows for
+  cluster-api-operator 0.28.0, cert-manager v1.21.2, Ubuntu cloud image serial 20260926 + upgrade
+  order note (bump provider CRs one at a time); `configs/capi/clusterctl.yaml.example`: controller
+  credentials section marked as unused (now in `ansible/secrets.yml`); CLAUDE.md version rule
+  "explicit versions to `clusterctl init`" → "…in `group_vars/all.yml` / provider CRs";
+  `docs/decisions.md` rows: Ansible for mgmt setup (alt: cloud-init + k3s manifests, bash,
+  OpenTofu), cluster-api-operator (alt: guarded `clusterctl init`), mgmt VM from cloud image
+  (alt: manual VM). Tick phase-1 checklist Step 2 and README status only after Step 3 passes;
+  CHANGELOG entry after the owner confirms.
+
+## Critical files
+
+- New: `ansible/**` (above)
+- Edit: `.gitignore`, `docs/phase-1-proxmox-capi.md` (Step 2, troubleshooting), `docs/versions.md`,
+  `docs/decisions.md`, `configs/capi/clusterctl.yaml.example`, `CLAUDE.md` (version rules), `README.md` (status)
+- Reuse: `.claude/cluster-inspector/make-kubeconfig.sh` + `rbac.yaml` (Play 5), existing Proxmox
+  facts from `configs/capi/clusterctl.yaml.example` (URL, token id, node, bridge/VLAN)
+
+## Verification
+
+1. `ansible-playbook mgmt.yml --syntax-check` passes.
+2. After the full run: `KUBECONFIG=~/.kube/lab-mgmt.kubeconfig kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A`
+   → all Ready with the pinned versions; `kubectl get pods -A` → capi/capmox/ipam/cert-manager/operator Running.
+3. `kubectl -n capmox-system logs deploy/capmox-controller-manager` shows no Proxmox auth errors.
+4. `clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --kubernetes-version v1.35.9 ... --list-variables`
+   works against the new cluster (readiness for phase-1 Step 3).
+5. Second full playbook run: `changed=0`.
+6. `cluster-inspector` agent can read the cluster with `lab-mgmt-ro` (`can-i` output as in its README).