diff --git a/docs/plans/.gitkeep b/docs/plans/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md b/docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md new file mode 100644 index 0000000..5f893fa --- /dev/null +++ b/docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md @@ -0,0 +1,155 @@ +# Plan: Idempotent management cluster (Ansible + cluster-api-operator) + +**Created:** 2026-09-27 00:58 + +## Context + +Phase 1 step 2 (management cluster + CAPI providers) is next. The runbook today is a series of +hand-typed commands (`curl | sh` for k3s, `clusterctl init`), which is not re-runnable and leaves +no declarative record of what the management cluster contains. Goal: one Ansible playbook, run +from the laptop, that creates the management VM on Proxmox, installs pinned k3s, cert-manager, +cluster-api-operator and the five CAPI providers, and fetches kubeconfigs. Re-running it is a +no-op (`changed=0`). Keep it simple: happy path only, no VM resizing/reconciliation, no HA, no +token rotation automation. + +Decisions made by the owner: Ansible playbook; providers via **cluster-api-operator** Provider CRs +(not `clusterctl init`); the playbook also **creates the VM from the Ubuntu 24.04 cloud image**. + +Upstream facts verified 2026-09-27 (upstream-researcher, from tags/release assets): +- cluster-api-operator **v0.28.0** (chart 0.28.0, repo `https://kubernetes-sigs.github.io/cluster-api-operator`) + is built against CAPI v1.12.10 → closest match to our v1.12.11 pin (v0.29.0 is on CAPI 1.13.5). + Only gate: provider `metadata.yaml` contract v1beta1/v1beta2; CAPI v1.12.11, CAPMOX v0.9.1, + IPAM in-cluster v1.1.1 all declare v1beta2. +- cert-manager must be pre-installed (operator doesn't manage it): **v1.21.2**, supports k8s 1.33–1.36. +- CRs are `operator.cluster.x-k8s.io/v1alpha2`; CR name = provider name, CR namespace = install + namespace (must pre-exist); `proxmox` and `in-cluster` are built-in (no `fetchConfig`). Non-core + providers wait for the CoreProvider on their own, so all five can be applied at once. +- CAPMOX v0.9.1 reads `PROXMOX_URL`, `PROXMOX_TOKEN`, `PROXMOX_SECRET` from the provider's + `configSecret`. No feature gates needed for `lab1` (no ClusterClass). +- Components are downloaded from GitHub release URLs (not the rate-limited API) and cached per version. +- Re-applying unchanged CRs is a no-op; bumping `spec.version` upgrades in place (one provider at a time). +- Gotchas: don't put providers in the operator chart values (they become Helm hooks, #1052); + stale cache secret `---cache` must be deleted by hand (#1075); `configSecret` + is not watched, so treat it as immutable (rotate = new Secret name). +- `clusterctl describe cluster` and `clusterctl generate cluster --infrastructure proxmox:v0.9.1` + keep working; **don't use `clusterctl upgrade/move/delete`** against an operator-managed cluster. +- Ubuntu cloud image pinned: `https://cloud-images.ubuntu.com/noble/20260926/noble-server-cloudimg-amd64.img`, + sha256 `6a81c37564db9b1ee84e141922625e1d7c5b389b99bb3c572e0243607d5bb4d2`. + +Assumptions (confirm at Step 1): root SSH from the laptop to `pv1-1` works; `ansible-core` is +installed on the laptop; the owner provides VMID, static IP, gateway, DNS for the mgmt VM (VLAN 4). + +## Layout + +``` +ansible/ +├── ansible.cfg # inventory=inventory.yml, interpreter_python=auto_silent +├── inventory.yml # pve: pv1-1 (root) ; mgmt: lab-mgmt (ubuntu, become) +├── group_vars/all.yml # versions + VM params (no secrets) +├── secrets.yml.example # proxmox_token_secret: "" → copy to secrets.yml (gitignored) +├── mgmt.yml # the playbook, plays tagged vm / k3s / addons / capi / kubeconfig +└── templates/ + ├── k3s-config.yaml.j2 # /etc/rancher/k3s/config.yaml + ├── helmchart-cert-manager.yaml.j2 # k3s HelmChart CR + ├── helmchart-capi-operator.yaml.j2 # k3s HelmChart CR + └── capi-providers.yaml.j2 # namespaces + capmox-variables Secret + 5 provider CRs +``` + +Only `ansible.builtin` modules, no Galaxy collections, no helm/python-kubernetes on the VM: Helm +charts are installed through k3s's built-in helm-controller (`HelmChart` CRs written to +`/var/lib/rancher/k3s/server/manifests/`), which keeps them applied and upgrades on file change. + +`.gitignore`: `ansible/secrets.yml` is already caught by `*secret*`; add `!ansible/secrets.yml.example`. + +## Playbook (`ansible/mgmt.yml`) + +**Play 1 — `vm` (hosts: pve).** Guard everything on `qm status {{ mgmt_vmid }}` (rc≠0 → create): +1. `get_url` the pinned cloud image to `/var/lib/vz/template/iso/` with `checksum: sha256:…`. +2. `copy` the SSH public key to a temp file; create + configure in one block: + `qm create --name lab-mgmt --cores 2 --memory 4096 --net0 virtio,bridge=vmbr0,tag=4 + --scsihw virtio-scsi-pci --scsi0 local-lvm:0,import-from= --ide2 local-lvm:cloudinit + --boot order=scsi0 --serial0 socket --vga serial0 --agent 1 --ciuser ubuntu --sshkeys + --ipconfig0 ip=/24,gw= --nameserver `, then `qm resize scsi0 30G`, `qm start`. + (Disk lands raw on LVM-thin; bridge + separate tag per lab rules.) +3. `wait_for` port 22 on the VM IP (delegated to localhost). + +**Play 2 — `k3s` (hosts: mgmt, become).** +1. `apt` `qemu-guest-agent` (+ service started). +2. `template` `/etc/rancher/k3s/config.yaml`: `disable: [traefik]`, `tls-san: []`, + `write-kubeconfig-mode: "0600"`; notify restart handler. +3. `command: k3s --version` (failed_when false) → if it doesn't contain `k3s_version`, + `get_url` https://get.k3s.io to `/usr/local/bin/k3s-install.sh` and run it with + `INSTALL_K3S_VERSION={{ k3s_version }}` (so a version bump in `all.yml` = upgrade). +4. `k3s kubectl wait --for=condition=Ready node --all --timeout=180s` (changed_when false). + +**Play 3 — `addons` (hosts: mgmt, become).** +1. `template` cert-manager HelmChart (repo `https://charts.jetstack.io`, `v1.21.2`, + `targetNamespace: cert-manager`, `createNamespace: true`, `crds.enabled: true`). +2. Wait: `k3s kubectl -n cert-manager rollout status deploy/cert-manager-webhook` with + `retries/until` (deployment appears only after the helm job ran). +3. `template` cluster-api-operator HelmChart (`0.28.0`, `targetNamespace: capi-operator-system`, + no provider values), wait for its deployment rollout, then + `k3s kubectl wait --for condition=established crd/coreproviders.operator.cluster.x-k8s.io`. + +**Play 4 — `capi` (hosts: mgmt, become).** +1. `template` `/opt/lab-mgmt/capi-providers.yaml` (mode 0600, holds the token secret): + namespaces `capi-system`, `capi-kubeadm-bootstrap-system`, `capi-kubeadm-control-plane-system`, + `capmox-system`, `capi-ipam-in-cluster-system`; Secret `capmox-system/capmox-variables` + (`PROXMOX_URL`, `PROXMOX_TOKEN`, `PROXMOX_SECRET`); the five CRs: + `CoreProvider cluster-api v1.12.11`, `BootstrapProvider kubeadm v1.12.11`, + `ControlPlaneProvider kubeadm v1.12.11`, `InfrastructureProvider proxmox v0.9.1` + (`configSecret: {name: capmox-variables}`), `IPAMProvider in-cluster v1.1.1`. +2. `k3s kubectl apply -f` it; `changed_when` on `created|configured` in stdout. +3. `k3s kubectl wait --for=condition=Ready / -n --timeout=10m` for all five + (condition name to be confirmed on first run; fall back to deployment rollout status). + +**Play 5 — `kubeconfig` (hosts: mgmt → localhost).** +1. `fetch` `/etc/rancher/k3s/k3s.yaml` to `~/.kube/lab-mgmt.kubeconfig` (outside the repo), + `replace` `127.0.0.1` with the VM IP, context renamed `lab-mgmt`, mode 0600. +2. Run `.claude/cluster-inspector/make-kubeconfig.sh lab-mgmt --admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig --apply-rbac` + (tagged `inspector`, `changed_when: false`; it re-mints the 7-day token each run). + +## Execution steps + +- **Step 0** — Repo has no commits yet: commit the existing tree as the initial import, then + copy this plan to `docs/plans/-mgmt-cluster-ansible.md` and commit it alone. +- **Step 1** — Scaffold `ansible/` (all files above), `.gitignore` exception, owner fills VMID/IP/ + gw/DNS in `group_vars/all.yml` and `secrets.yml`. Check: `ansible-playbook mgmt.yml --syntax-check`, + `ansible -m ping all` (reaches pv1-1; mgmt unreachable until Play 1 ran). +- **Step 2** — Owner runs `ansible-playbook mgmt.yml --tags vm`; then `--tags k3s`; + then `--tags addons`; then `--tags capi`; then `--tags kubeconfig,inspector`. Per CLAUDE.md + safety rules the owner runs lab mutations (or explicitly tells Claude to); Claude diagnoses + failures via the `cluster-inspector` agent and records exact commands/outputs in the execution log. +- **Step 3** — Idempotency check: full `ansible-playbook mgmt.yml` twice; second run's recap must + show `changed=0` (fix any task that reports changed spuriously). +- **Step 4** — Docs: rewrite phase-1 Step 2 around the playbook (prereqs: ansible-core, pinned + clusterctl v1.12.11 **darwin** binary on the laptop for Step 3's `generate`/`describe`; + remove `clusterctl init`; note "no `clusterctl upgrade/move/delete`"); add troubleshooting rows + (provider CR not Ready → operator logs; stale cache secret #1075); `versions.md` rows for + cluster-api-operator 0.28.0, cert-manager v1.21.2, Ubuntu cloud image serial 20260926 + upgrade + order note (bump provider CRs one at a time); `configs/capi/clusterctl.yaml.example`: controller + credentials section marked as unused (now in `ansible/secrets.yml`); CLAUDE.md version rule + "explicit versions to `clusterctl init`" → "…in `group_vars/all.yml` / provider CRs"; + `docs/decisions.md` rows: Ansible for mgmt setup (alt: cloud-init + k3s manifests, bash, + OpenTofu), cluster-api-operator (alt: guarded `clusterctl init`), mgmt VM from cloud image + (alt: manual VM). Tick phase-1 checklist Step 2 and README status only after Step 3 passes; + CHANGELOG entry after the owner confirms. + +## Critical files + +- New: `ansible/**` (above) +- Edit: `.gitignore`, `docs/phase-1-proxmox-capi.md` (Step 2, troubleshooting), `docs/versions.md`, + `docs/decisions.md`, `configs/capi/clusterctl.yaml.example`, `CLAUDE.md` (version rules), `README.md` (status) +- Reuse: `.claude/cluster-inspector/make-kubeconfig.sh` + `rbac.yaml` (Play 5), existing Proxmox + facts from `configs/capi/clusterctl.yaml.example` (URL, token id, node, bridge/VLAN) + +## Verification + +1. `ansible-playbook mgmt.yml --syntax-check` passes. +2. After the full run: `KUBECONFIG=~/.kube/lab-mgmt.kubeconfig kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A` + → all Ready with the pinned versions; `kubectl get pods -A` → capi/capmox/ipam/cert-manager/operator Running. +3. `kubectl -n capmox-system logs deploy/capmox-controller-manager` shows no Proxmox auth errors. +4. `clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --kubernetes-version v1.35.9 ... --list-variables` + works against the new cluster (readiness for phase-1 Step 3). +5. Second full playbook run: `changed=0`. +6. `cluster-inspector` agent can read the cluster with `lab-mgmt-ro` (`can-i` output as in its README).