156 lines
11 KiB
Markdown
156 lines
11 KiB
Markdown
# Plan: Idempotent management cluster (Ansible + cluster-api-operator)
|
||
|
||
**Created:** 2026-09-27 00:58
|
||
|
||
## Context
|
||
|
||
Phase 1 step 2 (management cluster + CAPI providers) is next. The runbook today is a series of
|
||
hand-typed commands (`curl | sh` for k3s, `clusterctl init`), which is not re-runnable and leaves
|
||
no declarative record of what the management cluster contains. Goal: one Ansible playbook, run
|
||
from the laptop, that creates the management VM on Proxmox, installs pinned k3s, cert-manager,
|
||
cluster-api-operator and the five CAPI providers, and fetches kubeconfigs. Re-running it is a
|
||
no-op (`changed=0`). Keep it simple: happy path only, no VM resizing/reconciliation, no HA, no
|
||
token rotation automation.
|
||
|
||
Decisions made by the owner: Ansible playbook; providers via **cluster-api-operator** Provider CRs
|
||
(not `clusterctl init`); the playbook also **creates the VM from the Ubuntu 24.04 cloud image**.
|
||
|
||
Upstream facts verified 2026-09-27 (upstream-researcher, from tags/release assets):
|
||
- cluster-api-operator **v0.28.0** (chart 0.28.0, repo `https://kubernetes-sigs.github.io/cluster-api-operator`)
|
||
is built against CAPI v1.12.10 → closest match to our v1.12.11 pin (v0.29.0 is on CAPI 1.13.5).
|
||
Only gate: provider `metadata.yaml` contract v1beta1/v1beta2; CAPI v1.12.11, CAPMOX v0.9.1,
|
||
IPAM in-cluster v1.1.1 all declare v1beta2.
|
||
- cert-manager must be pre-installed (operator doesn't manage it): **v1.21.2**, supports k8s 1.33–1.36.
|
||
- CRs are `operator.cluster.x-k8s.io/v1alpha2`; CR name = provider name, CR namespace = install
|
||
namespace (must pre-exist); `proxmox` and `in-cluster` are built-in (no `fetchConfig`). Non-core
|
||
providers wait for the CoreProvider on their own, so all five can be applied at once.
|
||
- CAPMOX v0.9.1 reads `PROXMOX_URL`, `PROXMOX_TOKEN`, `PROXMOX_SECRET` from the provider's
|
||
`configSecret`. No feature gates needed for `lab1` (no ClusterClass).
|
||
- Components are downloaded from GitHub release URLs (not the rate-limited API) and cached per version.
|
||
- Re-applying unchanged CRs is a no-op; bumping `spec.version` upgrades in place (one provider at a time).
|
||
- Gotchas: don't put providers in the operator chart values (they become Helm hooks, #1052);
|
||
stale cache secret `<type>-<name>-<ver>-cache` must be deleted by hand (#1075); `configSecret`
|
||
is not watched, so treat it as immutable (rotate = new Secret name).
|
||
- `clusterctl describe cluster` and `clusterctl generate cluster --infrastructure proxmox:v0.9.1`
|
||
keep working; **don't use `clusterctl upgrade/move/delete`** against an operator-managed cluster.
|
||
- Ubuntu cloud image pinned: `https://cloud-images.ubuntu.com/noble/20260926/noble-server-cloudimg-amd64.img`,
|
||
sha256 `6a81c37564db9b1ee84e141922625e1d7c5b389b99bb3c572e0243607d5bb4d2`.
|
||
|
||
Assumptions (confirm at Step 1): root SSH from the laptop to `pv1-1` works; `ansible-core` is
|
||
installed on the laptop; the owner provides VMID, static IP, gateway, DNS for the mgmt VM (VLAN 4).
|
||
|
||
## Layout
|
||
|
||
```
|
||
ansible/
|
||
├── ansible.cfg # inventory=inventory.yml, interpreter_python=auto_silent
|
||
├── inventory.yml # pve: pv1-1 (root) ; mgmt: lab-mgmt (ubuntu, become)
|
||
├── group_vars/all.yml # versions + VM params (no secrets)
|
||
├── secrets.yml.example # proxmox_token_secret: "<secret>" → copy to secrets.yml (gitignored)
|
||
├── mgmt.yml # the playbook, plays tagged vm / k3s / addons / capi / kubeconfig
|
||
└── templates/
|
||
├── k3s-config.yaml.j2 # /etc/rancher/k3s/config.yaml
|
||
├── helmchart-cert-manager.yaml.j2 # k3s HelmChart CR
|
||
├── helmchart-capi-operator.yaml.j2 # k3s HelmChart CR
|
||
└── capi-providers.yaml.j2 # namespaces + capmox-variables Secret + 5 provider CRs
|
||
```
|
||
|
||
Only `ansible.builtin` modules, no Galaxy collections, no helm/python-kubernetes on the VM: Helm
|
||
charts are installed through k3s's built-in helm-controller (`HelmChart` CRs written to
|
||
`/var/lib/rancher/k3s/server/manifests/`), which keeps them applied and upgrades on file change.
|
||
|
||
`.gitignore`: `ansible/secrets.yml` is already caught by `*secret*`; add `!ansible/secrets.yml.example`.
|
||
|
||
## Playbook (`ansible/mgmt.yml`)
|
||
|
||
**Play 1 — `vm` (hosts: pve).** Guard everything on `qm status {{ mgmt_vmid }}` (rc≠0 → create):
|
||
1. `get_url` the pinned cloud image to `/var/lib/vz/template/iso/` with `checksum: sha256:…`.
|
||
2. `copy` the SSH public key to a temp file; create + configure in one block:
|
||
`qm create <vmid> --name lab-mgmt --cores 2 --memory 4096 --net0 virtio,bridge=vmbr0,tag=4
|
||
--scsihw virtio-scsi-pci --scsi0 local-lvm:0,import-from=<img> --ide2 local-lvm:cloudinit
|
||
--boot order=scsi0 --serial0 socket --vga serial0 --agent 1 --ciuser ubuntu --sshkeys <file>
|
||
--ipconfig0 ip=<ip>/24,gw=<gw> --nameserver <dns>`, then `qm resize <vmid> scsi0 30G`, `qm start`.
|
||
(Disk lands raw on LVM-thin; bridge + separate tag per lab rules.)
|
||
3. `wait_for` port 22 on the VM IP (delegated to localhost).
|
||
|
||
**Play 2 — `k3s` (hosts: mgmt, become).**
|
||
1. `apt` `qemu-guest-agent` (+ service started).
|
||
2. `template` `/etc/rancher/k3s/config.yaml`: `disable: [traefik]`, `tls-san: [<ip>]`,
|
||
`write-kubeconfig-mode: "0600"`; notify restart handler.
|
||
3. `command: k3s --version` (failed_when false) → if it doesn't contain `k3s_version`,
|
||
`get_url` https://get.k3s.io to `/usr/local/bin/k3s-install.sh` and run it with
|
||
`INSTALL_K3S_VERSION={{ k3s_version }}` (so a version bump in `all.yml` = upgrade).
|
||
4. `k3s kubectl wait --for=condition=Ready node --all --timeout=180s` (changed_when false).
|
||
|
||
**Play 3 — `addons` (hosts: mgmt, become).**
|
||
1. `template` cert-manager HelmChart (repo `https://charts.jetstack.io`, `v1.21.2`,
|
||
`targetNamespace: cert-manager`, `createNamespace: true`, `crds.enabled: true`).
|
||
2. Wait: `k3s kubectl -n cert-manager rollout status deploy/cert-manager-webhook` with
|
||
`retries/until` (deployment appears only after the helm job ran).
|
||
3. `template` cluster-api-operator HelmChart (`0.28.0`, `targetNamespace: capi-operator-system`,
|
||
no provider values), wait for its deployment rollout, then
|
||
`k3s kubectl wait --for condition=established crd/coreproviders.operator.cluster.x-k8s.io`.
|
||
|
||
**Play 4 — `capi` (hosts: mgmt, become).**
|
||
1. `template` `/opt/lab-mgmt/capi-providers.yaml` (mode 0600, holds the token secret):
|
||
namespaces `capi-system`, `capi-kubeadm-bootstrap-system`, `capi-kubeadm-control-plane-system`,
|
||
`capmox-system`, `capi-ipam-in-cluster-system`; Secret `capmox-system/capmox-variables`
|
||
(`PROXMOX_URL`, `PROXMOX_TOKEN`, `PROXMOX_SECRET`); the five CRs:
|
||
`CoreProvider cluster-api v1.12.11`, `BootstrapProvider kubeadm v1.12.11`,
|
||
`ControlPlaneProvider kubeadm v1.12.11`, `InfrastructureProvider proxmox v0.9.1`
|
||
(`configSecret: {name: capmox-variables}`), `IPAMProvider in-cluster v1.1.1`.
|
||
2. `k3s kubectl apply -f` it; `changed_when` on `created|configured` in stdout.
|
||
3. `k3s kubectl wait --for=condition=Ready <kind>/<name> -n <ns> --timeout=10m` for all five
|
||
(condition name to be confirmed on first run; fall back to deployment rollout status).
|
||
|
||
**Play 5 — `kubeconfig` (hosts: mgmt → localhost).**
|
||
1. `fetch` `/etc/rancher/k3s/k3s.yaml` to `~/.kube/lab-mgmt.kubeconfig` (outside the repo),
|
||
`replace` `127.0.0.1` with the VM IP, context renamed `lab-mgmt`, mode 0600.
|
||
2. Run `.claude/cluster-inspector/make-kubeconfig.sh lab-mgmt --admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig --apply-rbac`
|
||
(tagged `inspector`, `changed_when: false`; it re-mints the 7-day token each run).
|
||
|
||
## Execution steps
|
||
|
||
- **Step 0** — Repo has no commits yet: commit the existing tree as the initial import, then
|
||
copy this plan to `docs/plans/<ts>-mgmt-cluster-ansible.md` and commit it alone.
|
||
- **Step 1** — Scaffold `ansible/` (all files above), `.gitignore` exception, owner fills VMID/IP/
|
||
gw/DNS in `group_vars/all.yml` and `secrets.yml`. Check: `ansible-playbook mgmt.yml --syntax-check`,
|
||
`ansible -m ping all` (reaches pv1-1; mgmt unreachable until Play 1 ran).
|
||
- **Step 2** — Owner runs `ansible-playbook mgmt.yml --tags vm`; then `--tags k3s`;
|
||
then `--tags addons`; then `--tags capi`; then `--tags kubeconfig,inspector`. Per CLAUDE.md
|
||
safety rules the owner runs lab mutations (or explicitly tells Claude to); Claude diagnoses
|
||
failures via the `cluster-inspector` agent and records exact commands/outputs in the execution log.
|
||
- **Step 3** — Idempotency check: full `ansible-playbook mgmt.yml` twice; second run's recap must
|
||
show `changed=0` (fix any task that reports changed spuriously).
|
||
- **Step 4** — Docs: rewrite phase-1 Step 2 around the playbook (prereqs: ansible-core, pinned
|
||
clusterctl v1.12.11 **darwin** binary on the laptop for Step 3's `generate`/`describe`;
|
||
remove `clusterctl init`; note "no `clusterctl upgrade/move/delete`"); add troubleshooting rows
|
||
(provider CR not Ready → operator logs; stale cache secret #1075); `versions.md` rows for
|
||
cluster-api-operator 0.28.0, cert-manager v1.21.2, Ubuntu cloud image serial 20260926 + upgrade
|
||
order note (bump provider CRs one at a time); `configs/capi/clusterctl.yaml.example`: controller
|
||
credentials section marked as unused (now in `ansible/secrets.yml`); CLAUDE.md version rule
|
||
"explicit versions to `clusterctl init`" → "…in `group_vars/all.yml` / provider CRs";
|
||
`docs/decisions.md` rows: Ansible for mgmt setup (alt: cloud-init + k3s manifests, bash,
|
||
OpenTofu), cluster-api-operator (alt: guarded `clusterctl init`), mgmt VM from cloud image
|
||
(alt: manual VM). Tick phase-1 checklist Step 2 and README status only after Step 3 passes;
|
||
CHANGELOG entry after the owner confirms.
|
||
|
||
## Critical files
|
||
|
||
- New: `ansible/**` (above)
|
||
- Edit: `.gitignore`, `docs/phase-1-proxmox-capi.md` (Step 2, troubleshooting), `docs/versions.md`,
|
||
`docs/decisions.md`, `configs/capi/clusterctl.yaml.example`, `CLAUDE.md` (version rules), `README.md` (status)
|
||
- Reuse: `.claude/cluster-inspector/make-kubeconfig.sh` + `rbac.yaml` (Play 5), existing Proxmox
|
||
facts from `configs/capi/clusterctl.yaml.example` (URL, token id, node, bridge/VLAN)
|
||
|
||
## Verification
|
||
|
||
1. `ansible-playbook mgmt.yml --syntax-check` passes.
|
||
2. After the full run: `KUBECONFIG=~/.kube/lab-mgmt.kubeconfig kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A`
|
||
→ all Ready with the pinned versions; `kubectl get pods -A` → capi/capmox/ipam/cert-manager/operator Running.
|
||
3. `kubectl -n capmox-system logs deploy/capmox-controller-manager` shows no Proxmox auth errors.
|
||
4. `clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --kubernetes-version v1.35.9 ... --list-variables`
|
||
works against the new cluster (readiness for phase-1 Step 3).
|
||
5. Second full playbook run: `changed=0`.
|
||
6. `cluster-inspector` agent can read the cluster with `lab-mgmt-ro` (`can-i` output as in its README).
|