kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
cluster-autoscaler-with-capi/docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md
T

11 KiB
Raw Permalink Blame History

Plan: Idempotent management cluster (Ansible + cluster-api-operator)

Created: 2026-09-27 00:58

Context

Phase 1 step 2 (management cluster + CAPI providers) is next. The runbook today is a series of hand-typed commands (curl | sh for k3s, clusterctl init), which is not re-runnable and leaves no declarative record of what the management cluster contains. Goal: one Ansible playbook, run from the laptop, that creates the management VM on Proxmox, installs pinned k3s, cert-manager, cluster-api-operator and the five CAPI providers, and fetches kubeconfigs. Re-running it is a no-op (changed=0). Keep it simple: happy path only, no VM resizing/reconciliation, no HA, no token rotation automation.

Decisions made by the owner: Ansible playbook; providers via cluster-api-operator Provider CRs (not clusterctl init); the playbook also creates the VM from the Ubuntu 24.04 cloud image.

Upstream facts verified 2026-09-27 (upstream-researcher, from tags/release assets):

  • cluster-api-operator v0.28.0 (chart 0.28.0, repo https://kubernetes-sigs.github.io/cluster-api-operator) is built against CAPI v1.12.10 → closest match to our v1.12.11 pin (v0.29.0 is on CAPI 1.13.5). Only gate: provider metadata.yaml contract v1beta1/v1beta2; CAPI v1.12.11, CAPMOX v0.9.1, IPAM in-cluster v1.1.1 all declare v1beta2.
  • cert-manager must be pre-installed (operator doesn't manage it): v1.21.2, supports k8s 1.33–1.36.
  • CRs are operator.cluster.x-k8s.io/v1alpha2; CR name = provider name, CR namespace = install namespace (must pre-exist); proxmox and in-cluster are built-in (no fetchConfig). Non-core providers wait for the CoreProvider on their own, so all five can be applied at once.
  • CAPMOX v0.9.1 reads PROXMOX_URL, PROXMOX_TOKEN, PROXMOX_SECRET from the provider's configSecret. No feature gates needed for lab1 (no ClusterClass).
  • Components are downloaded from GitHub release URLs (not the rate-limited API) and cached per version.
  • Re-applying unchanged CRs is a no-op; bumping spec.version upgrades in place (one provider at a time).
  • Gotchas: don't put providers in the operator chart values (they become Helm hooks, #1052); stale cache secret <type>-<name>-<ver>-cache must be deleted by hand (#1075); configSecret is not watched, so treat it as immutable (rotate = new Secret name).
  • clusterctl describe cluster and clusterctl generate cluster --infrastructure proxmox:v0.9.1 keep working; don't use clusterctl upgrade/move/delete against an operator-managed cluster.
  • Ubuntu cloud image pinned: https://cloud-images.ubuntu.com/noble/20260926/noble-server-cloudimg-amd64.img, sha256 6a81c37564db9b1ee84e141922625e1d7c5b389b99bb3c572e0243607d5bb4d2.

Assumptions (confirm at Step 1): root SSH from the laptop to pv1-1 works; ansible-core is installed on the laptop; the owner provides VMID, static IP, gateway, DNS for the mgmt VM (VLAN 4).

Layout

ansible/
├── ansible.cfg                 # inventory=inventory.yml, interpreter_python=auto_silent
├── inventory.yml               # pve: pv1-1 (root) ; mgmt: lab-mgmt (ubuntu, become)
├── group_vars/all.yml          # versions + VM params (no secrets)
├── secrets.yml.example         # proxmox_token_secret: "<secret>"  → copy to secrets.yml (gitignored)
├── mgmt.yml                    # the playbook, plays tagged vm / k3s / addons / capi / kubeconfig
└── templates/
    ├── k3s-config.yaml.j2              # /etc/rancher/k3s/config.yaml
    ├── helmchart-cert-manager.yaml.j2  # k3s HelmChart CR
    ├── helmchart-capi-operator.yaml.j2 # k3s HelmChart CR
    └── capi-providers.yaml.j2          # namespaces + capmox-variables Secret + 5 provider CRs

Only ansible.builtin modules, no Galaxy collections, no helm/python-kubernetes on the VM: Helm charts are installed through k3s's built-in helm-controller (HelmChart CRs written to /var/lib/rancher/k3s/server/manifests/), which keeps them applied and upgrades on file change.

.gitignore: ansible/secrets.yml is already caught by *secret*; add !ansible/secrets.yml.example.

Playbook (ansible/mgmt.yml)

Play 1 — vm (hosts: pve). Guard everything on qm status {{ mgmt_vmid }} (rc≠0 → create):

  1. get_url the pinned cloud image to /var/lib/vz/template/iso/ with checksum: sha256:….
  2. copy the SSH public key to a temp file; create + configure in one block: qm create <vmid> --name lab-mgmt --cores 2 --memory 4096 --net0 virtio,bridge=vmbr0,tag=4 --scsihw virtio-scsi-pci --scsi0 local-lvm:0,import-from=<img> --ide2 local-lvm:cloudinit --boot order=scsi0 --serial0 socket --vga serial0 --agent 1 --ciuser ubuntu --sshkeys <file> --ipconfig0 ip=<ip>/24,gw=<gw> --nameserver <dns>, then qm resize <vmid> scsi0 30G, qm start. (Disk lands raw on LVM-thin; bridge + separate tag per lab rules.)
  3. wait_for port 22 on the VM IP (delegated to localhost).

Play 2 — k3s (hosts: mgmt, become).

  1. apt qemu-guest-agent (+ service started).
  2. template /etc/rancher/k3s/config.yaml: disable: [traefik], tls-san: [<ip>], write-kubeconfig-mode: "0600"; notify restart handler.
  3. command: k3s --version (failed_when false) → if it doesn't contain k3s_version, get_url https://get.k3s.io to /usr/local/bin/k3s-install.sh and run it with INSTALL_K3S_VERSION={{ k3s_version }} (so a version bump in all.yml = upgrade).
  4. k3s kubectl wait --for=condition=Ready node --all --timeout=180s (changed_when false).

Play 3 — addons (hosts: mgmt, become).

  1. template cert-manager HelmChart (repo https://charts.jetstack.io, v1.21.2, targetNamespace: cert-manager, createNamespace: true, crds.enabled: true).
  2. Wait: k3s kubectl -n cert-manager rollout status deploy/cert-manager-webhook with retries/until (deployment appears only after the helm job ran).
  3. template cluster-api-operator HelmChart (0.28.0, targetNamespace: capi-operator-system, no provider values), wait for its deployment rollout, then k3s kubectl wait --for condition=established crd/coreproviders.operator.cluster.x-k8s.io.

Play 4 — capi (hosts: mgmt, become).

  1. template /opt/lab-mgmt/capi-providers.yaml (mode 0600, holds the token secret): namespaces capi-system, capi-kubeadm-bootstrap-system, capi-kubeadm-control-plane-system, capmox-system, capi-ipam-in-cluster-system; Secret capmox-system/capmox-variables (PROXMOX_URL, PROXMOX_TOKEN, PROXMOX_SECRET); the five CRs: CoreProvider cluster-api v1.12.11, BootstrapProvider kubeadm v1.12.11, ControlPlaneProvider kubeadm v1.12.11, InfrastructureProvider proxmox v0.9.1 (configSecret: {name: capmox-variables}), IPAMProvider in-cluster v1.1.1.
  2. k3s kubectl apply -f it; changed_when on created|configured in stdout.
  3. k3s kubectl wait --for=condition=Ready <kind>/<name> -n <ns> --timeout=10m for all five (condition name to be confirmed on first run; fall back to deployment rollout status).

Play 5 — kubeconfig (hosts: mgmt → localhost).

  1. fetch /etc/rancher/k3s/k3s.yaml to ~/.kube/lab-mgmt.kubeconfig (outside the repo), replace 127.0.0.1 with the VM IP, context renamed lab-mgmt, mode 0600.
  2. Run .claude/cluster-inspector/make-kubeconfig.sh lab-mgmt --admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig --apply-rbac (tagged inspector, changed_when: false; it re-mints the 7-day token each run).

Execution steps

  • Step 0 — Repo has no commits yet: commit the existing tree as the initial import, then copy this plan to docs/plans/<ts>-mgmt-cluster-ansible.md and commit it alone.
  • Step 1 — Scaffold ansible/ (all files above), .gitignore exception, owner fills VMID/IP/ gw/DNS in group_vars/all.yml and secrets.yml. Check: ansible-playbook mgmt.yml --syntax-check, ansible -m ping all (reaches pv1-1; mgmt unreachable until Play 1 ran).
  • Step 2 — Owner runs ansible-playbook mgmt.yml --tags vm; then --tags k3s; then --tags addons; then --tags capi; then --tags kubeconfig,inspector. Per CLAUDE.md safety rules the owner runs lab mutations (or explicitly tells Claude to); Claude diagnoses failures via the cluster-inspector agent and records exact commands/outputs in the execution log.
  • Step 3 — Idempotency check: full ansible-playbook mgmt.yml twice; second run's recap must show changed=0 (fix any task that reports changed spuriously).
  • Step 4 — Docs: rewrite phase-1 Step 2 around the playbook (prereqs: ansible-core, pinned clusterctl v1.12.11 darwin binary on the laptop for Step 3's generate/describe; remove clusterctl init; note "no clusterctl upgrade/move/delete"); add troubleshooting rows (provider CR not Ready → operator logs; stale cache secret #1075); versions.md rows for cluster-api-operator 0.28.0, cert-manager v1.21.2, Ubuntu cloud image serial 20260926 + upgrade order note (bump provider CRs one at a time); configs/capi/clusterctl.yaml.example: controller credentials section marked as unused (now in ansible/secrets.yml); CLAUDE.md version rule "explicit versions to clusterctl init" → "…in group_vars/all.yml / provider CRs"; docs/decisions.md rows: Ansible for mgmt setup (alt: cloud-init + k3s manifests, bash, OpenTofu), cluster-api-operator (alt: guarded clusterctl init), mgmt VM from cloud image (alt: manual VM). Tick phase-1 checklist Step 2 and README status only after Step 3 passes; CHANGELOG entry after the owner confirms.

Critical files

  • New: ansible/** (above)
  • Edit: .gitignore, docs/phase-1-proxmox-capi.md (Step 2, troubleshooting), docs/versions.md, docs/decisions.md, configs/capi/clusterctl.yaml.example, CLAUDE.md (version rules), README.md (status)
  • Reuse: .claude/cluster-inspector/make-kubeconfig.sh + rbac.yaml (Play 5), existing Proxmox facts from configs/capi/clusterctl.yaml.example (URL, token id, node, bridge/VLAN)

Verification

  1. ansible-playbook mgmt.yml --syntax-check passes.
  2. After the full run: KUBECONFIG=~/.kube/lab-mgmt.kubeconfig kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A → all Ready with the pinned versions; kubectl get pods -A → capi/capmox/ipam/cert-manager/operator Running.
  3. kubectl -n capmox-system logs deploy/capmox-controller-manager shows no Proxmox auth errors.
  4. clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --kubernetes-version v1.35.9 ... --list-variables works against the new cluster (readiness for phase-1 Step 3).
  5. Second full playbook run: changed=0.
  6. cluster-inspector agent can read the cluster with lab-mgmt-ro (can-i output as in its README).
Reference in New Issue View Git Blame Copy Permalink