cluster-autoscaler-with-capi/docs/plans-executions/2026-09-27-0058-mgmt-cluster-ansible.md

Execution log: idempotent management cluster (Ansible + cluster-api-operator)

Pairs with docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md.

Status

• Step 0 — Initial import commit + plan commit
• Step 1 — Scaffold ansible/, .gitignore exception, syntax/lint checks
• Step 2 — Run the playbook tag by tag against the lab (owner)
• Step 3 — Idempotency check (second full run changed=0)
• Step 4 — Docs, versions, decisions, changelog

Step 0 — Initial import + plan

The repo had no commits, so the existing tree went in first as 5fd998c (after a grep for token/secret/key material found only placeholders), then the plan alone as ac7d5a4.

Step 1 — Scaffold ansible/ (2026-09-27 01:02 CEST)

Wrote ansible/{ansible.cfg,inventory.yml,group_vars/all.yml,secrets.yml.example,mgmt.yml} and four templates as planned. ansible/secrets.yml was already ignored by *secret*; added !ansible/secrets.yml.example.

ansible-core is not installed on the laptop, so checks ran through uvx (cached, nothing installed globally), pinned to ansible-core 2.18 (resolved 2.18.19):

cd ansible
uvx --from 'ansible-core>=2.18,<2.19' ansible-playbook mgmt.yml --syntax-check
uvx --with 'ansible-core>=2.18,<2.19' ansible-lint --profile production mgmt.yml

First lint run: 6× name[casing], 5× no-changed-when. The commands flagged are the create/resize/start/install tasks that only run behind a when: guard, so they got changed_when: true rather than a fake condition. Second run: production profile passed.

Rendered all templates with dummy values (-e mgmt_ip=192.0.2.10, secret DUMMY) through a throwaway local playbook: provider manifest parses as 11 documents (5 Namespaces, Secret, 5 provider CRs), k3s config and kubeconfig rewrite (server → VM IP, every default name → lab-mgmt) come out as intended.

Connectivity to the PVE host:

uvx --from 'ansible-core>=2.18,<2.19' ansible pve -m ping -o
# pv1-1 | UNREACHABLE!: ... root@192.168.0.8: Permission denied (publickey,password).

Worth noting:

• The plan's assumption "root SSH from the laptop to pv1-1 works" does not hold yet; the host answers but rejects the laptop's keys. Needed before Step 2.
• Condition-based waits: the playbook waits for Available Deployments in each provider namespace instead of a condition on the provider CRs, so it doesn't depend on the operator's condition naming (plan left that open).
• VM params (mgmt_vmid, mgmt_ip, mgmt_gateway, mgmt_dns) are empty in group_vars/all.yml; Play 1 asserts on them.
• clusterctl is not installed on the laptop either (needed for phase-1 Step 3, not for this playbook).

Step 2 — Playbook runs

--tags vm, first attempt (2026-09-27 09:30 CEST, owner)

Owner authorised the laptop key on pv1-1, installed ansible locally and filled in VM params (mgmt_vmid: 201, mgmt_ip: 192.168.4.21/24, gateway 192.168.4.8).

ansible-playbook mgmt.yml --tags vm
# TASK [Create the VM]
# "cmd": [..., "--nameserver", "8.8.8.8,", "2.2.2.2"]
# stderr: 400 too many arguments

mgmt_dns was the string "8.8.8.8, 2.2.2.2"; the command module splits on whitespace, so qm got two positional leftovers. Fix: mgmt_dns is now a list and the task passes --nameserver '{{ mgmt_dns | join(' ') }}' as one argument (space-separated is PVE's format for the cloud-init nameserver list). Image download and key staging had already succeeded and are idempotent on re-run; qm status 201 confirmed no half-created VM was left behind.

--tags vm second attempt, --tags k3s first attempt (2026-09-27 ~09:39 CEST, owner)

--tags vm created VM 201; the owner logged in as ubuntu. resolvectl dns on the VM shows eth0: 8.8.8.8 2.2.2.2, so PVE's space-separated nameserver list works.

--tags k3s failed on the first task: cloud-init status --wait → status: done, rc 2. cloud-init status --long showed extended_status: degraded done, errors: [], and only a recoverable DEPRECATED warning: PVE's generated user-data sets user: as a string (deprecated since cloud-init 22.2). rc 2 = "done with recoverable errors", so the task now fails only on rc other than 0/2.

--tags k3s, addons, capi, kubeconfig (owner, all succeeded by 2026-09-27 10:28 CEST)

No further playbook changes needed. Checked afterwards through the read-only kubeconfig (.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig):

kubectl get pods -A -o wide
# 14 pods: cert-manager (3), cluster-api-operator, capi / kubeadm bootstrap / kubeadm control
# plane / capmox / ipam in-cluster controllers — all 1/1 Running, 0 restarts;
# helm-install-cert-manager and helm-install-cluster-api-operator jobs Completed.
kubectl -n capmox-system logs deploy/capmox-controller-manager --tail=200 | grep -iE 'error|fail|401|403'
# (nothing) — but CAPMOX only talks to Proxmox once a ProxmoxCluster exists, so the token is
# first really exercised in phase-1 Step 3.

The same session hit Forbidden on provider CRs, helmcharts and nodes: the inspector role had no operator.cluster.x-k8s.io / helm.cattle.io access, and built-in view has no nodes (needed for the agent's own providerID check). Added both API groups plus nodes/namespaces read to capi-readonly in .claude/cluster-inspector/rbac.yaml; the next full playbook run re-applies it via the inspector task (--apply-rbac).

Worth noting: the .claude/settings.json allowlist picked up four one-off entries approved during this check (export KUBECONFIG=…, three exact kubectl … logs commands).

Step 3 — Idempotency check

Full run: Apply provider manifests always changed (2026-09-27 10:35 CEST)

Owner's full ansible-playbook mgmt.yml runs kept reporting changed on this one task. Found with two read-only commands on the VM:

ansible mgmt -b -m command -a "k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml"
# rc 0, empty — nothing actually differs
ansible mgmt -b -m command -a "k3s kubectl apply --dry-run=server -f /opt/lab-mgmt/capi-providers.yaml"
# secret/capmox-variables configured (server dry run)   <- everything else "unchanged"

Root cause: the Secret used stringData, which the API server folds into data and never stores, so client-side apply's three-way merge sees it missing every time, sends a patch and prints configured even though the stored data is identical. Fix: template writes data with | b64encode values. Expect one more changed (template file + last-applied annotation rewrite), then changed=0.

Also from this check: the widened inspector RBAC was not applied yet (get nodes still Forbidden), so no clean full run had completed before this fix. clusterctl v1.12.11 darwin/arm64 is installed in ~/bin; clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --target-namespace default --list-variables lists no PROXMOX_* credentials, confirming clusterctl.yaml only needs workload values now.

Result (2026-09-27 ~10:38 CEST, owner)

After the data fix, one run with the expected changed on the Secret, then:

PLAY RECAP
lab-mgmt : ok=19 changed=0 unreachable=0 failed=0 skipped=2
pv1-1    : ok=3  changed=0 unreachable=0 failed=0 skipped=6

Verified read-only with lab-mgmt-ro: node lab-mgmt Ready (v1.35.8+k3s1, Ubuntu 24.04.5); all five Provider CRs READY True at v1.12.11 / v0.9.1 / v1.1.1; HelmCharts cert-manager v1.21.2 and cluster-api-operator 0.28.0 not failed; auth can-i delete machines / get secrets → no. The widened inspector RBAC was applied by this run's inspector task.

Step 4 — Docs (2026-09-27 10:40 CEST)

• docs/phase-1-proxmox-capi.md: Step 2 rewritten around the playbook (prereqs incl. clusterctl darwin/arm64 install into ~/bin, tag-by-tag commands, operator rules), checklist ticked, topology row, 4 troubleshooting rows (provider not Ready, stale cache #1075, token rotation, non-idempotent task).
• docs/versions.md: cluster-api-operator 0.28.0, cert-manager v1.21.2, cloud image serial; provider upgrade procedure.
• docs/decisions.md #12–14; CLAUDE.md lab fact + version rules (no clusterctl init/upgrade/move/delete); configs/capi/clusterctl.yaml.example without controller credentials; README status, layout, decision 6.

Worth noting: the mgmt VM runs Ubuntu 24.04.5 (cloud image), the node template 24.04.4 (ISO) — different on purpose, they're separate images. Not covered by the plan and still open: token rotation, VM resize, re-creating the VM (SSH known_hosts would conflict with accept-new).