# Execution log: idempotent management cluster (Ansible + cluster-api-operator)

Pairs with [docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md](../plans/2026-09-27-0058-mgmt-cluster-ansible.md).

## Status

- [x] Step 0 — Initial import commit + plan commit
- [x] Step 1 — Scaffold `ansible/`, `.gitignore` exception, syntax/lint checks
- [x] Step 2 — Run the playbook tag by tag against the lab (owner)
- [x] Step 3 — Idempotency check (second full run `changed=0`)
- [x] Step 4 — Docs, versions, decisions, changelog

## Step 0 — Initial import + plan

The repo had no commits, so the existing tree went in first as `5fd998c` (after a grep for
token/secret/key material found only placeholders), then the plan alone as `ac7d5a4`.

## Step 1 — Scaffold `ansible/` (2026-09-27 01:02 CEST)

Wrote `ansible/{ansible.cfg,inventory.yml,group_vars/all.yml,secrets.yml.example,mgmt.yml}` and
four templates as planned. `ansible/secrets.yml` was already ignored by `*secret*`; added
`!ansible/secrets.yml.example`.

`ansible-core` is not installed on the laptop, so checks ran through `uvx` (cached, nothing
installed globally), pinned to ansible-core 2.18 (resolved 2.18.19):

```bash
cd ansible
uvx --from 'ansible-core>=2.18,<2.19' ansible-playbook mgmt.yml --syntax-check
uvx --with 'ansible-core>=2.18,<2.19' ansible-lint --profile production mgmt.yml
```

First lint run: 6× `name[casing]`, 5× `no-changed-when`. The commands flagged are the
create/resize/start/install tasks that only run behind a `when:` guard, so they got
`changed_when: true` rather than a fake condition. Second run: production profile passed.

Rendered all templates with dummy values (`-e mgmt_ip=192.0.2.10`, secret `DUMMY`) through a
throwaway local playbook: provider manifest parses as 11 documents (5 Namespaces, Secret, 5
provider CRs), k3s config and kubeconfig rewrite (`server` → VM IP, every `default` name →
`lab-mgmt`) come out as intended.

Connectivity to the PVE host:

```bash
uvx --from 'ansible-core>=2.18,<2.19' ansible pve -m ping -o
# pv1-1 | UNREACHABLE!: ... root@192.168.0.8: Permission denied (publickey,password).
```

**Worth noting:**
- The plan's assumption "root SSH from the laptop to `pv1-1` works" does not hold yet; the host
  answers but rejects the laptop's keys. Needed before Step 2.
- Condition-based waits: the playbook waits for `Available` Deployments in each provider
  namespace instead of a condition on the provider CRs, so it doesn't depend on the operator's
  condition naming (plan left that open).
- VM params (`mgmt_vmid`, `mgmt_ip`, `mgmt_gateway`, `mgmt_dns`) are empty in `group_vars/all.yml`;
  Play 1 asserts on them.
- `clusterctl` is not installed on the laptop either (needed for phase-1 Step 3, not for this playbook).

## Step 2 — Playbook runs

### `--tags vm`, first attempt (2026-09-27 09:30 CEST, owner)

Owner authorised the laptop key on `pv1-1`, installed ansible locally and filled in VM params
(`mgmt_vmid: 201`, `mgmt_ip: 192.168.4.21/24`, gateway `192.168.4.8`).

```bash
ansible-playbook mgmt.yml --tags vm
# TASK [Create the VM]
# "cmd": [..., "--nameserver", "8.8.8.8,", "2.2.2.2"]
# stderr: 400 too many arguments
```

`mgmt_dns` was the string `"8.8.8.8, 2.2.2.2"`; the command module splits on whitespace, so
`qm` got two positional leftovers. Fix: `mgmt_dns` is now a list and the task passes
`--nameserver '{{ mgmt_dns | join(' ') }}'` as one argument (space-separated is PVE's format for
the cloud-init nameserver list). Image download and key staging had already succeeded and are
idempotent on re-run; `qm status 201` confirmed no half-created VM was left behind.

### `--tags vm` second attempt, `--tags k3s` first attempt (2026-09-27 ~09:39 CEST, owner)

`--tags vm` created VM 201; the owner logged in as `ubuntu`. `resolvectl dns` on the VM shows
`eth0: 8.8.8.8 2.2.2.2`, so PVE's space-separated nameserver list works.

`--tags k3s` failed on the first task: `cloud-init status --wait` → `status: done`, rc 2.
`cloud-init status --long` showed `extended_status: degraded done`, `errors: []`, and only a
recoverable DEPRECATED warning: PVE's generated user-data sets `user:` as a string (deprecated
since cloud-init 22.2). rc 2 = "done with recoverable errors", so the task now fails only on rc
other than 0/2.

### `--tags k3s`, `addons`, `capi`, `kubeconfig` (owner, all succeeded by 2026-09-27 10:28 CEST)

No further playbook changes needed. Checked afterwards through the read-only kubeconfig
(`.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig`):

```bash
kubectl get pods -A -o wide
# 14 pods: cert-manager (3), cluster-api-operator, capi / kubeadm bootstrap / kubeadm control
# plane / capmox / ipam in-cluster controllers — all 1/1 Running, 0 restarts;
# helm-install-cert-manager and helm-install-cluster-api-operator jobs Completed.
kubectl -n capmox-system logs deploy/capmox-controller-manager --tail=200 | grep -iE 'error|fail|401|403'
# (nothing) — but CAPMOX only talks to Proxmox once a ProxmoxCluster exists, so the token is
# first really exercised in phase-1 Step 3.
```

The same session hit `Forbidden` on provider CRs, `helmcharts` and `nodes`: the inspector
role had no `operator.cluster.x-k8s.io` / `helm.cattle.io` access, and built-in `view` has no
nodes (needed for the agent's own providerID check). Added both API groups plus
`nodes`/`namespaces` read to `capi-readonly` in `.claude/cluster-inspector/rbac.yaml`; the next
full playbook run re-applies it via the `inspector` task (`--apply-rbac`).

**Worth noting:** the `.claude/settings.json` allowlist picked up four one-off entries approved
during this check (`export KUBECONFIG=…`, three exact `kubectl … logs` commands).

## Step 3 — Idempotency check

### Full run: `Apply provider manifests` always `changed` (2026-09-27 10:35 CEST)

Owner's full `ansible-playbook mgmt.yml` runs kept reporting `changed` on this one task. Found
with two read-only commands on the VM:

```bash
ansible mgmt -b -m command -a "k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml"
# rc 0, empty — nothing actually differs
ansible mgmt -b -m command -a "k3s kubectl apply --dry-run=server -f /opt/lab-mgmt/capi-providers.yaml"
# secret/capmox-variables configured (server dry run)   <- everything else "unchanged"
```

Root cause: the Secret used `stringData`, which the API server folds into `data` and never
stores, so client-side apply's three-way merge sees it missing every time, sends a patch and
prints `configured` even though the stored `data` is identical. Fix: template writes `data` with
`| b64encode` values. Expect one more `changed` (template file + last-applied annotation
rewrite), then `changed=0`.

Also from this check: the widened inspector RBAC was not applied yet (`get nodes` still
Forbidden), so no clean full run had completed before this fix. `clusterctl` v1.12.11
darwin/arm64 is installed in `~/bin`; `clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --target-namespace default --list-variables`
lists no `PROXMOX_*` credentials, confirming `clusterctl.yaml` only needs workload values now.

### Result (2026-09-27 ~10:38 CEST, owner)

After the `data` fix, one run with the expected `changed` on the Secret, then:

```text
PLAY RECAP
lab-mgmt : ok=19 changed=0 unreachable=0 failed=0 skipped=2
pv1-1    : ok=3  changed=0 unreachable=0 failed=0 skipped=6
```

Verified read-only with `lab-mgmt-ro`: node `lab-mgmt` Ready (v1.35.8+k3s1, Ubuntu 24.04.5);
all five Provider CRs `READY True` at v1.12.11 / v0.9.1 / v1.1.1; HelmCharts cert-manager v1.21.2
and cluster-api-operator 0.28.0 not failed; `auth can-i delete machines` / `get secrets` → `no`.
The widened inspector RBAC was applied by this run's `inspector` task.

## Step 4 — Docs (2026-09-27 10:40 CEST)

- `docs/phase-1-proxmox-capi.md`: Step 2 rewritten around the playbook (prereqs incl. `clusterctl`
  darwin/arm64 install into `~/bin`, tag-by-tag commands, operator rules), checklist ticked,
  topology row, 4 troubleshooting rows (provider not Ready, stale cache #1075, token rotation,
  non-idempotent task).
- `docs/versions.md`: cluster-api-operator 0.28.0, cert-manager v1.21.2, cloud image serial;
  provider upgrade procedure.
- `docs/decisions.md` #12–14; `CLAUDE.md` lab fact + version rules (no `clusterctl init/upgrade/move/delete`);
  `configs/capi/clusterctl.yaml.example` without controller credentials; README status, layout, decision 6.

**Worth noting:** the mgmt VM runs Ubuntu 24.04.5 (cloud image), the node template 24.04.4
(ISO) — different on purpose, they're separate images. Not covered by the plan and still open:
token rotation, VM resize, re-creating the VM (SSH `known_hosts` would conflict with `accept-new`).