kacerrandClaude Opus 5.5
d8efa3b961
Phase-1 Step 2 rewritten around ansible/mgmt.yml and cluster-api-operator, version pins for operator/cert-manager/cloud image, decisions #12-14, execution log and changelog. Phase-1 Step 2 checked off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
168 lines
8.5 KiB
Markdown
168 lines
8.5 KiB
Markdown
# Execution log: idempotent management cluster (Ansible + cluster-api-operator)
|
||
|
||
Pairs with [docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md](../plans/2026-09-27-0058-mgmt-cluster-ansible.md).
|
||
|
||
## Status
|
||
|
||
- [x] Step 0 — Initial import commit + plan commit
|
||
- [x] Step 1 — Scaffold `ansible/`, `.gitignore` exception, syntax/lint checks
|
||
- [x] Step 2 — Run the playbook tag by tag against the lab (owner)
|
||
- [x] Step 3 — Idempotency check (second full run `changed=0`)
|
||
- [x] Step 4 — Docs, versions, decisions, changelog
|
||
|
||
## Step 0 — Initial import + plan
|
||
|
||
The repo had no commits, so the existing tree went in first as `5fd998c` (after a grep for
|
||
token/secret/key material found only placeholders), then the plan alone as `ac7d5a4`.
|
||
|
||
## Step 1 — Scaffold `ansible/` (2026-09-27 01:02 CEST)
|
||
|
||
Wrote `ansible/{ansible.cfg,inventory.yml,group_vars/all.yml,secrets.yml.example,mgmt.yml}` and
|
||
four templates as planned. `ansible/secrets.yml` was already ignored by `*secret*`; added
|
||
`!ansible/secrets.yml.example`.
|
||
|
||
`ansible-core` is not installed on the laptop, so checks ran through `uvx` (cached, nothing
|
||
installed globally), pinned to ansible-core 2.18 (resolved 2.18.19):
|
||
|
||
```bash
|
||
cd ansible
|
||
uvx --from 'ansible-core>=2.18,<2.19' ansible-playbook mgmt.yml --syntax-check
|
||
uvx --with 'ansible-core>=2.18,<2.19' ansible-lint --profile production mgmt.yml
|
||
```
|
||
|
||
First lint run: 6× `name[casing]`, 5× `no-changed-when`. The commands flagged are the
|
||
create/resize/start/install tasks that only run behind a `when:` guard, so they got
|
||
`changed_when: true` rather than a fake condition. Second run: production profile passed.
|
||
|
||
Rendered all templates with dummy values (`-e mgmt_ip=192.0.2.10`, secret `DUMMY`) through a
|
||
throwaway local playbook: provider manifest parses as 11 documents (5 Namespaces, Secret, 5
|
||
provider CRs), k3s config and kubeconfig rewrite (`server` → VM IP, every `default` name →
|
||
`lab-mgmt`) come out as intended.
|
||
|
||
Connectivity to the PVE host:
|
||
|
||
```bash
|
||
uvx --from 'ansible-core>=2.18,<2.19' ansible pve -m ping -o
|
||
# pv1-1 | UNREACHABLE!: ... root@192.168.0.8: Permission denied (publickey,password).
|
||
```
|
||
|
||
**Worth noting:**
|
||
- The plan's assumption "root SSH from the laptop to `pv1-1` works" does not hold yet; the host
|
||
answers but rejects the laptop's keys. Needed before Step 2.
|
||
- Condition-based waits: the playbook waits for `Available` Deployments in each provider
|
||
namespace instead of a condition on the provider CRs, so it doesn't depend on the operator's
|
||
condition naming (plan left that open).
|
||
- VM params (`mgmt_vmid`, `mgmt_ip`, `mgmt_gateway`, `mgmt_dns`) are empty in `group_vars/all.yml`;
|
||
Play 1 asserts on them.
|
||
- `clusterctl` is not installed on the laptop either (needed for phase-1 Step 3, not for this playbook).
|
||
|
||
## Step 2 — Playbook runs
|
||
|
||
### `--tags vm`, first attempt (2026-09-27 09:30 CEST, owner)
|
||
|
||
Owner authorised the laptop key on `pv1-1`, installed ansible locally and filled in VM params
|
||
(`mgmt_vmid: 201`, `mgmt_ip: 192.168.4.21/24`, gateway `192.168.4.8`).
|
||
|
||
```bash
|
||
ansible-playbook mgmt.yml --tags vm
|
||
# TASK [Create the VM]
|
||
# "cmd": [..., "--nameserver", "8.8.8.8,", "2.2.2.2"]
|
||
# stderr: 400 too many arguments
|
||
```
|
||
|
||
`mgmt_dns` was the string `"8.8.8.8, 2.2.2.2"`; the command module splits on whitespace, so
|
||
`qm` got two positional leftovers. Fix: `mgmt_dns` is now a list and the task passes
|
||
`--nameserver '{{ mgmt_dns | join(' ') }}'` as one argument (space-separated is PVE's format for
|
||
the cloud-init nameserver list). Image download and key staging had already succeeded and are
|
||
idempotent on re-run; `qm status 201` confirmed no half-created VM was left behind.
|
||
|
||
### `--tags vm` second attempt, `--tags k3s` first attempt (2026-09-27 ~09:39 CEST, owner)
|
||
|
||
`--tags vm` created VM 201; the owner logged in as `ubuntu`. `resolvectl dns` on the VM shows
|
||
`eth0: 8.8.8.8 2.2.2.2`, so PVE's space-separated nameserver list works.
|
||
|
||
`--tags k3s` failed on the first task: `cloud-init status --wait` → `status: done`, rc 2.
|
||
`cloud-init status --long` showed `extended_status: degraded done`, `errors: []`, and only a
|
||
recoverable DEPRECATED warning: PVE's generated user-data sets `user:` as a string (deprecated
|
||
since cloud-init 22.2). rc 2 = "done with recoverable errors", so the task now fails only on rc
|
||
other than 0/2.
|
||
|
||
### `--tags k3s`, `addons`, `capi`, `kubeconfig` (owner, all succeeded by 2026-09-27 10:28 CEST)
|
||
|
||
No further playbook changes needed. Checked afterwards through the read-only kubeconfig
|
||
(`.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig`):
|
||
|
||
```bash
|
||
kubectl get pods -A -o wide
|
||
# 14 pods: cert-manager (3), cluster-api-operator, capi / kubeadm bootstrap / kubeadm control
|
||
# plane / capmox / ipam in-cluster controllers — all 1/1 Running, 0 restarts;
|
||
# helm-install-cert-manager and helm-install-cluster-api-operator jobs Completed.
|
||
kubectl -n capmox-system logs deploy/capmox-controller-manager --tail=200 | grep -iE 'error|fail|401|403'
|
||
# (nothing) — but CAPMOX only talks to Proxmox once a ProxmoxCluster exists, so the token is
|
||
# first really exercised in phase-1 Step 3.
|
||
```
|
||
|
||
The same session hit `Forbidden` on provider CRs, `helmcharts` and `nodes`: the inspector
|
||
role had no `operator.cluster.x-k8s.io` / `helm.cattle.io` access, and built-in `view` has no
|
||
nodes (needed for the agent's own providerID check). Added both API groups plus
|
||
`nodes`/`namespaces` read to `capi-readonly` in `.claude/cluster-inspector/rbac.yaml`; the next
|
||
full playbook run re-applies it via the `inspector` task (`--apply-rbac`).
|
||
|
||
**Worth noting:** the `.claude/settings.json` allowlist picked up four one-off entries approved
|
||
during this check (`export KUBECONFIG=…`, three exact `kubectl … logs` commands).
|
||
|
||
## Step 3 — Idempotency check
|
||
|
||
### Full run: `Apply provider manifests` always `changed` (2026-09-27 10:35 CEST)
|
||
|
||
Owner's full `ansible-playbook mgmt.yml` runs kept reporting `changed` on this one task. Found
|
||
with two read-only commands on the VM:
|
||
|
||
```bash
|
||
ansible mgmt -b -m command -a "k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml"
|
||
# rc 0, empty — nothing actually differs
|
||
ansible mgmt -b -m command -a "k3s kubectl apply --dry-run=server -f /opt/lab-mgmt/capi-providers.yaml"
|
||
# secret/capmox-variables configured (server dry run) <- everything else "unchanged"
|
||
```
|
||
|
||
Root cause: the Secret used `stringData`, which the API server folds into `data` and never
|
||
stores, so client-side apply's three-way merge sees it missing every time, sends a patch and
|
||
prints `configured` even though the stored `data` is identical. Fix: template writes `data` with
|
||
`| b64encode` values. Expect one more `changed` (template file + last-applied annotation
|
||
rewrite), then `changed=0`.
|
||
|
||
Also from this check: the widened inspector RBAC was not applied yet (`get nodes` still
|
||
Forbidden), so no clean full run had completed before this fix. `clusterctl` v1.12.11
|
||
darwin/arm64 is installed in `~/bin`; `clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --target-namespace default --list-variables`
|
||
lists no `PROXMOX_*` credentials, confirming `clusterctl.yaml` only needs workload values now.
|
||
|
||
### Result (2026-09-27 ~10:38 CEST, owner)
|
||
|
||
After the `data` fix, one run with the expected `changed` on the Secret, then:
|
||
|
||
```text
|
||
PLAY RECAP
|
||
lab-mgmt : ok=19 changed=0 unreachable=0 failed=0 skipped=2
|
||
pv1-1 : ok=3 changed=0 unreachable=0 failed=0 skipped=6
|
||
```
|
||
|
||
Verified read-only with `lab-mgmt-ro`: node `lab-mgmt` Ready (v1.35.8+k3s1, Ubuntu 24.04.5);
|
||
all five Provider CRs `READY True` at v1.12.11 / v0.9.1 / v1.1.1; HelmCharts cert-manager v1.21.2
|
||
and cluster-api-operator 0.28.0 not failed; `auth can-i delete machines` / `get secrets` → `no`.
|
||
The widened inspector RBAC was applied by this run's `inspector` task.
|
||
|
||
## Step 4 — Docs (2026-09-27 10:40 CEST)
|
||
|
||
- `docs/phase-1-proxmox-capi.md`: Step 2 rewritten around the playbook (prereqs incl. `clusterctl`
|
||
darwin/arm64 install into `~/bin`, tag-by-tag commands, operator rules), checklist ticked,
|
||
topology row, 4 troubleshooting rows (provider not Ready, stale cache #1075, token rotation,
|
||
non-idempotent task).
|
||
- `docs/versions.md`: cluster-api-operator 0.28.0, cert-manager v1.21.2, cloud image serial;
|
||
provider upgrade procedure.
|
||
- `docs/decisions.md` #12–14; `CLAUDE.md` lab fact + version rules (no `clusterctl init/upgrade/move/delete`);
|
||
`configs/capi/clusterctl.yaml.example` without controller credentials; README status, layout, decision 6.
|
||
|
||
**Worth noting:** the mgmt VM runs Ubuntu 24.04.5 (cloud image), the node template 24.04.4
|
||
(ISO) — different on purpose, they're separate images. Not covered by the plan and still open:
|
||
token rotation, VM resize, re-creating the VM (SSH `known_hosts` would conflict with `accept-new`).
|