kacerrandClaude Opus 5.5
aaa1ed64e2
stringData is never stored, so kubectl apply reported "configured" on every run and the capi play was never changed=0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
138 lines
7.0 KiB
Markdown
138 lines
7.0 KiB
Markdown
# Execution log: idempotent management cluster (Ansible + cluster-api-operator)
|
||
|
||
Pairs with [docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md](../plans/2026-09-27-0058-mgmt-cluster-ansible.md).
|
||
|
||
## Status
|
||
|
||
- [x] Step 0 — Initial import commit + plan commit
|
||
- [x] Step 1 — Scaffold `ansible/`, `.gitignore` exception, syntax/lint checks
|
||
- [x] Step 2 — Run the playbook tag by tag against the lab (owner)
|
||
- [ ] Step 3 — Idempotency check (second full run `changed=0`)
|
||
- [ ] Step 4 — Docs, versions, decisions, changelog
|
||
|
||
## Step 0 — Initial import + plan
|
||
|
||
The repo had no commits, so the existing tree went in first as `5fd998c` (after a grep for
|
||
token/secret/key material found only placeholders), then the plan alone as `ac7d5a4`.
|
||
|
||
## Step 1 — Scaffold `ansible/` (2026-09-27 01:02 CEST)
|
||
|
||
Wrote `ansible/{ansible.cfg,inventory.yml,group_vars/all.yml,secrets.yml.example,mgmt.yml}` and
|
||
four templates as planned. `ansible/secrets.yml` was already ignored by `*secret*`; added
|
||
`!ansible/secrets.yml.example`.
|
||
|
||
`ansible-core` is not installed on the laptop, so checks ran through `uvx` (cached, nothing
|
||
installed globally), pinned to ansible-core 2.18 (resolved 2.18.19):
|
||
|
||
```bash
|
||
cd ansible
|
||
uvx --from 'ansible-core>=2.18,<2.19' ansible-playbook mgmt.yml --syntax-check
|
||
uvx --with 'ansible-core>=2.18,<2.19' ansible-lint --profile production mgmt.yml
|
||
```
|
||
|
||
First lint run: 6× `name[casing]`, 5× `no-changed-when`. The commands flagged are the
|
||
create/resize/start/install tasks that only run behind a `when:` guard, so they got
|
||
`changed_when: true` rather than a fake condition. Second run: production profile passed.
|
||
|
||
Rendered all templates with dummy values (`-e mgmt_ip=192.0.2.10`, secret `DUMMY`) through a
|
||
throwaway local playbook: provider manifest parses as 11 documents (5 Namespaces, Secret, 5
|
||
provider CRs), k3s config and kubeconfig rewrite (`server` → VM IP, every `default` name →
|
||
`lab-mgmt`) come out as intended.
|
||
|
||
Connectivity to the PVE host:
|
||
|
||
```bash
|
||
uvx --from 'ansible-core>=2.18,<2.19' ansible pve -m ping -o
|
||
# pv1-1 | UNREACHABLE!: ... root@192.168.0.8: Permission denied (publickey,password).
|
||
```
|
||
|
||
**Worth noting:**
|
||
- The plan's assumption "root SSH from the laptop to `pv1-1` works" does not hold yet; the host
|
||
answers but rejects the laptop's keys. Needed before Step 2.
|
||
- Condition-based waits: the playbook waits for `Available` Deployments in each provider
|
||
namespace instead of a condition on the provider CRs, so it doesn't depend on the operator's
|
||
condition naming (plan left that open).
|
||
- VM params (`mgmt_vmid`, `mgmt_ip`, `mgmt_gateway`, `mgmt_dns`) are empty in `group_vars/all.yml`;
|
||
Play 1 asserts on them.
|
||
- `clusterctl` is not installed on the laptop either (needed for phase-1 Step 3, not for this playbook).
|
||
|
||
## Step 2 — Playbook runs (in progress)
|
||
|
||
### `--tags vm`, first attempt (2026-09-27 09:30 CEST, owner)
|
||
|
||
Owner authorised the laptop key on `pv1-1`, installed ansible locally and filled in VM params
|
||
(`mgmt_vmid: 201`, `mgmt_ip: 192.168.4.21/24`, gateway `192.168.4.8`).
|
||
|
||
```bash
|
||
ansible-playbook mgmt.yml --tags vm
|
||
# TASK [Create the VM]
|
||
# "cmd": [..., "--nameserver", "8.8.8.8,", "2.2.2.2"]
|
||
# stderr: 400 too many arguments
|
||
```
|
||
|
||
`mgmt_dns` was the string `"8.8.8.8, 2.2.2.2"`; the command module splits on whitespace, so
|
||
`qm` got two positional leftovers. Fix: `mgmt_dns` is now a list and the task passes
|
||
`--nameserver '{{ mgmt_dns | join(' ') }}'` as one argument (space-separated is PVE's format for
|
||
the cloud-init nameserver list). Image download and key staging had already succeeded and are
|
||
idempotent on re-run; `qm status 201` confirmed no half-created VM was left behind.
|
||
|
||
### `--tags vm` second attempt, `--tags k3s` first attempt (2026-09-27 ~09:39 CEST, owner)
|
||
|
||
`--tags vm` created VM 201; the owner logged in as `ubuntu`. `resolvectl dns` on the VM shows
|
||
`eth0: 8.8.8.8 2.2.2.2`, so PVE's space-separated nameserver list works.
|
||
|
||
`--tags k3s` failed on the first task: `cloud-init status --wait` → `status: done`, rc 2.
|
||
`cloud-init status --long` showed `extended_status: degraded done`, `errors: []`, and only a
|
||
recoverable DEPRECATED warning: PVE's generated user-data sets `user:` as a string (deprecated
|
||
since cloud-init 22.2). rc 2 = "done with recoverable errors", so the task now fails only on rc
|
||
other than 0/2.
|
||
|
||
### `--tags k3s`, `addons`, `capi`, `kubeconfig` (owner, all succeeded by 2026-09-27 10:28 CEST)
|
||
|
||
No further playbook changes needed. Checked afterwards through the read-only kubeconfig
|
||
(`.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig`):
|
||
|
||
```bash
|
||
kubectl get pods -A -o wide
|
||
# 14 pods: cert-manager (3), cluster-api-operator, capi / kubeadm bootstrap / kubeadm control
|
||
# plane / capmox / ipam in-cluster controllers — all 1/1 Running, 0 restarts;
|
||
# helm-install-cert-manager and helm-install-cluster-api-operator jobs Completed.
|
||
kubectl -n capmox-system logs deploy/capmox-controller-manager --tail=200 | grep -iE 'error|fail|401|403'
|
||
# (nothing) — but CAPMOX only talks to Proxmox once a ProxmoxCluster exists, so the token is
|
||
# first really exercised in phase-1 Step 3.
|
||
```
|
||
|
||
The same session hit `Forbidden` on provider CRs, `helmcharts` and `nodes`: the inspector
|
||
role had no `operator.cluster.x-k8s.io` / `helm.cattle.io` access, and built-in `view` has no
|
||
nodes (needed for the agent's own providerID check). Added both API groups plus
|
||
`nodes`/`namespaces` read to `capi-readonly` in `.claude/cluster-inspector/rbac.yaml`; the next
|
||
full playbook run re-applies it via the `inspector` task (`--apply-rbac`).
|
||
|
||
**Worth noting:** the `.claude/settings.json` allowlist picked up four one-off entries approved
|
||
during this check (`export KUBECONFIG=…`, three exact `kubectl … logs` commands).
|
||
|
||
## Step 3 — Idempotency check (in progress)
|
||
|
||
### Full run: `Apply provider manifests` always `changed` (2026-09-27 10:35 CEST)
|
||
|
||
Owner's full `ansible-playbook mgmt.yml` runs kept reporting `changed` on this one task. Found
|
||
with two read-only commands on the VM:
|
||
|
||
```bash
|
||
ansible mgmt -b -m command -a "k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml"
|
||
# rc 0, empty — nothing actually differs
|
||
ansible mgmt -b -m command -a "k3s kubectl apply --dry-run=server -f /opt/lab-mgmt/capi-providers.yaml"
|
||
# secret/capmox-variables configured (server dry run) <- everything else "unchanged"
|
||
```
|
||
|
||
Root cause: the Secret used `stringData`, which the API server folds into `data` and never
|
||
stores, so client-side apply's three-way merge sees it missing every time, sends a patch and
|
||
prints `configured` even though the stored `data` is identical. Fix: template writes `data` with
|
||
`| b64encode` values. Expect one more `changed` (template file + last-applied annotation
|
||
rewrite), then `changed=0`.
|
||
|
||
Also from this check: the widened inspector RBAC was not applied yet (`get nodes` still
|
||
Forbidden), so no clean full run had completed before this fix. `clusterctl` v1.12.11
|
||
darwin/arm64 is installed in `~/bin`; `clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --target-namespace default --list-variables`
|
||
lists no `PROXMOX_*` credentials, confirming `clusterctl.yaml` only needs workload values now.
|