diff --git a/.claude/settings.json b/.claude/settings.json index 79f6490..5607d2e 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -16,7 +16,9 @@ "Bash(clusterctl version *)", "Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get nodes)", "Bash(clusterctl generate *)", - "Bash(ansible mgmt *)" + "Bash(ansible mgmt *)", + "Bash(kubectl get *)", + "Bash(kubectl auth *)" ] } } diff --git a/CHANGELOG.md b/CHANGELOG.md index c580cf0..ef46484 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,17 @@ # Changelog +## 2026-09-27 10:40 CEST — Management cluster running, built by an idempotent Ansible playbook + +- Phase 1 Step 2 done: `ansible/mgmt.yml` creates VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1` + from the pinned Ubuntu cloud image, installs k3s v1.35.8+k3s1, cert-manager v1.21.2 and + cluster-api-operator 0.28.0, and CAPI v1.12.11 / CAPMOX v0.9.1 / IPAM v1.1.1 as operator + Provider CRs (all READY). A second full run reports `changed=0`. +- `clusterctl init` is gone from the runbook; `clusterctl` (v1.12.11 in `~/bin`) is used only for + `generate`/`describe`/`get kubeconfig`. Proxmox credentials moved to `ansible/secrets.yml`. +- cluster-inspector role can now read Provider CRs, HelmCharts and nodes. +- Files: `ansible/`, `docs/phase-1-proxmox-capi.md`, `docs/versions.md`, `docs/decisions.md` + (#12–14), `configs/capi/clusterctl.yaml.example`, `CLAUDE.md`, `README.md`. + ## 2026-09-27 00:02 CEST — Read-only cluster-inspector credentials next to the agent - Moved inspector RBAC from `configs/rbac/` to `.claude/cluster-inspector/` with `make-kubeconfig.sh` diff --git a/CLAUDE.md b/CLAUDE.md index 9f791a3..b6fbd13 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,6 +16,8 @@ Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux - Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2. - Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds). - Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55). Login only via cloud-init. +- Management cluster: VM 201 `lab-mgmt`, 192.168.4.21 (`ssh ubuntu@`), built by the idempotent + `ansible/mgmt.yml` (re-run must be `changed=0`); admin kubeconfig `~/.kube/lab-mgmt.kubeconfig`. - Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`, PVESDNUser on `/sdn/zones/localnetwork`. @@ -28,7 +30,11 @@ Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux - CAPI core is pinned to **v1.12.x** because CAPMOX v0.9 is only tested on CAPI 1.11/1.12. This caps workload Kubernetes at 1.35 and cluster-autoscaler at 1.35.x. Don't bump CAPI unless the CAPMOX compatibility table allows it. -- Always pass explicit versions to `clusterctl init` / `helm install`; never rely on "latest". +- Always pin versions explicitly (management cluster: `ansible/group_vars/all.yml`; `helm install + --version`); never rely on "latest". +- CAPI providers are managed by cluster-api-operator (Provider CRs from `ansible/mgmt.yml`). Never + run `clusterctl init/upgrade/move/delete` against the management cluster; `clusterctl` is only for + `generate cluster` (always with `--infrastructure proxmox:v0.9.1`), `describe`, `get kubeconfig`. - Check a provider's `metadata.yaml` contract and CAPI version (go.mod) before adding it to the management cluster — all providers share one CAPI core (known skew: Kamaji CP provider v0.21.0 is on CAPI 1.13). diff --git a/README.md b/README.md index 263fddf..918c71d 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,8 @@ Node autoscaling for Kubernetes clusters running as VMs on a home-lab hypervisor growing, in later phases, into a "poor man's Kubernetes-as-a-Service". > Status: **Phase 1 in progress** — VM template built and smoke-tested; management cluster -> and first workload cluster next. Versions verified 2026-09-24/26, see [docs/versions.md](docs/versions.md). +> running (k3s + CAPI via cluster-api-operator, built by [ansible/mgmt.yml](ansible/mgmt.yml)); +> first workload cluster `lab1` next. Versions verified 2026-09-24/26/27, see [docs/versions.md](docs/versions.md). ## Goals @@ -74,6 +75,7 @@ Phases 2 and 3 are independent and can be done in either order. ├── README.md # this file ├── CLAUDE.md # context + rules for Claude Code sessions ├── CHANGELOG.md # milestones and significant changes, newest first +├── ansible/ # mgmt.yml: management VM + k3s + cert-manager + CAPI operator/providers ├── .claude/ │ ├── agents/ # upstream-researcher, cluster-inspector (read-only) │ ├── commands/ # /ship @@ -113,3 +115,5 @@ Short summary; the full log with alternatives and status is [docs/decisions.md]( to keep its version under our control. 5. **Scale-from-zero via capacity annotations** on the MachineDeployment — CAPMOX does not publish `status.capacity` on `ProxmoxMachineTemplate`. +6. **Management cluster from one idempotent Ansible playbook**, CAPI providers installed by + **cluster-api-operator** instead of `clusterctl init` — re-runs are no-ops, upgrades are version bumps. diff --git a/configs/capi/clusterctl.yaml.example b/configs/capi/clusterctl.yaml.example index 8ea2818..611d333 100644 --- a/configs/capi/clusterctl.yaml.example +++ b/configs/capi/clusterctl.yaml.example @@ -1,8 +1,6 @@ # copy to ~/.cluster-api/clusterctl.yaml — never commit the real file -## controller -PROXMOX_URL: "https://192.168.0.8:8006" -PROXMOX_TOKEN: "capmox@pve!capi" -PROXMOX_SECRET: "" +# Only used by `clusterctl generate cluster`. No Proxmox credentials here: CAPMOX is installed by +# cluster-api-operator and gets them from ansible/secrets.yml (Secret capmox-system/capmox-variables). ## workload defaults PROXMOX_SOURCENODE: "pv1-1" # node holding the template diff --git a/docs/decisions.md b/docs/decisions.md index da4eecc..9903e67 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -20,3 +20,6 @@ image-builder fixes log; their dates are when the decision was recorded, not nec | 9 | 2026-09-26 | Phases 2 and 3 are independent and can be done in either order | strict sequence | no technical dependency between the KVM provider and hosted control planes | active | | 10 | 2026-09-26 | Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog ([session](sessions/2026-09-26-2355-history-conventions.md)) | chat history only; raw transcripts committed | research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets | active | | 11 | 2026-09-27 | Cluster inspection by agents only via read-only ServiceAccount kubeconfigs minted by `.claude/cluster-inspector/make-kubeconfig.sh` (short-lived TokenRequest tokens, gitignored) | admin kubeconfig + permission allowlist; long-lived SA token Secret; RBAC under `configs/` | RBAC (`view`, no Secrets) enforces read-only regardless of which kubectl verb an agent runs — also closes the `kubectl get secret` gap from the history-conventions session; short-lived tokens limit leak impact | active | +| 12 | 2026-09-27 | Management cluster built by one idempotent Ansible playbook, `ansible/mgmt.yml`, run from the laptop ([plan](plans/2026-09-27-0058-mgmt-cluster-ansible.md)) | cloud-init + k3s auto-deploy manifests; bash scripts with guards; OpenTofu + Helm | owner's choice; idempotent modules, one tool; only `ansible.builtin`, charts through k3s's helm-controller so nothing extra on the VM | active | +| 13 | 2026-09-27 | CAPI providers installed by cluster-api-operator 0.28.0 (Provider CRs), cert-manager v1.21.2 pre-installed | guarded `clusterctl init` | declarative; re-apply is a no-op; upgrade = `spec.version` bump; 0.28.0 built against CAPI 1.12.10. Consequence: no `clusterctl init/upgrade/move/delete` | active | +| 14 | 2026-09-27 | Management VM created by the playbook from the pinned Ubuntu 24.04 cloud image (VM 201, 192.168.4.21) | hand-made VM; the image-builder node template | reproducible from the repo; the image-builder template is a kubeadm node image, not meant for k3s | active | diff --git a/docs/phase-1-proxmox-capi.md b/docs/phase-1-proxmox-capi.md index 293ad00..e76bc8d 100644 --- a/docs/phase-1-proxmox-capi.md +++ b/docs/phase-1-proxmox-capi.md @@ -1,6 +1,6 @@ # Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler -Goal: a manually installed k3s management cluster runs CAPI + CAPMOX and creates workload +Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero. No custom code in this phase — only assembling upstream components. @@ -10,7 +10,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro | Role | Where | Size | | --- | --- | --- | -| Management cluster | 1 VM on Proxmox, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB | +| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB | | `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM | | `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 8 GB RAM | @@ -18,7 +18,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro - [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges) - [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested -- [ ] Step 2 — management cluster + `clusterctl init` +- [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers) - [ ] Step 3 — workload cluster `lab1` - [ ] Step 4 — CNI (Cilium) - [ ] Step 5 — MachineDeployment autoscaling annotations @@ -47,27 +47,50 @@ requires shared storage; with local storage keep `ALLOWED_NODES` to the node hol ## Step 2 — management cluster -```bash -curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.8+k3s1 sh -s - --disable traefik -mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config +One idempotent playbook, run from the laptop: [ansible/mgmt.yml](../ansible/mgmt.yml). It creates +the VM on `pv1-1` from the pinned Ubuntu 24.04 cloud image, installs k3s, then cert-manager and +cluster-api-operator (as k3s `HelmChart`s), then the five CAPI providers as operator Provider CRs. +Versions and VM parameters are in `ansible/group_vars/all.yml`. A re-run must report `changed=0`. -curl -L -o clusterctl https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-linux-amd64 -sudo install -m 0755 clusterctl /usr/local/bin/clusterctl -``` +Laptop prerequisites: -`~/.cluster-api/clusterctl.yaml` — start from [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example). +- root SSH to `pv1-1` with `~/.ssh/id_ed25519` (`ssh-copy-id root@192.168.0.8`); +- ansible-core 2.18 (`brew install ansible`, or prefix commands with `uvx --from 'ansible-core>=2.18,<2.19'`); +- `cp ansible/secrets.yml.example ansible/secrets.yml` and fill in the `capmox@pve!capi` token secret; +- `clusterctl` v1.12.11 for Step 3 (not `brew install clusterctl`, which installs 1.14.x): + + ```bash + curl -fL -o /tmp/clusterctl \ + https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-darwin-arm64 + install -m 0755 /tmp/clusterctl ~/bin/clusterctl && rm /tmp/clusterctl + ``` ```bash -clusterctl init \ - --core cluster-api:v1.12.11 \ - --bootstrap kubeadm:v1.12.11 \ - --control-plane kubeadm:v1.12.11 \ - --infrastructure proxmox:v0.9.1 \ - --ipam in-cluster:v1.1.1 +cd ansible +ansible-playbook mgmt.yml --tags vm # VM 201 on pv1-1, waits for SSH +ansible-playbook mgmt.yml --tags k3s # qemu-guest-agent, k3s, node Ready +ansible-playbook mgmt.yml --tags addons # cert-manager v1.21.2, cluster-api-operator 0.28.0 +ansible-playbook mgmt.yml --tags capi # capmox-variables Secret + 5 Provider CRs, waits for controllers +ansible-playbook mgmt.yml --tags kubeconfig # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfig +ansible-playbook mgmt.yml # everything; must be changed=0 on a built cluster ``` -Always pin versions explicitly — without them clusterctl installs the newest core (1.14.x) and -breaks the tested CAPMOX combination. +```bash +export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig +kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A +# all READY True, INSTALLEDVERSION v1.12.11 / v0.9.1 / v1.1.1 +``` + +cluster-api-operator replaces `clusterctl init`: + +- Upgrade a provider by bumping its version in `group_vars/all.yml` and re-running `--tags capi`, + one provider per run. +- Use `clusterctl` only for `generate cluster` (always with `--infrastructure proxmox:v0.9.1`), + `describe` and `get kubeconfig`. Never `init`, `upgrade`, `move` or `delete`: they need the + clusterctl inventory, which the operator doesn't create. +- Proxmox credentials live only in `ansible/secrets.yml`; `~/.cluster-api/clusterctl.yaml` (from + [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example)) holds just the + workload variables. ## Step 3 — workload cluster `lab1` @@ -165,6 +188,10 @@ Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, | clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` | | ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range | | `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` | +| Provider CR not `READY` / `capi` play keeps retrying | operator preflight or component download failed | `kubectl -n capi-operator-system logs deploy/cluster-api-operator`, `kubectl -n describe ` | +| Provider keeps broken components after a failed fetch | stale per-version cache (cluster-api-operator #1075) | delete that provider's `…-cache` Secret, re-run `--tags capi` | +| CAPMOX still uses the old token after editing `secrets.yml` | operator doesn't watch `configSecret` | new Secret name in the template + `configSecret` reference | +| `ansible-playbook mgmt.yml` reports `changed` on a built cluster | a task isn't idempotent | `-v` for command output; on the VM `k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml` | ```bash kubectl -n capmox-system logs deploy/capmox-controller-manager -f diff --git a/docs/plans-executions/2026-09-27-0058-mgmt-cluster-ansible.md b/docs/plans-executions/2026-09-27-0058-mgmt-cluster-ansible.md index c1a3fc5..459ab5d 100644 --- a/docs/plans-executions/2026-09-27-0058-mgmt-cluster-ansible.md +++ b/docs/plans-executions/2026-09-27-0058-mgmt-cluster-ansible.md @@ -7,8 +7,8 @@ Pairs with [docs/plans/2026-09-27-0058-mgmt-cluster-ansible.md](../plans/2026-09 - [x] Step 0 — Initial import commit + plan commit - [x] Step 1 — Scaffold `ansible/`, `.gitignore` exception, syntax/lint checks - [x] Step 2 — Run the playbook tag by tag against the lab (owner) -- [ ] Step 3 — Idempotency check (second full run `changed=0`) -- [ ] Step 4 — Docs, versions, decisions, changelog +- [x] Step 3 — Idempotency check (second full run `changed=0`) +- [x] Step 4 — Docs, versions, decisions, changelog ## Step 0 — Initial import + plan @@ -56,7 +56,7 @@ uvx --from 'ansible-core>=2.18,<2.19' ansible pve -m ping -o Play 1 asserts on them. - `clusterctl` is not installed on the laptop either (needed for phase-1 Step 3, not for this playbook). -## Step 2 — Playbook runs (in progress) +## Step 2 — Playbook runs ### `--tags vm`, first attempt (2026-09-27 09:30 CEST, owner) @@ -111,7 +111,7 @@ full playbook run re-applies it via the `inspector` task (`--apply-rbac`). **Worth noting:** the `.claude/settings.json` allowlist picked up four one-off entries approved during this check (`export KUBECONFIG=…`, three exact `kubectl … logs` commands). -## Step 3 — Idempotency check (in progress) +## Step 3 — Idempotency check ### Full run: `Apply provider manifests` always `changed` (2026-09-27 10:35 CEST) @@ -135,3 +135,33 @@ Also from this check: the widened inspector RBAC was not applied yet (`get nodes Forbidden), so no clean full run had completed before this fix. `clusterctl` v1.12.11 darwin/arm64 is installed in `~/bin`; `clusterctl generate cluster lab1 --infrastructure proxmox:v0.9.1 --target-namespace default --list-variables` lists no `PROXMOX_*` credentials, confirming `clusterctl.yaml` only needs workload values now. + +### Result (2026-09-27 ~10:38 CEST, owner) + +After the `data` fix, one run with the expected `changed` on the Secret, then: + +```text +PLAY RECAP +lab-mgmt : ok=19 changed=0 unreachable=0 failed=0 skipped=2 +pv1-1 : ok=3 changed=0 unreachable=0 failed=0 skipped=6 +``` + +Verified read-only with `lab-mgmt-ro`: node `lab-mgmt` Ready (v1.35.8+k3s1, Ubuntu 24.04.5); +all five Provider CRs `READY True` at v1.12.11 / v0.9.1 / v1.1.1; HelmCharts cert-manager v1.21.2 +and cluster-api-operator 0.28.0 not failed; `auth can-i delete machines` / `get secrets` → `no`. +The widened inspector RBAC was applied by this run's `inspector` task. + +## Step 4 — Docs (2026-09-27 10:40 CEST) + +- `docs/phase-1-proxmox-capi.md`: Step 2 rewritten around the playbook (prereqs incl. `clusterctl` + darwin/arm64 install into `~/bin`, tag-by-tag commands, operator rules), checklist ticked, + topology row, 4 troubleshooting rows (provider not Ready, stale cache #1075, token rotation, + non-idempotent task). +- `docs/versions.md`: cluster-api-operator 0.28.0, cert-manager v1.21.2, cloud image serial; + provider upgrade procedure. +- `docs/decisions.md` #12–14; `CLAUDE.md` lab fact + version rules (no `clusterctl init/upgrade/move/delete`); + `configs/capi/clusterctl.yaml.example` without controller credentials; README status, layout, decision 6. + +**Worth noting:** the mgmt VM runs Ubuntu 24.04.5 (cloud image), the node template 24.04.4 +(ISO) — different on purpose, they're separate images. Not covered by the plan and still open: +token rotation, VM resize, re-creating the VM (SSH `known_hosts` would conflict with `accept-new`). diff --git a/docs/versions.md b/docs/versions.md index e865fbf..8e65232 100644 --- a/docs/versions.md +++ b/docs/versions.md @@ -1,6 +1,7 @@ # Versions -Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go module proxy, +Verified 2026-09-24 (phase 1), 2026-09-26 (phase 2/3 candidates) and 2026-09-27 (management +cluster add-ons) from the Go module proxy, `dl.k8s.io`, `registry.k8s.io`, upstream Helm indexes and git tags. Re-verify before every deploy. ## Phase 1 — pinned @@ -12,6 +13,9 @@ Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go | IPAM provider in-cluster | **v1.1.1** | v1.1.1 | CAPMOX v0.9 requires ≥ 1.1.0 | | Workload Kubernetes | **v1.35.9** | v1.37.1 | CAPI 1.12 supports workload clusters up to 1.35 | | Management cluster (k3s) | **v1.35.8+k3s1** | v1.37.0+k3s1 | CAPI 1.12 supports management clusters 1.31–1.35 | +| cluster-api-operator (Helm chart) | **0.28.0** | 0.29.0 | installs the providers instead of `clusterctl init`; 0.28.0 is built against CAPI v1.12.10, 0.29.0 against v1.13.5 | +| cert-manager (Helm chart) | **v1.21.2** | v1.21.2 | required by the operator and all providers (webhook certs); supports k8s 1.33–1.36 | +| Management VM image | Ubuntu cloud image `noble/20260926` (24.04.5) | — | sha256 `6a81c375…5bb4d2`; `current/` is a rolling pointer, pin the dated serial | | cluster-autoscaler | **v1.35.2** | v1.36.1 | CA minor must match workload Kubernetes minor | | cluster-autoscaler Helm chart | **9.59.0** | 9.59.0 | default appVersion 1.35.0 → override `image.tag` | | Cilium | **1.20.2** | 1.20.2 | | @@ -27,6 +31,11 @@ Verified 2026-09-24 (phase 1) and 2026-09-26 (phase 2/3 candidates) from the Go When CAPMOX adds CAPI 1.13/1.14 to its compatibility table: bump CAPI core → k3s mgmt to 1.36+ → rebuild template with Kubernetes 1.36/1.37 → cluster-autoscaler to the matching minor. +All management-cluster pins live in `ansible/group_vars/all.yml`. Provider upgrades are a +`spec.version` bump there + `ansible-playbook mgmt.yml --tags capi`, one provider per run (the +operator blocks minor downgrades). Bump cluster-api-operator only to a release built against the +target CAPI minor. + ## Phase 2 / 3 — candidates (not pinned yet) | Component | Latest | Built against CAPI | Notes |