kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document Ansible-built management cluster (plan step 4)

Browse Source 
Phase-1 Step 2 rewritten around ansible/mgmt.yml and cluster-api-operator,
version pins for operator/cert-manager/cloud image, decisions #12-14,
execution log and changelog. Phase-1 Step 2 checked off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-27 10:41:03 +02:00
co-authored by Claude Opus 5.5
parent aaa1ed64e2
commit d8efa3b961
9 changed files with 121 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-1-proxmox-capi.md
+45 -18
View File
@@ -1,6 +1,6 @@
# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler
Goal: a manually installed k3s management cluster runs CAPI + CAPMOX and creates workload
Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload
cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
No custom code in this phase — only assembling upstream components.
@@ -10,7 +10,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
| Role | Where | Size |
| --- | --- | --- |
| Management cluster | 1 VM on Proxmox, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 8 GB RAM |
@@ -18,7 +18,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
- [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
- [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
- [ ] Step 2 — management cluster + `clusterctl init`
- [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers)
- [ ] Step 3 — workload cluster `lab1`
- [ ] Step 4 — CNI (Cilium)
- [ ] Step 5 — MachineDeployment autoscaling annotations
@@ -47,27 +47,50 @@ requires shared storage; with local storage keep `ALLOWED_NODES` to the node hol
## Step 2 — management cluster
```bash
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.8+k3s1 sh -s - --disable traefik
mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config
One idempotent playbook, run from the laptop: [ansible/mgmt.yml](../ansible/mgmt.yml). It creates
the VM on `pv1-1` from the pinned Ubuntu 24.04 cloud image, installs k3s, then cert-manager and
cluster-api-operator (as k3s `HelmChart`s), then the five CAPI providers as operator Provider CRs.
Versions and VM parameters are in `ansible/group_vars/all.yml`. A re-run must report `changed=0`.
curl -L -o clusterctl https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-linux-amd64
sudo install -m 0755 clusterctl /usr/local/bin/clusterctl
```
Laptop prerequisites:
`~/.cluster-api/clusterctl.yaml` — start from [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example).
- root SSH to `pv1-1` with `~/.ssh/id_ed25519` (`ssh-copy-id root@192.168.0.8`);
- ansible-core 2.18 (`brew install ansible`, or prefix commands with `uvx --from 'ansible-core>=2.18,<2.19'`);
- `cp ansible/secrets.yml.example ansible/secrets.yml` and fill in the `capmox@pve!capi` token secret;
- `clusterctl` v1.12.11 for Step 3 (not `brew install clusterctl`, which installs 1.14.x):
```bash
curl -fL -o /tmp/clusterctl \
https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-darwin-arm64
install -m 0755 /tmp/clusterctl ~/bin/clusterctl && rm /tmp/clusterctl
```
```bash
clusterctl init \
--core cluster-api:v1.12.11 \
--bootstrap kubeadm:v1.12.11 \
--control-plane kubeadm:v1.12.11 \
--infrastructure proxmox:v0.9.1 \
--ipam in-cluster:v1.1.1
cd ansible
ansible-playbook mgmt.yml --tags vm # VM 201 on pv1-1, waits for SSH
ansible-playbook mgmt.yml --tags k3s # qemu-guest-agent, k3s, node Ready
ansible-playbook mgmt.yml --tags addons # cert-manager v1.21.2, cluster-api-operator 0.28.0
ansible-playbook mgmt.yml --tags capi # capmox-variables Secret + 5 Provider CRs, waits for controllers
ansible-playbook mgmt.yml --tags kubeconfig # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfig
ansible-playbook mgmt.yml # everything; must be changed=0 on a built cluster
```
Always pin versions explicitly — without them clusterctl installs the newest core (1.14.x) and
breaks the tested CAPMOX combination.
```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A
# all READY True, INSTALLEDVERSION v1.12.11 / v0.9.1 / v1.1.1
```
cluster-api-operator replaces `clusterctl init`:
- Upgrade a provider by bumping its version in `group_vars/all.yml` and re-running `--tags capi`,
one provider per run.
- Use `clusterctl` only for `generate cluster` (always with `--infrastructure proxmox:v0.9.1`),
`describe` and `get kubeconfig`. Never `init`, `upgrade`, `move` or `delete`: they need the
clusterctl inventory, which the operator doesn't create.
- Proxmox credentials live only in `ansible/secrets.yml`; `~/.cluster-api/clusterctl.yaml` (from
[configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example)) holds just the
workload variables.
## Step 3 — workload cluster `lab1`
@@ -165,6 +188,10 @@ Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods,
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
| Provider CR not `READY` / `capi` play keeps retrying | operator preflight or component download failed | `kubectl -n capi-operator-system logs deploy/cluster-api-operator`, `kubectl -n <ns> describe <kind> <name>` |
| Provider keeps broken components after a failed fetch | stale per-version cache (cluster-api-operator #1075) | delete that provider's `…-cache` Secret, re-run `--tags capi` |
| CAPMOX still uses the old token after editing `secrets.yml` | operator doesn't watch `configSecret` | new Secret name in the template + `configSecret` reference |
| `ansible-playbook mgmt.yml` reports `changed` on a built cluster | a task isn't idempotent | `-v` for command output; on the VM `k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml` |
```bash
kubectl -n capmox-system logs deploy/capmox-controller-manager -f