cluster-autoscaler-with-capi/docs/phase-1-proxmox-capi.md

# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler

Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload
cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
No custom code in this phase — only assembling upstream components.

Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md).

## Topology

| Role | Where | Size |
| --- | --- | --- |
| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 4 GB RAM |

## Checklist

- [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
- [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
- [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers)
- [x] Step 3 — workload cluster `lab1` (VMs 102/103, `.31`/`.32`, VIP `.30`)
- [x] Step 4 — CNI (Cilium 1.20.2, `configs/cilium/values.yaml`)
- [ ] Step 5 — MachineDeployment autoscaling annotations
- [ ] Step 6 — cluster-autoscaler
- [ ] Step 7 — scale tests

## Step 0 — Proxmox prep

```bash
pveum user add capmox@pve
pveum user token add capmox@pve capi -privsep 0          # token inherits user ACLs

pveum aclmod /                       -user capmox@pve -role PVEVMAdmin
pveum aclmod /                       -user capmox@pve -role PVEAuditor        # Sys.Audit etc.
pveum aclmod /storage                -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks
pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser        # NIC on any local bridge (PVE 8+)
```

Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`.

Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP
for the control plane endpoint in the same subnet.

Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node
requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template.

## Step 2 — management cluster

One idempotent playbook, run from the laptop: [ansible/mgmt.yml](../ansible/mgmt.yml). It creates
the VM on `pv1-1` from the pinned Ubuntu 24.04 cloud image, installs k3s, then cert-manager and
cluster-api-operator (as k3s `HelmChart`s), then the five CAPI providers as operator Provider CRs.
Versions and VM parameters are in `ansible/group_vars/all.yml`. A re-run must report `changed=0`.

Laptop prerequisites:

- root SSH to `pv1-1` with `~/.ssh/id_ed25519` (`ssh-copy-id root@192.168.0.8`);
- ansible-core 2.18 (`brew install ansible`, or prefix commands with `uvx --from 'ansible-core>=2.18,<2.19'`);
- `cp ansible/secrets.yml.example ansible/secrets.yml` and fill in the `capmox@pve!capi` token secret;
- `clusterctl` v1.12.11 for Step 3 (not `brew install clusterctl`, which installs 1.14.x):

  ```bash
  curl -fL -o /tmp/clusterctl \
    https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-darwin-arm64
  install -m 0755 /tmp/clusterctl ~/bin/clusterctl && rm /tmp/clusterctl
  ```

```bash
cd ansible
ansible-playbook mgmt.yml --tags vm           # VM 201 on pv1-1, waits for SSH
ansible-playbook mgmt.yml --tags k3s          # qemu-guest-agent, k3s, node Ready
ansible-playbook mgmt.yml --tags addons       # cert-manager v1.21.2, cluster-api-operator 0.28.0
ansible-playbook mgmt.yml --tags capi         # capmox-variables Secret + 5 Provider CRs, waits for controllers
ansible-playbook mgmt.yml --tags kubeconfig   # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfigs
ansible-playbook mgmt.yml --tags inspector    # only the read-only kubeconfigs (lab-mgmt + inspector_workload_clusters); weekly token refresh
ansible-playbook mgmt.yml                     # everything; must be changed=0 on a built cluster
```

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A
# all READY True, INSTALLEDVERSION v1.12.11 / v0.9.1 / v1.1.1
```

cluster-api-operator replaces `clusterctl init`:

- Upgrade a provider by bumping its version in `group_vars/all.yml` and re-running `--tags capi`,
  one provider per run.
- Use `clusterctl` only for `generate cluster` (always with `--infrastructure proxmox:v0.9.1`),
  `describe` and `get kubeconfig`. Never `init`, `upgrade`, `move` or `delete`: they need the
  clusterctl inventory, which the operator doesn't create.
- Proxmox credentials live only in `ansible/secrets.yml`; `~/.cluster-api/clusterctl.yaml` (from
  [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example)) holds just the
  workload variables.

## Step 3 — workload cluster `lab1`

The generated and hand-edited manifest is committed as
[configs/capi/lab1.yaml](../configs/capi/lab1.yaml): template VMID 101, control-plane VIP
`192.168.4.30`, node IPs `192.168.4.31–39`. Its header holds the exact `clusterctl generate`
command (all variables as env vars, so no `~/.cluster-api/clusterctl.yaml` is needed). Regenerate
only when a version changes, then redo these edits:

1. **Pod CIDR** — `Cluster.spec.clusterNetwork.pods.cidrBlocks` defaults to `192.168.0.0/16`, which
   overlaps the lab network → use e.g. `10.244.0.0/16`.
2. **kube-vip** — bump `ghcr.io/kube-vip/kube-vip:v0.7.1` → `v1.2.4`; keep `kube-vip-prepare.sh`
   (handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 4096`.
6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl apply -f configs/capi/lab1.yaml
clusterctl describe cluster lab1 --show-conditions all
kubectl get proxmoxmachines,machines -w
clusterctl get kubeconfig lab1 > lab1.kubeconfig
```

Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io`;
plain `ipaddresses` resolves to the built-in `networking.k8s.io` type on k8s ≥ 1.33). providerID is set by
the CAPMOX template as `proxmox://{{ ds.meta_data.instance_id }}` — don't touch it, the autoscaler
matches Nodes to Machines by it.

## Step 4 — Cilium

Values in [configs/cilium/values.yaml](../configs/cilium/values.yaml); `upgrade --install` is
safe to re-run.

```bash
clusterctl get kubeconfig lab1 --kubeconfig ~/.kube/lab-mgmt.kubeconfig > ~/.kube/lab1.kubeconfig
chmod 600 ~/.kube/lab1.kubeconfig
export KUBECONFIG=~/.kube/lab1.kubeconfig

helm repo add cilium https://helm.cilium.io && helm repo update cilium
helm upgrade --install cilium cilium/cilium --version 1.20.2 -n kube-system \
  -f configs/cilium/values.yaml --wait --timeout 10m

kubectl get nodes                                          # both Ready
kubectl -n kube-system get pods -o wide                    # cilium DaemonSet 2/2, operator 1/1, coredns Running
```

- `ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default.
- `operator.replicas: 1`: the default 2 replicas with node anti-affinity put one on a worker, and a
  kube-system pod without a PDB blocks scale-down to zero.
- kube-proxy stays (kubeadm default); no kube-proxy replacement in this phase.

## Step 5 — autoscaling annotations

```yaml
metadata:
  name: lab1-workers
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "4"            # numSockets × numCores
    capacity.cluster-autoscaler.kubernetes.io/memory: "4Gi"       # memoryMiB 4096
    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
    capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
```

They are part of `MachineDeployment/lab1-workers` in [configs/capi/lab1.yaml](../configs/capi/lab1.yaml);
re-apply the file (the only diff is these annotations; `replicas` isn't in the manifest, so it
stays as is):

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl diff -f configs/capi/lab1.yaml       # expect only the six annotations on lab1-workers
kubectl apply -f configs/capi/lab1.yaml
```

Capacity annotations are required for scale-from-zero because CAPMOX does not publish
`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints`
annotations if pods target the group by selector/toleration. They have no effect until
cluster-autoscaler runs (Step 6).

## Step 6 — cluster-autoscaler

Install into the **same namespace as the `lab1-kubeconfig` secret** (here `default`), values in
[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):

```bash
helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update
helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \
  -f configs/cluster-autoscaler/values.yaml
```

The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5.

## Step 7 — scale tests

Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 1Gi each →
CPU-bound at 2 pods per 4-core / 4 GiB worker).

- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear.
- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.

`kubectl describe pod <pending>` shows `TriggeredScaleUp` / `NotTriggerScaleUp` events — fastest diagnostic.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
| Provider CR not `READY` / `capi` play keeps retrying | operator preflight or component download failed | `kubectl -n capi-operator-system logs deploy/cluster-api-operator`, `kubectl -n <ns> describe <kind> <name>` |
| Provider keeps broken components after a failed fetch | stale per-version cache (cluster-api-operator #1075) | delete that provider's `…-cache` Secret, re-run `--tags capi` |
| CAPMOX still uses the old token after editing `secrets.yml` | operator doesn't watch `configSecret` | new Secret name in the template + `configSecret` reference |
| `ansible-playbook mgmt.yml` reports `changed` on a built cluster | a task isn't idempotent | `-v` for command output; on the VM `k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml` |

```bash
kubectl -n capmox-system logs deploy/capmox-controller-manager -f
kubectl -n capi-system   logs deploy/capi-controller-manager -f
ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100'
```