cluster-autoscaler-with-capi/docs/phase-1-proxmox-capi.md

# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler

Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload
cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
No custom code in this phase — only assembling upstream components.

Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md).

## Topology

| Role | Where | Size |
| --- | --- | --- |
| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 2 GB RAM |

## Checklist

- [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
- [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
- [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers)
- [x] Step 3 — workload cluster `lab1` (VMs 102/103, `.31`/`.32`, VIP `.30`)
- [x] Step 4 — CNI (Cilium 1.20.2, `configs/cilium/values.yaml`)
- [x] Step 5 — MachineDeployment autoscaling annotations
- [ ] Step 6 — cluster-autoscaler
- [ ] Step 7 — scale tests

## Step 0 — Proxmox prep

```bash
pveum user add capmox@pve
pveum user token add capmox@pve capi -privsep 0          # token inherits user ACLs

pveum aclmod /                       -user capmox@pve -role PVEVMAdmin
pveum aclmod /                       -user capmox@pve -role PVEAuditor        # Sys.Audit etc.
pveum aclmod /storage                -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks
pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser        # NIC on any local bridge (PVE 8+)
```

Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`.

Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP
for the control plane endpoint in the same subnet.

Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node
requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template.

## Step 2 — management cluster

One idempotent playbook, run from the laptop: [ansible/mgmt.yml](../ansible/mgmt.yml). It creates
the VM on `pv1-1` from the pinned Ubuntu 24.04 cloud image, installs k3s, then cert-manager and
cluster-api-operator (as k3s `HelmChart`s), then the five CAPI providers as operator Provider CRs.
Versions and VM parameters are in `ansible/group_vars/all.yml`. A re-run must report `changed=0`.

Laptop prerequisites:

- root SSH to `pv1-1` with `~/.ssh/id_ed25519` (`ssh-copy-id root@192.168.0.8`);
- ansible-core 2.18 (`brew install ansible`, or prefix commands with `uvx --from 'ansible-core>=2.18,<2.19'`);
- `cp ansible/secrets.yml.example ansible/secrets.yml` and fill in the `capmox@pve!capi` token secret;
- `clusterctl` v1.12.11 for Step 3 (not `brew install clusterctl`, which installs 1.14.x):

  ```bash
  curl -fL -o /tmp/clusterctl \
    https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-darwin-arm64
  install -m 0755 /tmp/clusterctl ~/bin/clusterctl && rm /tmp/clusterctl
  ```

```bash
cd ansible
ansible-playbook mgmt.yml --tags vm           # VM 201 on pv1-1, waits for SSH
ansible-playbook mgmt.yml --tags k3s          # qemu-guest-agent, k3s, node Ready
ansible-playbook mgmt.yml --tags addons       # cert-manager v1.21.2, cluster-api-operator 0.28.0
ansible-playbook mgmt.yml --tags capi         # capmox-variables Secret + 5 Provider CRs, waits for controllers
ansible-playbook mgmt.yml --tags kubeconfig   # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfigs
ansible-playbook mgmt.yml --tags inspector    # only the read-only kubeconfigs (lab-mgmt + inspector_workload_clusters); weekly token refresh
ansible-playbook mgmt.yml                     # everything; must be changed=0 on a built cluster
```

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A
# all READY True, INSTALLEDVERSION v1.12.11 / v0.9.1 / v1.1.1
```

cluster-api-operator replaces `clusterctl init`:

- Upgrade a provider by bumping its version in `group_vars/all.yml` and re-running `--tags capi`,
  one provider per run.
- Use `clusterctl` only for `generate cluster` (always with `--infrastructure proxmox:v0.9.1`),
  `describe` and `get kubeconfig`. Never `init`, `upgrade`, `move` or `delete`: they need the
  clusterctl inventory, which the operator doesn't create.
- Proxmox credentials live only in `ansible/secrets.yml`; `~/.cluster-api/clusterctl.yaml` (from
  [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example)) holds just the
  workload variables.

## Step 3 — workload cluster `lab1`

The generated and hand-edited manifest is committed as
[configs/capi/lab1.yaml](../configs/capi/lab1.yaml): template VMID 101, control-plane VIP
`192.168.4.30`, node IPs `192.168.4.31–39`. Its header holds the exact `clusterctl generate`
command (all variables as env vars, so no `~/.cluster-api/clusterctl.yaml` is needed). Regenerate
only when a version changes, then redo these edits:

1. **Pod CIDR** — `Cluster.spec.clusterNetwork.pods.cidrBlocks` defaults to `192.168.0.0/16`, which
   overlaps the lab network → use e.g. `10.244.0.0/16`.
2. **kube-vip** — bump `ghcr.io/kube-vip/kube-vip:v0.7.1` → `v1.2.4`; keep `kube-vip-prepare.sh`
   (handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
5. **Worker size** — rename the worker template to `ProxmoxMachineTemplate/lab1-worker-2g` (and the MD
   `infrastructureRef`): `numSockets: 1`, `numCores: 4`, `memoryMiB: 2048`. To resize later, add a
   template with a new name and point the MD at it; CAPI then rolls the workers. Don't edit a
   template in place.
6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl apply -f configs/capi/lab1.yaml
clusterctl describe cluster lab1 --show-conditions all
kubectl get proxmoxmachines,machines -w
clusterctl get kubeconfig lab1 > lab1.kubeconfig
```

Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io`;
plain `ipaddresses` resolves to the built-in `networking.k8s.io` type on k8s ≥ 1.33). providerID is set by
the CAPMOX template as `proxmox://{{ ds.meta_data.instance_id }}` — don't touch it, the autoscaler
matches Nodes to Machines by it.

## Step 4 — Cilium

Values in [configs/cilium/values.yaml](../configs/cilium/values.yaml); `upgrade --install` is
safe to re-run.

```bash
clusterctl get kubeconfig lab1 --kubeconfig ~/.kube/lab-mgmt.kubeconfig > ~/.kube/lab1.kubeconfig
chmod 600 ~/.kube/lab1.kubeconfig
export KUBECONFIG=~/.kube/lab1.kubeconfig

helm repo add cilium https://helm.cilium.io && helm repo update cilium
helm upgrade --install cilium cilium/cilium --version 1.20.2 -n kube-system \
  -f configs/cilium/values.yaml --wait --timeout 10m

kubectl get nodes                                          # both Ready
kubectl -n kube-system get pods -o wide                    # cilium DaemonSet 2/2, operator 1/1, coredns Running
```

- `ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default.
- `operator.replicas: 1` + `operator.nodeSelector` on `node-role.kubernetes.io/control-plane`: the
  default 2 replicas with node anti-affinity put one on a worker, and a kube-system pod without a
  PDB blocks scale-down to zero. `replicas: 1` alone isn't enough: the single replica was scheduled
  onto the worker (the chart tolerates the control-plane taint, so the selector pins it there).
- kube-proxy stays (kubeadm default); no kube-proxy replacement in this phase.

## Step 5 — autoscaling annotations

```yaml
metadata:
  name: lab1-workers
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "4"            # numSockets × numCores
    capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi"       # memoryMiB 2048
    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
    capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
```

They are part of `MachineDeployment/lab1-workers` in [configs/capi/lab1.yaml](../configs/capi/lab1.yaml);
re-apply the file (the only diff is these annotations; `replicas` isn't in the manifest, so it
stays as is):

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl diff -f configs/capi/lab1.yaml       # expect only the six annotations on lab1-workers
kubectl apply -f configs/capi/lab1.yaml
```

Capacity annotations are required for scale-from-zero because CAPMOX does not publish
`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints`
annotations if pods target the group by selector/toleration. They have no effect until
cluster-autoscaler runs (Step 6).

## Step 6 — cluster-autoscaler

Install into the **same namespace as the `lab1-kubeconfig` secret** (here `default`), values in
[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler \
  --version 9.59.0 -n default -f configs/cluster-autoscaler/values.yaml
```

The values add `rbac.additionalRules` to read `proxmoxmachinetemplates`. The chart's ClusterRole
doesn't cover infrastructure templates, and without them the first main loop hangs (see
troubleshooting). Health check: the log shows `Starting main loop` followed by regular
`Autoscaler loop triggered` lines, and `cluster-autoscaler-status` on lab1 leaves `Initializing`.

CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`. Its log must
show the node group; the line is logged at `v: 4` on every loop (`clusterapi_controller.go`,
CA 1.35.2):

```bash
kubectl logs deploy/ca-clusterapi-cluster-autoscaler -n default --kubeconfig ~/.kube/lab-mgmt.kubeconfig \
  | grep 'discovered node group'
# discovered node group: MachineDeployment/default/lab1-workers (min: 0, max: 5, replicas: 1)
```

The status ConfigMap and the leader-election lock live on the workload cluster, not on the management
cluster: `kubectl get cm cluster-autoscaler-status -n kube-system --kubeconfig ~/.kube/lab1.kubeconfig`
(chart arg `--namespace=kube-system`, because `clusterAPIConfigMapsNamespace` is empty).

## Step 7 — scale tests

Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 512Mi each →
CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocatable).

- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
- [ ] Scale-down: `--replicas=0` → after `scale-down-delay-after-add` (2m) + `scale-down-unneeded-time`
  (1m) Machines and VMs disappear. That's the aggressive lab profile in the CA values; per-MD overrides are
  `cluster.x-k8s.io/autoscaling-options-{scaledownunneededtime,scaledownutilizationthreshold,…}` annotations.
- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.

`kubectl describe pod <pending>` shows `TriggeredScaleUp` / `NotTriggerScaleUp` events — fastest diagnostic.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
| CA runs and discovers the node group but never scales; log stops at `waiting for cache sync on infrastructure resource`, `proxmoxmachinetemplates … forbidden` every ~40 s; status ConfigMap stuck on `Initializing` | chart 9.59.0 ClusterRole has no rule for `infrastructure.cluster.x-k8s.io`; CA's first main loop waits for a template informer that can never sync | `rbac.additionalRules` with get/list/watch on `proxmoxmachinetemplates` in the CA values, `helm upgrade` |
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
| scale-up adds one node too many (6 pods at 2 per node → 4 workers, second plan `3->4` about 70 s after `1->3`) | a new node registers with Cilium's `node.cilium.io/agent-not-ready:NoSchedule`; once registered it's no longer an "upcoming" template node, and CA sees pending pods that don't fit on it | `extraArgs.startup-taint: node.cilium.io/agent-not-ready` in the CA values (CA treats the node as still starting) |
| CA scales the MD up, new Machines stay without a VM; ProxmoxMachine `VMProvisionFailed: cannot reserve 4294967296B of memory on node pv1-1: …B available memory left` although `free` shows GBs available | CAPMOX v0.9.1 scheduler (`GetReservableMemoryBytes`): node total × `schedulerHints.memoryAdjustment` (default 100%) minus the **configured** memory of every non-template VM, **running or stopped**, and every container. Stopped VMs 100/1001/2001 hold 12 GiB | free the reservation (delete unused stopped VMs, or convert them to templates), or set `ProxmoxCluster.spec.schedulerHints.memoryAdjustment` > 100 (overcommit); keep the MD max-size within what the host can actually run |
| MD rollout never finishes and CA never scales down: CA logs `Node … will be skipped as it is currently under rollout`, new ProxmoxMachine `cannot reserve …B of memory` | deadlock. Default rollout surge (`maxSurge 1 / maxUnavailable 0`) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (`ScaleDownNodeUpgradeProcessor`) | `spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1}` on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory |
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
| `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` |
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
| Provider CR not `READY` / `capi` play keeps retrying | operator preflight or component download failed | `kubectl -n capi-operator-system logs deploy/cluster-api-operator`, `kubectl -n <ns> describe <kind> <name>` |
| Provider keeps broken components after a failed fetch | stale per-version cache (cluster-api-operator #1075) | delete that provider's `…-cache` Secret, re-run `--tags capi` |
| CAPMOX still uses the old token after editing `secrets.yml` | operator doesn't watch `configSecret` | new Secret name in the template + `configSecret` reference |
| `ansible-playbook mgmt.yml` reports `changed` on a built cluster | a task isn't idempotent | `-v` for command output; on the VM `k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml` |

```bash
kubectl -n capmox-system logs deploy/capmox-controller-manager -f
kubectl -n capi-system   logs deploy/capi-controller-manager -f
ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100'
```