cluster-autoscaler-with-capi/docs/phase-1-proxmox-capi.md

# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler

Goal: a manually installed k3s management cluster runs CAPI + CAPMOX and creates workload
cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
No custom code in this phase — only assembling upstream components.

Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md).

## Topology

| Role | Where | Size |
| --- | --- | --- |
| Management cluster | 1 VM on Proxmox, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 8 GB RAM |

## Checklist

- [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
- [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
- [ ] Step 2 — management cluster + `clusterctl init`
- [ ] Step 3 — workload cluster `lab1`
- [ ] Step 4 — CNI (Cilium)
- [ ] Step 5 — MachineDeployment autoscaling annotations
- [ ] Step 6 — cluster-autoscaler
- [ ] Step 7 — scale tests

## Step 0 — Proxmox prep

```bash
pveum user add capmox@pve
pveum user token add capmox@pve capi -privsep 0          # token inherits user ACLs

pveum aclmod /                       -user capmox@pve -role PVEVMAdmin
pveum aclmod /                       -user capmox@pve -role PVEAuditor        # Sys.Audit etc.
pveum aclmod /storage                -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks
pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser        # NIC on any local bridge (PVE 8+)
```

Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`.

Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP
for the control plane endpoint in the same subnet.

Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node
requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template.

## Step 2 — management cluster

```bash
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.8+k3s1 sh -s - --disable traefik
mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config

curl -L -o clusterctl https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-linux-amd64
sudo install -m 0755 clusterctl /usr/local/bin/clusterctl
```

`~/.cluster-api/clusterctl.yaml` — start from [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example).

```bash
clusterctl init \
  --core cluster-api:v1.12.11 \
  --bootstrap kubeadm:v1.12.11 \
  --control-plane kubeadm:v1.12.11 \
  --infrastructure proxmox:v0.9.1 \
  --ipam in-cluster:v1.1.1
```

Always pin versions explicitly — without them clusterctl installs the newest core (1.14.x) and
breaks the tested CAPMOX combination.

## Step 3 — workload cluster `lab1`

```bash
clusterctl generate cluster lab1 \
  --infrastructure proxmox:v0.9.1 \
  --kubernetes-version v1.35.9 \
  --control-plane-machine-count 1 \
  --worker-machine-count 1 > lab1.yaml
```

Edit `lab1.yaml` before applying:

1. **Pod CIDR** — `Cluster.spec.clusterNetwork.pods.cidrBlocks` defaults to `192.168.0.0/16`, which
   overlaps the lab network → use e.g. `10.244.0.0/16`.
2. **kube-vip** — bump `ghcr.io/kube-vip/kube-vip:v0.7.1` → `v1.2.4`; keep `kube-vip-prepare.sh`
   (handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 8192`.
6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.

```bash
kubectl apply -f lab1.yaml
clusterctl describe cluster lab1 --show-conditions all
kubectl get proxmoxmachines,machines -w
clusterctl get kubeconfig lab1 > lab1.kubeconfig
```

Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses`). providerID is set by
the CAPMOX template as `proxmox://{{ ds.meta_data.instance_id }}` — don't touch it, the autoscaler
matches Nodes to Machines by it.

## Step 4 — Cilium

```bash
export KUBECONFIG=$PWD/lab1.kubeconfig
helm repo add cilium https://helm.cilium.io && helm repo update
helm install cilium cilium/cilium --version 1.20.2 -n kube-system --set ipam.mode=kubernetes
```

`ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default.

## Step 5 — autoscaling annotations

```yaml
metadata:
  name: lab1-workers
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "4"            # numSockets × numCores
    capacity.cluster-autoscaler.kubernetes.io/memory: "8Gi"
    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
    capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
```

Capacity annotations are required for scale-from-zero because CAPMOX does not publish
`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints`
annotations if pods target the group by selector/toleration.

## Step 6 — cluster-autoscaler

Install into the **same namespace as the `lab1-kubeconfig` secret** (here `default`), values in
[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):

```bash
helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update
helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \
  -f configs/cluster-autoscaler/values.yaml
```

The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5.

## Step 7 — scale tests

Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 2Gi each).

- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear.
- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.

`kubectl describe pod <pending>` shows `TriggeredScaleUp` / `NotTriggerScaleUp` events — fastest diagnostic.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |

```bash
kubectl -n capmox-system logs deploy/capmox-controller-manager -f
kubectl -n capi-system   logs deploy/capi-controller-manager -f
ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100'
```