cluster-autoscaler-with-capi/docs/phase-1-proxmox-capi.md

# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler

Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload
cluster `lab1` as Proxmox VMs on a two-node PVE cluster; cluster-autoscaler scales its workers 0–18
in one node group per PVE node, balanced, including scale-from-zero.
No custom code in this phase — only assembling upstream components.

Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md).

## Topology

| Role | Where | Size |
| --- | --- | --- |
| PVE cluster `homelab` | `pv1-1` 192.168.0.8 (API endpoint for CAPMOX), `pv1-2` 192.168.0.10; `local-lvm` on each, no shared storage | 16 / 12 cores, 29.2 / 40.9 GiB |
| Node template | `ubuntu-2404-kube-v1.35.9`: VMID 101 on `pv1-1`, copy 9101 on `pv1-2` | 20 GB |
| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
| `lab1` control plane | 1 VM on `pv1-1`, VIP `.30` via kube-vip | 2 vCPU, 4 GB RAM |
| `lab1` workers | MachineDeployments `lab1-workers-pv1-1` (min 0 / max 6) and `lab1-workers-pv1-2` (min 0 / max 12); node IP pool `.31–.50` | 4 vCPU, 2 GB RAM |

## Checklist

- [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
- [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
- [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers)
- [x] Step 3 — workload cluster `lab1` (VMs 102/103, `.31`/`.32`, VIP `.30`)
- [x] Step 4 — CNI (Cilium 1.20.2, `configs/cilium/values.yaml`)
- [x] Step 5 — MachineDeployment autoscaling annotations
- [ ] Step 6 — cluster-autoscaler
- [ ] Step 7 — scale tests

## Step 0 — Proxmox prep

```bash
pveum user add capmox@pve
pveum user token add capmox@pve capi -privsep 0          # token inherits user ACLs

pveum aclmod /                       -user capmox@pve -role PVEVMAdmin
pveum aclmod /                       -user capmox@pve -role PVEAuditor        # Sys.Audit etc.
pveum aclmod /storage                -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks
pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser        # NIC on any local bridge (PVE 8+)
```

Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`.

The token and ACLs are cluster-wide in a PVE cluster, so a node that joins later needs nothing here.

Network: `vmbr0` is VLAN-aware on **every** PVE node, and VMs live in VLAN 4. On a new node, add
`bridge-vlan-aware yes` + `bridge-vids 2-10` (or wider) to `iface vmbr0`, run `ifreload -a`, and
trunk VLAN 4 on its switch port. CAPMOX writes `bridge`/`vlan` into the VM without checking them.
Reserve a node IP range outside DHCP and one VIP for the control plane endpoint in the same subnet.

Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2.
- PVE refuses to clone a VM to another node from non-shared storage, and CAPMOX v0.9.1 always
  passes the scheduled node as the clone target.
- `templateSelector` needs exactly one match in the whole PVE cluster, so it can't pick a
  per-node copy.
- Without shared storage, therefore, every node gets its own template VMID (VMIDs are
  cluster-wide) and its own worker MachineDeployment. Each MD's `ProxmoxMachineTemplate` sets
  `sourceNode`, `templateID` and `allowedNodes: [<node>]`.
- `ProxmoxCluster.allowedNodes` stays `[pv1-1]` for the control plane. The copy procedure is in
  [image-builder-proxmox.md](image-builder-proxmox.md#template-on-other-pve-nodes), and the
  reasoning in [decision #18](decisions.md).

## Step 2 — management cluster

One idempotent playbook, run from the laptop: [ansible/mgmt.yml](../ansible/mgmt.yml). It creates
the VM on `pv1-1` from the pinned Ubuntu 24.04 cloud image, installs k3s, then cert-manager and
cluster-api-operator (as k3s `HelmChart`s), then the five CAPI providers as operator Provider CRs.
Versions and VM parameters are in `ansible/group_vars/all.yml`. A re-run must report `changed=0`.

Laptop prerequisites:

- root SSH to `pv1-1` with `~/.ssh/id_ed25519` (`ssh-copy-id root@192.168.0.8`);
- ansible-core 2.18 (`brew install ansible`, or prefix commands with `uvx --from 'ansible-core>=2.18,<2.19'`);
- `cp ansible/secrets.yml.example ansible/secrets.yml` and fill in the `capmox@pve!capi` token secret;
- `clusterctl` v1.12.11 for Step 3 (not `brew install clusterctl`, which installs 1.14.x):

  ```bash
  curl -fL -o /tmp/clusterctl \
    https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-darwin-arm64
  install -m 0755 /tmp/clusterctl ~/bin/clusterctl && rm /tmp/clusterctl
  ```

```bash
cd ansible
ansible-playbook mgmt.yml --tags vm           # VM 201 on pv1-1, waits for SSH
ansible-playbook mgmt.yml --tags k3s          # qemu-guest-agent, k3s, node Ready
ansible-playbook mgmt.yml --tags addons       # cert-manager v1.21.2, cluster-api-operator 0.28.0
ansible-playbook mgmt.yml --tags capi         # capmox-variables Secret + 5 Provider CRs, waits for controllers
ansible-playbook mgmt.yml --tags kubeconfig   # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfigs
ansible-playbook mgmt.yml --tags inspector    # only the read-only kubeconfigs (lab-mgmt + inspector_workload_clusters); weekly token refresh
ansible-playbook mgmt.yml                     # everything; must be changed=0 on a built cluster
```

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,ipamproviders -A
# all READY True, INSTALLEDVERSION v1.12.11 / v0.9.1 / v1.1.1
```

cluster-api-operator replaces `clusterctl init`:

- Upgrade a provider by bumping its version in `group_vars/all.yml` and re-running `--tags capi`,
  one provider per run.
- Use `clusterctl` only for `generate cluster` (always with `--infrastructure proxmox:v0.9.1`),
  `describe` and `get kubeconfig`. Never `init`, `upgrade`, `move` or `delete`: they need the
  clusterctl inventory, which the operator doesn't create.
- Proxmox credentials live only in `ansible/secrets.yml`; `~/.cluster-api/clusterctl.yaml` (from
  [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example)) holds just the
  workload variables.

## Step 3 — workload cluster `lab1`

The generated and hand-edited manifest is committed as
[configs/capi/lab1.yaml](../configs/capi/lab1.yaml): template VMID 101, control-plane VIP
`192.168.4.30`, node IPs `192.168.4.31–39`. Its header holds the exact `clusterctl generate`
command (all variables as env vars, so no `~/.cluster-api/clusterctl.yaml` is needed). Regenerate
only when a version changes, then redo these edits:

1. **Pod CIDR** — `Cluster.spec.clusterNetwork.pods.cidrBlocks` defaults to `192.168.0.0/16`, which
   overlaps the lab network → use e.g. `10.244.0.0/16`.
2. **kube-vip** — bump `ghcr.io/kube-vip/kube-vip:v0.7.1` → `v1.2.4`; keep `kube-vip-prepare.sh`
   (handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
5. **Worker size** — rename the worker template to `ProxmoxMachineTemplate/lab1-worker-2g` (and the MD
   `infrastructureRef`): `numSockets: 1`, `numCores: 4`, `memoryMiB: 2048`. To resize later, add a
   template with a new name and point the MD at it; CAPI then rolls the workers. Don't edit a
   template in place.
6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl apply -f configs/capi/lab1.yaml
clusterctl describe cluster lab1 --show-conditions all
kubectl get proxmoxmachines,machines -w
clusterctl get kubeconfig lab1 > lab1.kubeconfig
```

Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io`;
plain `ipaddresses` resolves to the built-in `networking.k8s.io` type on k8s ≥ 1.33). providerID is set by
the CAPMOX template as `proxmox://{{ ds.meta_data.instance_id }}` — don't touch it, the autoscaler
matches Nodes to Machines by it.

## Step 4 — Cilium

Values in [configs/cilium/values.yaml](../configs/cilium/values.yaml); `upgrade --install` is
safe to re-run.

```bash
clusterctl get kubeconfig lab1 --kubeconfig ~/.kube/lab-mgmt.kubeconfig > ~/.kube/lab1.kubeconfig
chmod 600 ~/.kube/lab1.kubeconfig
export KUBECONFIG=~/.kube/lab1.kubeconfig

helm repo add cilium https://helm.cilium.io && helm repo update cilium
helm upgrade --install cilium cilium/cilium --version 1.20.2 -n kube-system \
  -f configs/cilium/values.yaml --wait --timeout 10m

kubectl get nodes                                          # both Ready
kubectl -n kube-system get pods -o wide                    # cilium DaemonSet 2/2, operator 1/1, coredns Running
```

- `ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default.
- `operator.replicas: 1` + `operator.nodeSelector` on `node-role.kubernetes.io/control-plane`: the
  default 2 replicas with node anti-affinity put one on a worker, and a kube-system pod without a
  PDB blocks scale-down to zero. `replicas: 1` alone isn't enough: the single replica was scheduled
  onto the worker (the chart tolerates the control-plane taint, so the selector pins it there).
- kube-proxy stays (kubeadm default); no kube-proxy replacement in this phase.

## Step 5 — autoscaling annotations

Both worker MDs carry the same block. Only the name and max-size differ (6 on `pv1-1`, 12 on
`pv1-2`):

```yaml
metadata:
  name: lab1-workers-pv1-2
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "4"            # numSockets × numCores
    capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi"       # memoryMiB 2048
    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
    capacity.cluster-autoscaler.kubernetes.io/maxPods: "110"      # not "maximum-pods" (ignored)
    capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64"
spec:
  template:
    metadata:
      labels:
        node.cluster.x-k8s.io/pool: workers    # synced to the Node by CAPI; CA --balancing-label
```

They are part of both MDs in [configs/capi/lab1.yaml](../configs/capi/lab1.yaml). Re-apply the file.
`replicas` isn't in the manifest, so it stays as it is, and a new MD starts at the min-size:

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
kubectl diff -f configs/capi/lab1.yaml
kubectl apply -f configs/capi/lab1.yaml
```

Scale-from-zero needs capacity annotations because CAPMOX does not publish `status.capacity`, and
they must match the `ProxmoxMachineTemplate`. The `.../labels` annotation is needed as well, for two
reasons in CA 1.35.2:
- It reads MD template labels from `spec.template.spec.metadata.labels`, a path that doesn't exist in
  CAPI v1beta2.
- CAPMOX publishes no `status.nodeInfo`, so the from-zero template node would otherwise get
  `kubernetes.io/os=""` and `arch=""`. A pod with an `os: linux` selector then can't trigger a
  scale-up.

Add a `.../taints` annotation if pods target the group by toleration. None of this has any effect
until cluster-autoscaler runs (Step 6).

## Step 6 — cluster-autoscaler

Install into the **same namespace as the `lab1-kubeconfig` secret** (here `default`), values in
[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):

```bash
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler \
  --version 9.59.0 -n default -f configs/cluster-autoscaler/values.yaml
```

The values add `rbac.additionalRules` to read `proxmoxmachinetemplates`. The chart's ClusterRole
doesn't cover infrastructure templates, and without them the first main loop hangs (see
troubleshooting). Health check: the log shows `Starting main loop` followed by regular
`Autoscaler loop triggered` lines, and `cluster-autoscaler-status` on lab1 leaves `Initializing`.

CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`.

**Settings in the values that matter with two node groups:**
- `balancing-label: node.cluster.x-k8s.io/pool`. CA compares groups only on this label, so a
  scale-up is split across both PVE nodes.
- `scan-interval: 1m`. Unschedulable pods and finished scale-ups/downs still start a loop
  immediately; only the idle scale-down evaluation runs less often.
- `v: 2`.

**Checking the node groups:** `discovered node group` is only logged at `v: 4`
(`clusterapi_controller.go:800`). At `v: 2`, check the node groups in the status ConfigMap (below)
instead. A balanced scale-up logs:

```text
Splitting scale-up between 2 similar node groups: {MachineDeployment/default/lab1-workers-pv1-2, MachineDeployment/default/lab1-workers-pv1-1}
Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}]
```

The status ConfigMap and the leader-election lock live on the workload cluster, not on the management
cluster: `kubectl get cm cluster-autoscaler-status -n kube-system --kubeconfig ~/.kube/lab1.kubeconfig`
(chart arg `--namespace=kube-system`, because `clusterAPIConfigMapsNamespace` is empty).

## Step 7 — scale tests

Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 512Mi each →
CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocatable).

- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
- [ ] Scale-down: `--replicas=0` → after `scale-down-delay-after-add` (2m) + `scale-down-unneeded-time`
  (1m) Machines and VMs disappear. That's the aggressive lab profile in the CA values; per-MD overrides are
  `cluster.x-k8s.io/autoscaling-options-{scaledownunneededtime,scaledownutilizationthreshold,…}` annotations.
- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.

`kubectl describe pod <pending>` shows `TriggeredScaleUp` / `NotTriggerScaleUp` events — fastest diagnostic.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
| CA runs and discovers the node group but never scales; log stops at `waiting for cache sync on infrastructure resource`, `proxmoxmachinetemplates … forbidden` every ~40 s; status ConfigMap stuck on `Initializing` | chart 9.59.0 ClusterRole has no rule for `infrastructure.cluster.x-k8s.io`; CA's first main loop waits for a template informer that can never sync | `rbac.additionalRules` with get/list/watch on `proxmoxmachinetemplates` in the CA values, `helm upgrade` |
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
| scale-up adds one node too many (6 pods at 2 per node → 4 workers, second plan `3->4` about 70 s after `1->3`) | a new node registers with Cilium's `node.cilium.io/agent-not-ready:NoSchedule`; once registered it's no longer an "upcoming" template node, and CA sees pending pods that don't fit on it | `extraArgs.startup-taint: node.cilium.io/agent-not-ready` in the CA values (CA treats the node as still starting) |
| CA scales the MD up, new Machines stay without a VM; ProxmoxMachine `VMProvisionFailed: cannot reserve 4294967296B of memory on node pv1-1: …B available memory left` although `free` shows GBs available | CAPMOX v0.9.1 scheduler (`GetReservableMemoryBytes`): node total × `schedulerHints.memoryAdjustment` (default 100%) minus the **configured** memory of every non-template VM, **running or stopped**, and every container. Stopped VMs 100/1001/2001 hold 12 GiB | free the reservation (delete unused stopped VMs, or convert them to templates), or set `ProxmoxCluster.spec.schedulerHints.memoryAdjustment` > 100 (overcommit); keep the MD max-size within what the host can actually run |
| MD rollout never finishes and CA never scales down: CA logs `Node … will be skipped as it is currently under rollout`, new ProxmoxMachine `cannot reserve …B of memory` | deadlock. Default rollout surge (`maxSurge 1 / maxUnavailable 0`) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (`ScaleDownNodeUpgradeProcessor`) | `spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1}` on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory |
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
| `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` |
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maxPods (`maximum-pods` is not a CA key and is silently ignored) |
| a pod with `nodeSelector: kubernetes.io/os: linux` doesn't trigger scale-from-zero | CA 1.35.2 builds the template node's os/arch labels from the infra template's `status.nodeInfo`. CAPMOX has none, so they become `""`. MD template labels aren't read either (CA reads the non-existent v1beta2 path `spec.template.spec.metadata.labels`) | `capacity.cluster-autoscaler.kubernetes.io/labels: "kubernetes.io/os=linux,kubernetes.io/arch=amd64,…"` on the MD |
| scale-up goes into one group only: `No similar node groups found`, next loop the second group (6 + 2 instead of 4 + 4) | the generic comparator fails when one group's template comes from a real node (cached for `--node-info-cache-expire-time`, default 10 years: ~1.78 Gi allocatable, os/arch set) and the other from the synthetic from-zero template (2 Gi, os/arch empty). The mismatch reason is only logged at V(5). `cluster.x-k8s.io/deployment-name` is **not** the cause, because CA never reads it | same label on both MDs (`node.cluster.x-k8s.io/pool`: template label + capacity labels annotation) and `balancing-label: node.cluster.x-k8s.io/pool` in the CA values. This turns off the resource comparison, so keep the MDs' capacity identical |
| CA loop takes 8–26 s; the log is full of `Waited before sending request … client-side throttling` on `/api`, `/apis`, `/apis/infrastructure.cluster.x-k8s.io/v1alpha2` | the CA 1.35.2 clusterapi provider runs **uncached** discovery (`getKindPreferredVersion`) for every MD at 0 replicas, on every `NodeGroups()` call (about 15× per loop). The client-go default is 5 QPS / burst 10. There is no upstream fix up to 1.36.1 | fewer loops: `scan-interval: 1m` (scale-up is still triggered by pending pods at once). Alternatively `kube-client-qps`/`-burst` (more requests, not chosen). `v: 2` hides the noise. Upstream fix: [future tasks](future-tasks.md) |
| `W clusterstate.go:510 Failed to find acceptable ranges for MachineDeployment/…` | the MD was created during a loop: acceptable ranges are snapshotted at the start of the loop, and the group showed up in a later `NodeGroups()` call | none if it happens once. If it repeats every loop, the group is dropping out of `NodeGroups()` (scale-from-zero check failing: infra template unreadable, capacity annotations missing) |
| ProxmoxMachine `VMProvisionFailed` … `can't clone VM to node '<node>' (VM uses local storage)` | the scheduled node (`allowedNodes`) isn't the node holding the template, and the template is on non-shared storage. A cross-node clone is then impossible, whatever `full`/`storage` are set to | per-node template copy + per-node MD with `allowedNodes: [<node>]` (see Step 0); or shared storage |
| VMs on a new PVE node get no IP (no DHCP/IPAM connectivity on VLAN 4) | `vmbr0` on that node isn't VLAN-aware, or its switch port doesn't trunk VLAN 4 | `bridge-vlan-aware yes` + `bridge-vids` in `/etc/network/interfaces`, `ifreload -a`; check via `pvesh get /nodes/<node>/network/vmbr0` |
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
| Provider CR not `READY` / `capi` play keeps retrying | operator preflight or component download failed | `kubectl -n capi-operator-system logs deploy/cluster-api-operator`, `kubectl -n <ns> describe <kind> <name>` |
| Provider keeps broken components after a failed fetch | stale per-version cache (cluster-api-operator #1075) | delete that provider's `…-cache` Secret, re-run `--tags capi` |
| CAPMOX still uses the old token after editing `secrets.yml` | operator doesn't watch `configSecret` | new Secret name in the template + `configSecret` reference |
| `ansible-playbook mgmt.yml` reports `changed` on a built cluster | a task isn't idempotent | `-v` for command output; on the VM `k3s kubectl diff -f /opt/lab-mgmt/capi-providers.yaml` |

```bash
kubectl -n capmox-system logs deploy/capmox-controller-manager -f
kubectl -n capi-system   logs deploy/capi-controller-manager -f
ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100'
```