cluster-autoscaler-with-capi/docs/plans-executions/2026-09-29-2315-pv1-2-workers.md

# Execution log: lab1 workers on both PVE nodes (template copy per node)

Pairs with [docs/plans/2026-09-29-2315-pv1-2-workers.md](../plans/2026-09-29-2315-pv1-2-workers.md).

## Status

- [x] Step 0 — Record the plan
- [x] Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner)
- [x] Step 2 — Template 9101 on `pv1-2` (owner)
- [x] Step 3 — Second worker MachineDeployment `lab1-workers-pv1-2`
- [x] Step 4 — Scale test across both nodes
- [x] Step 5 — Docs and history

## Step 0 — Record the plan (2026-09-29 23:15 CEST)

Plan and this log committed on their own, together with the
[session note](../sessions/2026-09-29-2311-pve-two-node-cluster.md) that motivated them.

Starting state (read-only PVE API):
- cluster `homelab`: `pv1-1` 192.168.0.8, `pv1-2` 192.168.0.10
- `vmbr0` on `pv1-2` has no `bridge_vlan_aware`
- template 101 is only on `pv1-1`, with disks `scsi0` + `efidisk0`, both on `local-lvm`
- `nextid` is 103

## Steps 1–2 — bridge and template on `pv1-2` (owner, verified 2026-09-30 08:38 CEST)

The owner made `vmbr0` VLAN-aware on `pv1-2` (`/etc/network/interfaces`, then `ifreload -a`):

```
bridge-vlan-aware yes
bridge-vids 2-10
```

This deviates from the plan: `bridge-vids` is `2-10`, not `2-4094`. That's enough for VLAN 4, and
any VLAN above 10 on `pv1-2` would need the range widened first.

Template copy, run on `pv1-1`. A cross-node clone is refused from `local-lvm`, so the copy is a
clone on the same node followed by an offline migration:

```bash
qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm
qm migrate 9101 pv1-2 --with-local-disks
ssh root@192.168.0.10 qm template 9101
```

The PVE API shows 9101 on `pv1-2` with `template: 1`, and `scsi0`/`efidisk0` as
`local-lvm:base-9101-disk-*`.

Smoke test ([image-builder-proxmox.md](../image-builder-proxmox.md#smoke-test-of-the-template)) on
`pv1-2`, with `TPL=9101 ID=9901`: the VM got a VLAN 4 DHCP address through the guest agent, and was
destroyed afterwards. Before running it:
- `/root/jan.pub` had to be copied to `pv1-2` first: `scp ~/.ssh/id_ed25519.pub root@192.168.0.10:/root/jan.pub`.
- The doc's `ID=9001` now collides with `debian-13-template`, and the snippet ends with
  `qm destroy $ID`. The snippet now uses 9901 and checks that the ID is free (`a9f5879`).

Worth noting:
- VMIDs are cluster-wide (pmxcfs), which is why the copy needs its own ID. Names don't have to be
  unique.
- The copied `efidisk0` reports `size=528K` (the source says 4M). This doesn't matter: the template
  boots SeaBIOS.

## Step 3 — second worker MachineDeployment (prepared 2026-09-30, apply pending)

Changes in `configs/capi/lab1.yaml`:
- new `MachineDeployment/lab1-workers-pv1-2` and `ProxmoxMachineTemplate/lab1-worker-2g-pv1-2`
  (`sourceNode: pv1-2`, `templateID: 9101`, `allowedNodes: [pv1-2]`)
- `ProxmoxCluster.allowedNodes` stays `[pv1-1]` for the control plane
- header hand edits #10 and #11

Deviation from the plan: the IP pool is enlarged and the max sizes are raised (owner's request,
"20 nodes"):
- `ipv4Config.addresses` `192.168.4.31-192.168.4.39` → `192.168.4.31-192.168.4.50`
- `lab1-workers` max 5 → **6**, `lab1-workers-pv1-2` max **12**
- 6 + 12 + 1 CP = 19 of 20 addresses. The spare covers the KCP rollout surge, and CAPMOX pools
  hold a freed address for 60 s (`AddressReuseGracePeriodSeconds`, `pkg/kubernetes/ipam/ipam.go:318`).
- An ARP probe from `lab-mgmt` found `.32–.60` silent; `.30` (VIP) and `.31` (CP) share the CP VM's
  MAC.
- The owner confirmed `.31–.50` is outside the VLAN 4 DHCP range.
- Changing the pool is safe: the ProxmoxCluster webhook only validates the addresses, and the
  controller copies them into the InClusterIPPool on every reconcile (`ipam.go:323-324`).

Sizing:
- `pv1-1`: 6 × 2 GiB within ~17.7 GiB of reservable headroom (160 %).
- `pv1-2`: 12 × 2 GiB of 65.4 GiB reservable (40.9 GiB physical); 48 vCPU on 12 cores, which is
  fine for the pause-container `inflate` tests but CPU-bound under real load.

`yq` parse check: 9 documents; both worker PMTs point at the right node, template and `allowedNodes`.

Applied by the owner (`kubectl diff` / `kubectl apply -f configs/capi/lab1.yaml`). The CA
discovered `lab1-workers (max 6)` and `lab1-workers-pv1-2 (max 12)`.

## Step 4 — scale test, first run (2026-09-30 ~06:45 UTC)

The `inflate` Deployment had 16 pods (1500m / 512Mi each, 2 per worker).
- **Result:** 8 workers Ready by 06:47Z.
  - `lab1-workers` 0→6: VMIDs 103–108 on `pv1-1`, IPs .34–.39.
  - `lab1-workers-pv1-2` 0→2: VMIDs 109–110 on `pv1-2`, IPs .32–.33.
  - No `VMProvisionFailed`. `status.proxmoxNode` and providerIDs are consistent. The pool had 9 of
    20 addresses in use.
- **Checked with:** the `cluster-inspector` agent (read-only). Root causes came from the source
  (`upstream-researcher`, CA tag `cluster-autoscaler-1.35.2`, CAPI v1.12.11, CAPMOX v0.9.1).

Two problems.

**1. Not balanced.** The log shows `Best option to resize: lab1-workers` → `No similar node groups
found` → `0->6 (max: 6)`. Only in the next loop did `lab1-workers-pv1-2` get `0->2`.
- The inspector guessed the `cluster.x-k8s.io/deployment-name` label. The source rules that out:
  CA never reads it.
- What the source does show in CA 1.35.2 `cloudprovider/clusterapi`:
  - `Labels()` reads `spec.template.spec.metadata.labels`, which doesn't exist in v1beta2
    (`clusterapi_unstructured.go:207`). MD template labels therefore never reach the from-zero
    template node.
  - CAPMOX `ProxmoxMachineTemplate` has no `status`, so `kubernetes.io/os` and `arch` become `""` on
    the template (`clusterapi_nodegroup.go:393-405`). A pod with `nodeSelector: kubernetes.io/os:
    linux` would not trigger a scale-up from zero.
  - The capacity key is `maxPods`; our `maximum-pods` was ignored, and 110 only applied because it
    is the default.
  - `MixedTemplateNodeInfoProvider` caches a template built from a real node for
    `--node-info-cache-expire-time` (default 87600h). A group that has had nodes keeps that
    template: ~1.78Gi allocatable, os/arch set, `node-role` label. A from-zero group gets the
    synthetic template: 2Gi, empty os/arch. The generic comparator rejects that pair.
  - That this cache caused this particular decision is inferred. The reason for a mismatch is only
    logged at V(5).
- **Fix (`lab1.yaml` hand edit #12):**
  - both MDs carry the template label `node.cluster.x-k8s.io/pool: workers`, which CAPI syncs to the
    Node;
  - the annotation `capacity.cluster-autoscaler.kubernetes.io/labels:
    node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64`;
  - `maximum-pods` → `maxPods`;
  - CA `balancing-label: node.cluster.x-k8s.io/pool`, which compares only that label and turns off
    the resource checks.

**2. Loop 8–26 s, throttled.**
- The log is full of `Waited before sending request … client-side throttling` on `/api`, `/apis`
  and `/apis/infrastructure.cluster.x-k8s.io/v1alpha2` against the lab-mgmt API.
- Cause:
  - For every MD at 0 replicas, `CanScaleFromZero` → `readInfrastructureReferenceResource` →
    `getKindPreferredVersion` runs through the provider's **uncached** discovery client. The cached
    one is built at `clusterapi_provider.go:195` but not passed to the controller.
  - That is 3 GETs per MD per `NodeGroups()` call, and `NodeGroups()` runs about 15× per loop.
  - The client uses QPS 5 / burst 10 (`--kube-client-qps` / `--kube-client-burst` defaults).
- There is no upstream fix up to 1.36.1 or master (2026-09-29).
- The owner rejected raising QPS (50/100) and wants fewer requests instead. Fix in
  `configs/cluster-autoscaler/values.yaml`:
  - `scan-interval: 1m` (was 10s). `loop/trigger.go:74-100`: unschedulable pods and finished
    scale-ups/downs still start a loop immediately. Only idle loops are spaced out, so scale-down
    takes up to about +1 min.
  - `v: 2` (chart default 4). The throttling and "discovered node group" lines are V(3)/V(4).
- Considered and not taken:
  - `min-size: 1`: no MD at 0 means no discovery, but two idle VMs.
  - `CAPI_VERSION=v1beta1`: v1beta1 refs carry `apiVersion`, so no discovery. But v1beta1 is
    deprecated and the path is untested with CAPMOX.
  - An upstream patch to pass the cached discovery client (a future task).

## Rename of the pv1-1 group (owner's request, 2026-09-30)

This deviates from the plan: the plan kept `lab1-workers`. The owner wants symmetric per-node
names now that node names show the MD name (`lab1-workers-zldvn-*` next to
`lab1-workers-pv1-2-nlv6s-*`).

Changes in `configs/capi/lab1.yaml` (hand edit #13):
- `MachineDeployment/lab1-workers` → `lab1-workers-pv1-1`.
- `ProxmoxMachineTemplate/lab1-worker-2g` → `lab1-worker-2g-pv1-1`, now with explicit
  `allowedNodes: [pv1-1]`.

A rename creates new objects. `kubectl apply` doesn't prune, so the owner deletes the old MD and PMT
once the old MD has scaled to 0. The timing is right because the CA was already scaling every worker
down (`inflate` at 0). The new MD comes up with 0 replicas: `replicas` is unset, and the CAPI webhook
defaults it to the autoscaler min-size annotation.

## Step 4 — re-test after the fixes (2026-09-30 07:23 UTC)

The owner applied the changes in this order:
1. `kubectl apply -f configs/capi/lab1.yaml`: new `lab1-workers-pv1-1` + `lab1-worker-2g-pv1-1`, the
   pool label, `maxPods`, os/arch.
2. Deleted the old `MachineDeployment/lab1-workers` and `ProxmoxMachineTemplate/lab1-worker-2g`.
3. `helm upgrade --install ca … --version 9.59.0 -f configs/cluster-autoscaler/values.yaml`:
   `scan-interval 1m`, `v 2`, `balancing-label`.
4. Scaled `inflate` 0 → 16.

The CA log (`--since=15m`):

```
orchestrator.go:752] Splitting scale-up between 2 similar node groups: {…/lab1-workers-pv1-2, …/lab1-workers-pv1-1}
orchestrator.go:263] Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}]
```

- **Placement:** ProxmoxMachines `lab1-workers-pv1-1-*` on `pv1-1` (VMIDs 103, 104, 107, 110) and
  `lab1-workers-pv1-2-*` on `pv1-2` (105, 106, 108, 109).
- **Nodes:** all 8 Ready about 1–1.5 min after creation, with labels
  `node.cluster.x-k8s.io/pool=workers` and `kubernetes.io/os=linux`.
- **Loop timing:** idle loops now run on the `1m0s timer`.
- **One transient warning:** `W clusterstate.go:510 Failed to find acceptable ranges for
  MachineDeployment/default/lab1-workers-pv1-1`, once, in the loop during which the new MD was
  created. Acceptable ranges are snapshotted at the start of the loop, and the group appeared in a
  later `NodeGroups()` call. Harmless; it did not repeat.

Worth noting:
- `ProxmoxMachineTemplate/lab1-worker` (4 GiB, decision #15) is still in the cluster, and no MD
  references it. Deleting it is left to the owner.

## Step 5 — docs and history (2026-09-30 09:28 CEST)

- `docs/decisions.md`: #7 → `superseded by #18`. #18 (template copy + MD per PVE node, pool
  `.31–.50`, symmetric names) and #19 (CA `balancing-label`, `scan-interval 1m`, `v 2`, QPS left at
  the default) are both `active`, since the lab confirmed them in Step 4.
- `docs/phase-1-proxmox-capi.md`:
  - intro and topology (PVE cluster, templates, both MDs, pool);
  - Step 0: bridge on every node, per-node template;
  - Step 5: annotations incl. `maxPods`, `labels`, the pool label;
  - Step 6: the settings for two groups, and what a balanced scale-up logs;
  - troubleshooting: 7 rows (maxPods, empty os/arch, no balancing, throttling, the
    acceptable-ranges warning, cross-node clone, VLAN on a new node).
- `docs/image-builder-proxmox.md`: section "Template on other PVE nodes"; the result line names
  101/9101.
- `configs/capi/clusterctl.yaml.example`: comments on `TEMPLATE_VMID` / `ALLOWED_NODES`.
- `docs/future-tasks.md`: report the three CA clusterapi bugs upstream. The namespace task now uses
  the new MD names.
- `README.md` (lab inventory, key decision 7), `CLAUDE.md` / `AGENTS.md` (lab facts):
  PVE cluster, `pv1-2`, template 9101.
- `.claude/settings.json`: two inspector allowlist rules `md lab1-workers` → `md lab1-workers-pv1-1`.
- `CHANGELOG.md`: entry added after the owner confirmed the setup works (2026-09-30 11:05 CEST). The owner's pending #16/#17 work went in first as its own commit (`5e5b6c5`).