diff --git a/.claude/settings.json b/.claude/settings.json index ee1515e..1c3d092 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -32,9 +32,9 @@ "Bash(awk -F/ '{print $3}')", "Bash(curl -fsSL https://kubernetes.github.io/autoscaler/index.yaml)", "Bash(awk '/^ cluster-autoscaler:/{f=1} f&&/appVersion:/{a=$2} f&&/^ version:/{print $2, a; n++} n>=6{exit}')", - "Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get md lab1-workers -n default -o json)", + "Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get md lab1-workers-pv1-1 -n default -o json)", "Bash(clusterctl --kubeconfig /Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig describe cluster lab1 -n default --show-conditions all)", - "Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get md lab1-workers -n default -o json --show-managed-fields)", + "Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get md lab1-workers-pv1-1 -n default -o json --show-managed-fields)", "Bash(export KUBECONFIG=$K)", "Bash(helm show *)", "Bash(grep -rniE 'backlog|future tasks|TODO' --include='*.md' . | head; sed -n '/## Key decisions/,/^## /p' README.md | head -5; grep -n 'docs/' README.md | head -20)", diff --git a/AGENTS.md b/AGENTS.md index 275bc26..cde1177 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,12 +10,15 @@ Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux ## Lab facts (don't re-derive) -- PVE node `pv1-1`, API `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0). +- PVE cluster `homelab`: nodes `pv1-1` (192.168.0.8) and `pv1-2` (192.168.0.10); API + `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0). VMIDs are cluster-wide. - VM network: `vmbr0` is VLAN-aware, VMs in **VLAN 4** → always `bridge=vmbr0` + tag/`vlan: 4`, never `vmbr0.4@vmbr0` (that is an `ip link` name, not a bridge). - Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2. - Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds). -- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55). Login only via cloud-init. +- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55), VMID 101 on `pv1-1` and copy + 9101 on `pv1-2` (local-lvm can't clone cross-node → one worker MD per PVE node). Login only via + cloud-init. - Management cluster: VM 201 `lab-mgmt`, 192.168.4.21 (`ssh ubuntu@`), built by the idempotent `ansible/mgmt.yml` (re-run must be `changed=0`); admin kubeconfig `~/.kube/lab-mgmt.kubeconfig`. - Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`, diff --git a/CHANGELOG.md b/CHANGELOG.md index ef46484..9907e4a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,33 @@ # Changelog +## 2026-09-30 11:05 CEST — lab1 workers on both PVE nodes, balanced autoscaling + +- **PVE cluster.** It is now the 2-node cluster `homelab`: `pv1-1` plus `pv1-2` (192.168.0.10). + - `vmbr0` on `pv1-2` is VLAN-aware. + - Template 101 is copied to `pv1-2` as 9101 (clone + offline migrate). `local-lvm` can't clone + across nodes. +- **lab1 workers.** One MachineDeployment per PVE node: `lab1-workers-pv1-1` (max 6) and + `lab1-workers-pv1-2` (max 12). + - Each is pinned by `allowedNodes`. + - The node IP pool is `.31–.50`. + - The control plane stays on `pv1-1`. + - The old `lab1-workers` was replaced to get symmetric names. +- **cluster-autoscaler.** Balances the two groups (a scale-up of 16 pods gave 4 + 4). + - Settings: `balancing-label: node.cluster.x-k8s.io/pool`, `scan-interval: 1m` and `v: 2`. + - The MDs got the pool label, `capacity…/labels` (os/arch) and `maxPods`. + - The request rate is left at the default. `scan-interval` cuts the discovery calls the + clusterapi provider makes without a cache. +- **Findings.** Three CA 1.35.2 clusterapi bugs: uncached discovery, labels read from a wrong + v1beta2 path, and empty os/arch labels for CAPMOX. They are worked around in config and logged + in `docs/future-tasks.md` for reporting upstream. +- **Files:** + - `configs/capi/lab1.yaml`, `configs/cluster-autoscaler/values.yaml`, + `configs/capi/clusterctl.yaml.example` + - `docs/phase-1-proxmox-capi.md`, `docs/image-builder-proxmox.md`, `docs/decisions.md` (#18, #19; + #7 superseded), `docs/future-tasks.md` + - plan, execution log and session note + - `README.md`, `CLAUDE.md`, `AGENTS.md` + ## 2026-09-27 10:40 CEST — Management cluster running, built by an idempotent Ansible playbook - Phase 1 Step 2 done: `ansible/mgmt.yml` creates VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1` diff --git a/CLAUDE.md b/CLAUDE.md index 5da2645..51fffa8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,12 +10,15 @@ Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux ## Lab facts (don't re-derive) -- PVE node `pv1-1`, API `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0). +- PVE cluster `homelab`: nodes `pv1-1` (192.168.0.8) and `pv1-2` (192.168.0.10); API + `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0). VMIDs are cluster-wide. - VM network: `vmbr0` is VLAN-aware, VMs in **VLAN 4** → always `bridge=vmbr0` + tag/`vlan: 4`, never `vmbr0.4@vmbr0` (that is an `ip link` name, not a bridge). - Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2. - Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds). -- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55). Login only via cloud-init. +- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55), VMID 101 on `pv1-1` and copy + 9101 on `pv1-2` (local-lvm can't clone cross-node → one worker MD per PVE node). Login only via + cloud-init. - Management cluster: VM 201 `lab-mgmt`, 192.168.4.21 (`ssh ubuntu@`), built by the idempotent `ansible/mgmt.yml` (re-run must be `changed=0`); admin kubeconfig `~/.kube/lab-mgmt.kubeconfig`. - Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`, diff --git a/README.md b/README.md index fae1799..141eb42 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ growing, in later phases, into a "poor man's Kubernetes-as-a-Service". | Hypervisor type | Management API | CAPI infrastructure provider | | --- | --- | --- | -| Proxmox VE (e.g. `pv1-1`, 192.168.0.8) | PVE REST API :8006 | CAPMOX (phase 1) | +| Proxmox VE cluster `homelab`: `pv1-1` 192.168.0.8, `pv1-2` 192.168.0.10 | PVE REST API :8006 | CAPMOX (phase 1) | | Ubuntu 24.04 + KVM/libvirt | libvirt RPC (`qemu+ssh`/`qemu+tls`) | none mature upstream → phase 2 | VM network: `vmbr0` (VLAN-aware), VLAN 4, build host 192.168.4.181. @@ -118,3 +118,6 @@ Short summary; the full log with alternatives and status is [docs/decisions.md]( `status.capacity` on `ProxmoxMachineTemplate`. 6. **Management cluster from one idempotent Ansible playbook**, CAPI providers installed by **cluster-api-operator** instead of `clusterctl init` — re-runs are no-ops, upgrades are version bumps. +7. **One template copy and one worker MachineDeployment per PVE node** (no shared storage): + `local-lvm` templates can't be cloned across nodes. cluster-autoscaler balances the groups on a + shared `node.cluster.x-k8s.io/pool` label. diff --git a/configs/capi/clusterctl.yaml.example b/configs/capi/clusterctl.yaml.example index 611d333..145548f 100644 --- a/configs/capi/clusterctl.yaml.example +++ b/configs/capi/clusterctl.yaml.example @@ -4,8 +4,9 @@ ## workload defaults PROXMOX_SOURCENODE: "pv1-1" # node holding the template -TEMPLATE_VMID: "" -ALLOWED_NODES: "[pv1-1]" # more nodes only with shared storage +TEMPLATE_VMID: "" # 101 on pv1-1 (copy 9101 on pv1-2) +ALLOWED_NODES: "[pv1-1]" # local-lvm: only the template's node. More PVE nodes = one + # worker MD + PMT per node by hand (lab1.yaml, decision #18) VM_SSH_KEYS: "ssh-ed25519 AAAA..." ## network (VLAN 4 on vmbr0 — add `vlan: 4` to the network device in ProxmoxMachineTemplates) diff --git a/configs/capi/lab1.yaml b/configs/capi/lab1.yaml index afab13b..79b3b71 100644 --- a/configs/capi/lab1.yaml +++ b/configs/capi/lab1.yaml @@ -20,6 +20,17 @@ # 7. MachineDeployment lab1-workers: autoscaler min/max + capacity annotations (runbook Step 5) # 8. MachineDeployment lab1-workers: rollout maxSurge 0 / maxUnavailable 1 (memory-tight host) # 9. ProxmoxCluster lab1: schedulerHints.memoryAdjustment 160 (CAPMOX memory overcommit, decision #17) +# 10. Second worker group on PVE node pv1-2 (decision #18): MachineDeployment lab1-workers-pv1-2 + +# ProxmoxMachineTemplate lab1-worker-2g-pv1-2 (template copy 9101, allowedNodes [pv1-2]). +# local-lvm is node-local, so every node needs its own template VMID and its own MD. +# 11. ProxmoxCluster lab1: node IP pool .31-.39 -> .31-.50 (20 addresses). Max sizes 6 (pv1-1) + +# 12 (pv1-2) + 1 CP = 19; one address stays free for the KCP rollout surge (CAPMOX pools also +# hold a freed address for 60 s before reuse). +# 12. Both worker MDs: label node.cluster.x-k8s.io/pool=workers (template + capacity labels +# annotation) for CA --balancing-label; capacity key maximum-pods -> maxPods. +# 13. Symmetric per-node names (owner's request): MachineDeployment lab1-workers -> lab1-workers-pv1-1, +# ProxmoxMachineTemplate lab1-worker-2g -> lab1-worker-2g-pv1-1 (+ explicit allowedNodes [pv1-1]). +# A rename is a new object: the old MD and PMT are deleted by hand once the old MD is at 0. apiVersion: cluster.x-k8s.io/v1beta2 kind: Cluster metadata: @@ -59,7 +70,7 @@ spec: - 1.1.1.1 ipv4Config: addresses: - - 192.168.4.31-192.168.4.39 + - 192.168.4.31-192.168.4.50 gateway: 192.168.4.8 prefix: 24 # CAPMOX reserves the configured memory of every non-template VM, stopped ones included, against @@ -235,15 +246,18 @@ kind: MachineDeployment metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0" - cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5" - # scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g + cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "6" + # scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g-pv1-1 capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets 1 × numCores 4 capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048 capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" # bootVolume sizeGb 40 - capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110" + capacity.cluster-autoscaler.kubernetes.io/maxPods: "110" # CA key; "maximum-pods" is ignored + # Labels for the from-zero template node. CA 1.35.2 reads MD labels from a path v1beta2 doesn't + # have, and CAPMOX publishes no nodeInfo, so without this os/arch are "" on the template. + capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64" labels: cluster.x-k8s.io/cluster-name: lab1 - name: lab1-workers + name: lab1-workers-pv1-1 namespace: default spec: clusterName: lab1 @@ -263,6 +277,8 @@ spec: labels: cluster.x-k8s.io/cluster-name: lab1 node-role.kubernetes.io/node: "" + # CA --balancing-label; node.cluster.x-k8s.io/* is synced to the Node by CAPI + node.cluster.x-k8s.io/pool: workers spec: bootstrap: configRef: @@ -273,17 +289,21 @@ spec: infrastructureRef: apiGroup: infrastructure.cluster.x-k8s.io kind: ProxmoxMachineTemplate - name: lab1-worker-2g + name: lab1-worker-2g-pv1-1 version: v1.35.9 --- apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2 kind: ProxmoxMachineTemplate metadata: - name: lab1-worker-2g + name: lab1-worker-2g-pv1-1 namespace: default spec: template: spec: + # Pinned like the pv1-2 template, independent of ProxmoxCluster.allowedNodes: template 101 + # lives on pv1-1's local-lvm. + allowedNodes: + - pv1-1 disks: bootVolume: disk: scsi0 @@ -301,6 +321,85 @@ spec: sourceNode: pv1-1 templateID: 101 --- +# Worker group on pv1-2. Keep in sync with lab1-workers-pv1-1 / lab1-worker-2g-pv1-1 except node, +# template and max-size: CA --balancing-label skips resource checks, so capacity must stay identical. +apiVersion: cluster.x-k8s.io/v1beta2 +kind: MachineDeployment +metadata: + annotations: + cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0" + cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12" + # scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g-pv1-2 + capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets 1 × numCores 4 + capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048 + capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" # bootVolume sizeGb 40 + capacity.cluster-autoscaler.kubernetes.io/maxPods: "110" # CA key; "maximum-pods" is ignored + # Labels for the from-zero template node. CA 1.35.2 reads MD labels from a path v1beta2 doesn't + # have, and CAPMOX publishes no nodeInfo, so without this os/arch are "" on the template. + capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64" + labels: + cluster.x-k8s.io/cluster-name: lab1 + name: lab1-workers-pv1-2 + namespace: default +spec: + clusterName: lab1 + rollout: + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + selector: + matchLabels: null + template: + metadata: + labels: + cluster.x-k8s.io/cluster-name: lab1 + node-role.kubernetes.io/node: "" + # CA --balancing-label; node.cluster.x-k8s.io/* is synced to the Node by CAPI + node.cluster.x-k8s.io/pool: workers + spec: + bootstrap: + configRef: + apiGroup: bootstrap.cluster.x-k8s.io + kind: KubeadmConfigTemplate + name: lab1-worker + clusterName: lab1 + infrastructureRef: + apiGroup: infrastructure.cluster.x-k8s.io + kind: ProxmoxMachineTemplate + name: lab1-worker-2g-pv1-2 + version: v1.35.9 +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2 +kind: ProxmoxMachineTemplate +metadata: + name: lab1-worker-2g-pv1-2 + namespace: default +spec: + template: + spec: + # Overrides ProxmoxCluster.allowedNodes ([pv1-1], kept for the control plane): template 9101 + # lives on pv1-2's local-lvm, and PVE refuses cross-node clones from local storage. + allowedNodes: + - pv1-2 + disks: + bootVolume: + disk: scsi0 + sizeGb: 40 + full: true + memoryMiB: 2048 + network: + networkDevices: + - bridge: vmbr0 + model: virtio + name: net0 + vlan: 4 + numCores: 4 + numSockets: 1 + sourceNode: pv1-2 + templateID: 9101 +--- apiVersion: bootstrap.cluster.x-k8s.io/v1beta2 kind: KubeadmConfigTemplate metadata: diff --git a/configs/cluster-autoscaler/values.yaml b/configs/cluster-autoscaler/values.yaml index 9701734..f3f4d2e 100644 --- a/configs/cluster-autoscaler/values.yaml +++ b/configs/cluster-autoscaler/values.yaml @@ -18,7 +18,8 @@ rbac: verbs: [get, list, watch] extraArgs: expander: least-waste - # Aggressive lab scale-down profile (CA 1.35 defaults in brackets): idle worker gone in ~3 min. + # Aggressive lab scale-down profile (CA 1.35 defaults in brackets): idle worker gone in ~3-4 min + # (scan-interval 1m adds up to a minute to the 1m unneeded time). # Keep delay-after-add >= node startup (~2-3 min here) or bursts flap up/down. scale-down-unneeded-time: 1m # [10m] scale-down-delay-after-add: 2m # [10m] cluster-wide cooldown after any scale-up @@ -27,7 +28,15 @@ extraArgs: scale-down-utilization-threshold: "0.6" # [0.5] max-graceful-termination-sec: "60" # [600] drain wait per pod balance-similar-node-groups: true + # One worker MD per PVE node. Compare groups only on this label: the generic comparator fails when + # one group has a real-node template (cached, 2Gi VM = ~1.78Gi allocatable, os/arch set) and the + # other a from-zero one. Disables the resource checks, so keep the MDs' capacity identical. + balancing-label: node.cluster.x-k8s.io/pool + # Fewer API requests instead of a higher kube-client-qps: every MD at 0 replicas costs ~3 uncached + # discovery GETs per NodeGroups() call (~15x per loop). Unschedulable pods and scale-ups/downs still + # trigger a loop immediately; only idle loops (scale-down evaluation) are spaced out. + scan-interval: 1m # [10s] # Cilium taints new nodes NoSchedule until its agent is up. Without this, CA simulates the freshly # registered (still tainted) node, finds the pending pods don't fit and adds one node too many. startup-taint: node.cilium.io/agent-not-ready - v: 4 + v: 2 # [chart 4] 4 logs every throttled request and MD lookup diff --git a/docs/decisions.md b/docs/decisions.md index 6a1a35f..5011805 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -15,7 +15,7 @@ image-builder fixes log; their dates are when the decision was recorded, not nec | 4 | 2026-09-24 | image-builder overrides go in a `PACKER_VAR_FILES` JSON | `PACKER_FLAGS` | flags were silently lost when `PACKER_FLAGS` was re-exported ([fixes log #4](image-builder-proxmox.md#fixes-log)) | active | | 5 | 2026-09-24 | Cilium installed by Helm with `ipam.mode=kubernetes`, pod CIDR `10.244.0.0/16` | ClusterResourceSet CNI flavors; pod CIDR `192.168.0.0/16` | keeps the CNI version under our control; `192.168.0.0/16` overlaps the lab LAN | active | | 6 | 2026-09-24 | Scale-from-zero via `capacity.cluster-autoscaler.kubernetes.io/*` annotations on the MachineDeployment | infrastructure provider publishing `status.capacity` | CAPMOX does not publish `status.capacity` on `ProxmoxMachineTemplate` | active | -| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | active | +| 7 | 2026-09-24 | Template only on `pv1-1`, `ALLOWED_NODES=[pv1-1]` | template on every node; shared storage | `local-lvm` is node-local and there is no shared storage yet | superseded by #18 | | 8 | 2026-09-26 | Phase 2: spike CAPLV first; own provider if insufficient; KubeVirt as fallback ([phase-2](phase-2-kvm-provider.md#suggested-approach)) | CAPK + KubeVirt, Metal3, own provider from the start | CAPLV shares CAPI 1.12 with CAPMOX and fits plain libvirt hosts directly | proposed | | 9 | 2026-09-26 | Phases 2 and 3 are independent and can be done in either order | strict sequence | no technical dependency between the KVM provider and hosted control planes | active | | 10 | 2026-09-26 | Keep project history in the repo: plans, execution logs, session notes, decisions, prompts, reviews, changelog ([session](sessions/2026-09-26-2355-history-conventions.md)) | chat history only; raw transcripts committed | research repo, reasoning must outlive sessions; transcripts are noisy and may contain secrets | active | @@ -26,3 +26,5 @@ image-builder fixes log; their dates are when the decision was recorded, not nec | 15 | 2026-09-27 | `lab1` workers 4 vCPU / **4 GiB** (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker | 8 GiB workers with 2Gi inflate pods | owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape ([configs/capi/lab1.yaml](../configs/capi/lab1.yaml)) | superseded by #16 | | 16 | 2026-09-28 | `lab1` workers 4 vCPU / **2 GiB** (new `ProxmoxMachineTemplate/lab1-worker-2g`, rolled in via the MD); inflate pods 1500m / **512Mi**, still CPU-bound at 2 per worker | keep 4 GiB and delete stopped VMs; CAPMOX `memoryAdjustment` overcommit | CAPMOX reserves the configured memory of every VM, stopped ones included, so 4 GiB workers ran out after the first scale-up ([troubleshooting](phase-1-proxmox-capi.md#troubleshooting)); 2 GiB (~1.78 GiB allocatable) still leaves room for kubelet/containerd plus real pods, and max 5 workers now fits the host | proposed | | 17 | 2026-09-28 | CAPMOX memory overcommit on `lab1`: `ProxmoxCluster.spec.schedulerHints.memoryAdjustment: 160` (46.7 GiB reservable on the 29.16 GiB `pv1-1`); MD rollout `maxSurge 0 / maxUnavailable 1` | 100% (default); 120% (minimum sizing for max 5 × 2 GiB workers); 0 (memory check off); deleting stopped VMs | CAPMOX counts the configured memory of stopped VMs (12 GiB here), and no release can use real free memory; 160% keeps a bound, and host-side protection is KSM + swap, not ballooning ([session](sessions/2026-09-28-1007-capmox-memory-overcommit.md)) | proposed | +| 18 | 2026-09-30 | PVE is now a 2-node cluster (`pv1-1` + `pv1-2`), no shared storage: a **template copy per node** (101 on `pv1-1`, 9101 on `pv1-2`, copied by clone + offline migrate), **one worker MD per node** (`lab1-workers-pv1-1` max 6, `lab1-workers-pv1-2` max 12, PMT `allowedNodes: []`), control plane stays on `pv1-1`, node IP pool `.31–.50` | shared storage (NFS); a second image-builder run on `pv1-2`; `templateSelector` with the same tags on both nodes; keep `lab1-workers` and only add a second MD | PVE refuses cross-node clones from local storage and CAPMOX v0.9.1 always passes the scheduled node as target; `templateSelector` needs exactly one match cluster-wide; a copy is byte-identical; owner wants no shared storage for now and symmetric names ([plan](plans/2026-09-29-2315-pv1-2-workers.md), [session](sessions/2026-09-29-2311-pve-two-node-cluster.md)) | active | +| 19 | 2026-09-30 | cluster-autoscaler with two node groups: `balancing-label: node.cluster.x-k8s.io/pool` (label on both MDs + `capacity…/labels` annotation), `scan-interval: 1m`, `v: 2`; client QPS left at the default 5/10 | generic similarity comparator; `balancing-ignore-label` for `deployment-name`; `kube-client-qps 50 / burst 100`; `min-size 1` per group; `CAPI_VERSION=v1beta1` | the comparator fails between a cached real-node template and a from-zero template (6+2 instead of 4+4); CA 1.35.2 runs uncached discovery per zero-replica MD ~15× per loop, and the owner prefers fewer requests over a higher rate — pending pods still trigger loops immediately ([execution log](plans-executions/2026-09-29-2315-pv1-2-workers.md)) | active | diff --git a/docs/future-tasks.md b/docs/future-tasks.md index 1195a0d..3b45cfa 100644 --- a/docs/future-tasks.md +++ b/docs/future-tasks.md @@ -3,6 +3,40 @@ Deferred work, newest first. When a task is picked up it gets a plan in `docs/plans/`; then tick it here and link the plan. +## - [ ] Report three cluster-autoscaler clusterapi-provider bugs upstream + +**Added:** 2026-09-30 09:28 CEST · **Why deferred:** worked around in config (decision #19); +reporting is not blocking. + +**Problem.** Three bugs in CA 1.35.2 `cloudprovider/clusterapi`, all still on master and 1.36.1 as +of 2026-09-29, found in [the pv1-2 execution log](plans-executions/2026-09-29-2315-pv1-2-workers.md): +1. **Uncached discovery.** + - `clusterapi_provider.go:195` builds a `memory.NewMemCacheClient`, but passes the raw + discovery client to the controller (`:209`). + - `getKindPreferredVersion` therefore does 3 GETs per MD at 0 replicas on every `NodeGroups()` + call, about 15× per loop. The loop took 18 s at 5 QPS. +2. **`Labels()` reads the wrong path.** It reads `spec.template.spec.metadata.labels` + (`clusterapi_unstructured.go:207`), which doesn't exist in CAPI v1beta2, so MD template labels + never reach the from-zero template node. The unit tests use the same wrong path. +3. **Empty os/arch labels.** For an infra template without `status`, as with CAPMOX, + `InstanceSystemInfo()` returns an empty struct. The template node then gets + `kubernetes.io/os=""` and `arch=""`, which override the generic `linux`/`amd64`. + +**Options:** +- Issues only. +- Issues plus small PRs: pass the cached client, fix the path, skip empty nodeInfo fields. +- Carry a patched CA image: rejected, versions must stay upstream and pinned. + +**Cost.** About 1–2 h for issues with source references. PRs would add the CLA and a test for each. + +**Changes when done:** +- Link the issues/PRs in `docs/phase-1-proxmox-capi.md` troubleshooting. +- Once fixed in a 1.35.x patch release: bump CA in `docs/versions.md`, then revisit + `scan-interval` and the `capacity…/labels` annotation. + +**Run by the owner:** nothing on the lab. Filing on GitHub is the owner's call, since it goes out +under their account. + ## - [ ] Namespace per workload cluster (`lab1` out of `default`) **Added:** 2026-09-28 09:29 CEST · **Why deferred:** keep phase 1 moving; `default` works for now. @@ -36,7 +70,7 @@ next cluster. (`helm uninstall ca -n default` first if it was installed in `default`). - `ansible/group_vars/all.yml`: `inspector_workload_clusters` entry `{ name: lab1, namespace: lab1 }`. - `docs/phase-1-proxmox-capi.md`: Steps 3, 5 and 6 (namespace in commands and log expectations, - `MachineDeployment/lab1/lab1-workers`). + `MachineDeployment/lab1/lab1-workers-pv1-{1,2}`). - `.claude/agents/cluster-inspector.md`: the "Where things live" table says CA runs in `default`. - `docs/decisions.md`: new row (namespace per workload cluster). diff --git a/docs/image-builder-proxmox.md b/docs/image-builder-proxmox.md index 660f423..1faa20a 100644 --- a/docs/image-builder-proxmox.md +++ b/docs/image-builder-proxmox.md @@ -2,7 +2,8 @@ Result: Proxmox template `ubuntu-2404-kube-v1.35.9` on node `pv1-1` — Ubuntu 24.04.4, containerd, kubeadm/kubelet/kubectl v1.35.9, pre-pulled control-plane images, cloud-init, qemu-guest-agent. -Its VMID is `TEMPLATE_VMID` for CAPMOX. +Its VMID (101) is `TEMPLATE_VMID` for CAPMOX. A byte-identical copy, 9101, sits on `pv1-2` +([Template on other PVE nodes](#template-on-other-pve-nodes)). ## Build host prerequisites (Ubuntu 24.04) @@ -88,3 +89,28 @@ qm stop $ID && qm destroy $ID --purge ``` kubelet crash-looping before `kubeadm init/join` is expected. + +## Template on other PVE nodes + +Without shared storage, a template on `local-lvm` can only be cloned on its own node. That holds +for PVE and for CAPMOX: `can't clone VM to node '' (VM uses local storage)`. Each additional +PVE node therefore gets its own copy under its own VMID, since VMIDs are cluster-wide. + +Copy rather than rebuild: the copy is byte-identical, whereas a second Packer run would pick up +newer apt packages. The source must have no disk or CD-ROM on non-shared storage other than +`local-lvm`; check with `qm config 101`. + +```bash +# on the node holding the template (pv1-1); 9101 = copy for pv1-2 +qm clone 101 9101 --full 1 --name ubuntu-2404-kube-v1.35.9 --storage local-lvm # clone stays on pv1-1 +qm migrate 9101 pv1-2 --with-local-disks # offline: copies the LVs +ssh root@192.168.0.10 qm template 9101 +``` + +Then run the smoke test on the target node with `TPL=9101`. It also proves that VLAN 4 works on +that node's `vmbr0`. The copy is referenced by `templateID: 9101` + `sourceNode: pv1-2` in that +node's `ProxmoxMachineTemplate` +([phase 1, Step 0](phase-1-proxmox-capi.md#step-0--proxmox-prep)). + +A new template build (a new Kubernetes version) means building on `pv1-1` and then repeating the +copy for every other node, with a new VMID pair per version. diff --git a/docs/phase-1-proxmox-capi.md b/docs/phase-1-proxmox-capi.md index 73ca25f..26717ea 100644 --- a/docs/phase-1-proxmox-capi.md +++ b/docs/phase-1-proxmox-capi.md @@ -1,7 +1,8 @@ # Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler Goal: a k3s management cluster (built by an Ansible playbook) runs CAPI + CAPMOX and creates workload -cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero. +cluster `lab1` as Proxmox VMs on a two-node PVE cluster; cluster-autoscaler scales its workers 0–18 +in one node group per PVE node, balanced, including scale-from-zero. No custom code in this phase — only assembling upstream components. Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md). @@ -10,9 +11,11 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro | Role | Where | Size | | --- | --- | --- | +| PVE cluster `homelab` | `pv1-1` 192.168.0.8 (API endpoint for CAPMOX), `pv1-2` 192.168.0.10; `local-lvm` on each, no shared storage | 16 / 12 cores, 29.2 / 40.9 GiB | +| Node template | `ubuntu-2404-kube-v1.35.9`: VMID 101 on `pv1-1`, copy 9101 on `pv1-2` | 20 GB | | Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB | -| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM | -| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 2 GB RAM | +| `lab1` control plane | 1 VM on `pv1-1`, VIP `.30` via kube-vip | 2 vCPU, 4 GB RAM | +| `lab1` workers | MachineDeployments `lab1-workers-pv1-1` (min 0 / max 6) and `lab1-workers-pv1-2` (min 0 / max 12); node IP pool `.31–.50` | 4 vCPU, 2 GB RAM | ## Checklist @@ -39,11 +42,24 @@ pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser # Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`. -Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP -for the control plane endpoint in the same subnet. +The token and ACLs are cluster-wide in a PVE cluster, so a node that joins later needs nothing here. -Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node -requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template. +Network: `vmbr0` is VLAN-aware on **every** PVE node, and VMs live in VLAN 4. On a new node, add +`bridge-vlan-aware yes` + `bridge-vids 2-10` (or wider) to `iface vmbr0`, run `ifreload -a`, and +trunk VLAN 4 on its switch port. CAPMOX writes `bridge`/`vlan` into the VM without checking them. +Reserve a node IP range outside DHCP and one VIP for the control plane endpoint in the same subnet. + +Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. +- PVE refuses to clone a VM to another node from non-shared storage, and CAPMOX v0.9.1 always + passes the scheduled node as the clone target. +- `templateSelector` needs exactly one match in the whole PVE cluster, so it can't pick a + per-node copy. +- Without shared storage, therefore, every node gets its own template VMID (VMIDs are + cluster-wide) and its own worker MachineDeployment. Each MD's `ProxmoxMachineTemplate` sets + `sourceNode`, `templateID` and `allowedNodes: []`. +- `ProxmoxCluster.allowedNodes` stays `[pv1-1]` for the control plane. The copy procedure is in + [image-builder-proxmox.md](image-builder-proxmox.md#template-on-other-pve-nodes), and the + reasoning in [decision #18](decisions.md). ## Step 2 — management cluster @@ -153,32 +169,47 @@ kubectl -n kube-system get pods -o wide # cilium DaemonSet 2/ ## Step 5 — autoscaling annotations +Both worker MDs carry the same block. Only the name and max-size differ (6 on `pv1-1`, 12 on +`pv1-2`): + ```yaml metadata: - name: lab1-workers + name: lab1-workers-pv1-2 annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0" - cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5" + cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12" capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048 capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" - capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110" + capacity.cluster-autoscaler.kubernetes.io/maxPods: "110" # not "maximum-pods" (ignored) + capacity.cluster-autoscaler.kubernetes.io/labels: "node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64" +spec: + template: + metadata: + labels: + node.cluster.x-k8s.io/pool: workers # synced to the Node by CAPI; CA --balancing-label ``` -They are part of `MachineDeployment/lab1-workers` in [configs/capi/lab1.yaml](../configs/capi/lab1.yaml); -re-apply the file (the only diff is these annotations; `replicas` isn't in the manifest, so it -stays as is): +They are part of both MDs in [configs/capi/lab1.yaml](../configs/capi/lab1.yaml). Re-apply the file. +`replicas` isn't in the manifest, so it stays as it is, and a new MD starts at the min-size: ```bash export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig -kubectl diff -f configs/capi/lab1.yaml # expect only the six annotations on lab1-workers +kubectl diff -f configs/capi/lab1.yaml kubectl apply -f configs/capi/lab1.yaml ``` -Capacity annotations are required for scale-from-zero because CAPMOX does not publish -`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints` -annotations if pods target the group by selector/toleration. They have no effect until -cluster-autoscaler runs (Step 6). +Scale-from-zero needs capacity annotations because CAPMOX does not publish `status.capacity`, and +they must match the `ProxmoxMachineTemplate`. The `.../labels` annotation is needed as well, for two +reasons in CA 1.35.2: +- It reads MD template labels from `spec.template.spec.metadata.labels`, a path that doesn't exist in + CAPI v1beta2. +- CAPMOX publishes no `status.nodeInfo`, so the from-zero template node would otherwise get + `kubernetes.io/os=""` and `arch=""`. A pod with an `os: linux` selector then can't trigger a + scale-up. + +Add a `.../taints` annotation if pods target the group by toleration. None of this has any effect +until cluster-autoscaler runs (Step 6). ## Step 6 — cluster-autoscaler @@ -196,14 +227,22 @@ doesn't cover infrastructure templates, and without them the first main loop han troubleshooting). Health check: the log shows `Starting main loop` followed by regular `Autoscaler loop triggered` lines, and `cluster-autoscaler-status` on lab1 leaves `Initializing`. -CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`. Its log must -show the node group; the line is logged at `v: 4` on every loop (`clusterapi_controller.go`, -CA 1.35.2): +CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`. -```bash -kubectl logs deploy/ca-clusterapi-cluster-autoscaler -n default --kubeconfig ~/.kube/lab-mgmt.kubeconfig \ - | grep 'discovered node group' -# discovered node group: MachineDeployment/default/lab1-workers (min: 0, max: 5, replicas: 1) +**Settings in the values that matter with two node groups:** +- `balancing-label: node.cluster.x-k8s.io/pool`. CA compares groups only on this label, so a + scale-up is split across both PVE nodes. +- `scan-interval: 1m`. Unschedulable pods and finished scale-ups/downs still start a loop + immediately; only the idle scale-down evaluation runs less often. +- `v: 2`. + +**Checking the node groups:** `discovered node group` is only logged at `v: 4` +(`clusterapi_controller.go:800`). At `v: 2`, check the node groups in the status ConfigMap (below) +instead. A balanced scale-up logs: + +```text +Splitting scale-up between 2 similar node groups: {MachineDeployment/default/lab1-workers-pv1-2, MachineDeployment/default/lab1-workers-pv1-1} +Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}] ``` The status ConfigMap and the leader-election lock live on the workload cluster, not on the management @@ -237,7 +276,13 @@ CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocata | MD rollout never finishes and CA never scales down: CA logs `Node … will be skipped as it is currently under rollout`, new ProxmoxMachine `cannot reserve …B of memory` | deadlock. Default rollout surge (`maxSurge 1 / maxUnavailable 0`) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (`ScaleDownNodeUpgradeProcessor`) | `spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1}` on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory | | last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` | | `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` | -| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods | +| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maxPods (`maximum-pods` is not a CA key and is silently ignored) | +| a pod with `nodeSelector: kubernetes.io/os: linux` doesn't trigger scale-from-zero | CA 1.35.2 builds the template node's os/arch labels from the infra template's `status.nodeInfo`. CAPMOX has none, so they become `""`. MD template labels aren't read either (CA reads the non-existent v1beta2 path `spec.template.spec.metadata.labels`) | `capacity.cluster-autoscaler.kubernetes.io/labels: "kubernetes.io/os=linux,kubernetes.io/arch=amd64,…"` on the MD | +| scale-up goes into one group only: `No similar node groups found`, next loop the second group (6 + 2 instead of 4 + 4) | the generic comparator fails when one group's template comes from a real node (cached for `--node-info-cache-expire-time`, default 10 years: ~1.78 Gi allocatable, os/arch set) and the other from the synthetic from-zero template (2 Gi, os/arch empty). The mismatch reason is only logged at V(5). `cluster.x-k8s.io/deployment-name` is **not** the cause, because CA never reads it | same label on both MDs (`node.cluster.x-k8s.io/pool`: template label + capacity labels annotation) and `balancing-label: node.cluster.x-k8s.io/pool` in the CA values. This turns off the resource comparison, so keep the MDs' capacity identical | +| CA loop takes 8–26 s; the log is full of `Waited before sending request … client-side throttling` on `/api`, `/apis`, `/apis/infrastructure.cluster.x-k8s.io/v1alpha2` | the CA 1.35.2 clusterapi provider runs **uncached** discovery (`getKindPreferredVersion`) for every MD at 0 replicas, on every `NodeGroups()` call (about 15× per loop). The client-go default is 5 QPS / burst 10. There is no upstream fix up to 1.36.1 | fewer loops: `scan-interval: 1m` (scale-up is still triggered by pending pods at once). Alternatively `kube-client-qps`/`-burst` (more requests, not chosen). `v: 2` hides the noise. Upstream fix: [future tasks](future-tasks.md) | +| `W clusterstate.go:510 Failed to find acceptable ranges for MachineDeployment/…` | the MD was created during a loop: acceptable ranges are snapshotted at the start of the loop, and the group showed up in a later `NodeGroups()` call | none if it happens once. If it repeats every loop, the group is dropping out of `NodeGroups()` (scale-from-zero check failing: infra template unreadable, capacity annotations missing) | +| ProxmoxMachine `VMProvisionFailed` … `can't clone VM to node '' (VM uses local storage)` | the scheduled node (`allowedNodes`) isn't the node holding the template, and the template is on non-shared storage. A cross-node clone is then impossible, whatever `full`/`storage` are set to | per-node template copy + per-node MD with `allowedNodes: []` (see Step 0); or shared storage | +| VMs on a new PVE node get no IP (no DHCP/IPAM connectivity on VLAN 4) | `vmbr0` on that node isn't VLAN-aware, or its switch port doesn't trunk VLAN 4 | `bridge-vlan-aware yes` + `bridge-vids` in `/etc/network/interfaces`, `ifreload -a`; check via `pvesh get /nodes//network/vmbr0` | | clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` | | ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range | | `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` | diff --git a/docs/plans-executions/2026-09-29-2315-pv1-2-workers.md b/docs/plans-executions/2026-09-29-2315-pv1-2-workers.md index 52f89df..5dc4004 100644 --- a/docs/plans-executions/2026-09-29-2315-pv1-2-workers.md +++ b/docs/plans-executions/2026-09-29-2315-pv1-2-workers.md @@ -7,9 +7,9 @@ Pairs with [docs/plans/2026-09-29-2315-pv1-2-workers.md](../plans/2026-09-29-231 - [x] Step 0 — Record the plan - [x] Step 1 — `vmbr0` on `pv1-2` VLAN-aware (owner) - [x] Step 2 — Template 9101 on `pv1-2` (owner) -- [ ] Step 3 — Second worker MachineDeployment `lab1-workers-pv1-2` -- [ ] Step 4 — Scale test across both nodes -- [ ] Step 5 — Docs and history +- [x] Step 3 — Second worker MachineDeployment `lab1-workers-pv1-2` +- [x] Step 4 — Scale test across both nodes +- [x] Step 5 — Docs and history ## Step 0 — Record the plan (2026-09-29 23:15 CEST) @@ -58,3 +58,165 @@ Worth noting: unique. - The copied `efidisk0` reports `size=528K` (the source says 4M). This doesn't matter: the template boots SeaBIOS. + +## Step 3 — second worker MachineDeployment (prepared 2026-09-30, apply pending) + +Changes in `configs/capi/lab1.yaml`: +- new `MachineDeployment/lab1-workers-pv1-2` and `ProxmoxMachineTemplate/lab1-worker-2g-pv1-2` + (`sourceNode: pv1-2`, `templateID: 9101`, `allowedNodes: [pv1-2]`) +- `ProxmoxCluster.allowedNodes` stays `[pv1-1]` for the control plane +- header hand edits #10 and #11 + +Deviation from the plan: the IP pool is enlarged and the max sizes are raised (owner's request, +"20 nodes"): +- `ipv4Config.addresses` `192.168.4.31-192.168.4.39` → `192.168.4.31-192.168.4.50` +- `lab1-workers` max 5 → **6**, `lab1-workers-pv1-2` max **12** +- 6 + 12 + 1 CP = 19 of 20 addresses. The spare covers the KCP rollout surge, and CAPMOX pools + hold a freed address for 60 s (`AddressReuseGracePeriodSeconds`, `pkg/kubernetes/ipam/ipam.go:318`). +- An ARP probe from `lab-mgmt` found `.32–.60` silent; `.30` (VIP) and `.31` (CP) share the CP VM's + MAC. +- The owner confirmed `.31–.50` is outside the VLAN 4 DHCP range. +- Changing the pool is safe: the ProxmoxCluster webhook only validates the addresses, and the + controller copies them into the InClusterIPPool on every reconcile (`ipam.go:323-324`). + +Sizing: +- `pv1-1`: 6 × 2 GiB within ~17.7 GiB of reservable headroom (160 %). +- `pv1-2`: 12 × 2 GiB of 65.4 GiB reservable (40.9 GiB physical); 48 vCPU on 12 cores, which is + fine for the pause-container `inflate` tests but CPU-bound under real load. + +`yq` parse check: 9 documents; both worker PMTs point at the right node, template and `allowedNodes`. + +Applied by the owner (`kubectl diff` / `kubectl apply -f configs/capi/lab1.yaml`). The CA +discovered `lab1-workers (max 6)` and `lab1-workers-pv1-2 (max 12)`. + +## Step 4 — scale test, first run (2026-09-30 ~06:45 UTC) + +The `inflate` Deployment had 16 pods (1500m / 512Mi each, 2 per worker). +- **Result:** 8 workers Ready by 06:47Z. + - `lab1-workers` 0→6: VMIDs 103–108 on `pv1-1`, IPs .34–.39. + - `lab1-workers-pv1-2` 0→2: VMIDs 109–110 on `pv1-2`, IPs .32–.33. + - No `VMProvisionFailed`. `status.proxmoxNode` and providerIDs are consistent. The pool had 9 of + 20 addresses in use. +- **Checked with:** the `cluster-inspector` agent (read-only). Root causes came from the source + (`upstream-researcher`, CA tag `cluster-autoscaler-1.35.2`, CAPI v1.12.11, CAPMOX v0.9.1). + +Two problems. + +**1. Not balanced.** The log shows `Best option to resize: lab1-workers` → `No similar node groups +found` → `0->6 (max: 6)`. Only in the next loop did `lab1-workers-pv1-2` get `0->2`. +- The inspector guessed the `cluster.x-k8s.io/deployment-name` label. The source rules that out: + CA never reads it. +- What the source does show in CA 1.35.2 `cloudprovider/clusterapi`: + - `Labels()` reads `spec.template.spec.metadata.labels`, which doesn't exist in v1beta2 + (`clusterapi_unstructured.go:207`). MD template labels therefore never reach the from-zero + template node. + - CAPMOX `ProxmoxMachineTemplate` has no `status`, so `kubernetes.io/os` and `arch` become `""` on + the template (`clusterapi_nodegroup.go:393-405`). A pod with `nodeSelector: kubernetes.io/os: + linux` would not trigger a scale-up from zero. + - The capacity key is `maxPods`; our `maximum-pods` was ignored, and 110 only applied because it + is the default. + - `MixedTemplateNodeInfoProvider` caches a template built from a real node for + `--node-info-cache-expire-time` (default 87600h). A group that has had nodes keeps that + template: ~1.78Gi allocatable, os/arch set, `node-role` label. A from-zero group gets the + synthetic template: 2Gi, empty os/arch. The generic comparator rejects that pair. + - That this cache caused this particular decision is inferred. The reason for a mismatch is only + logged at V(5). +- **Fix (`lab1.yaml` hand edit #12):** + - both MDs carry the template label `node.cluster.x-k8s.io/pool: workers`, which CAPI syncs to the + Node; + - the annotation `capacity.cluster-autoscaler.kubernetes.io/labels: + node.cluster.x-k8s.io/pool=workers,kubernetes.io/os=linux,kubernetes.io/arch=amd64`; + - `maximum-pods` → `maxPods`; + - CA `balancing-label: node.cluster.x-k8s.io/pool`, which compares only that label and turns off + the resource checks. + +**2. Loop 8–26 s, throttled.** +- The log is full of `Waited before sending request … client-side throttling` on `/api`, `/apis` + and `/apis/infrastructure.cluster.x-k8s.io/v1alpha2` against the lab-mgmt API. +- Cause: + - For every MD at 0 replicas, `CanScaleFromZero` → `readInfrastructureReferenceResource` → + `getKindPreferredVersion` runs through the provider's **uncached** discovery client. The cached + one is built at `clusterapi_provider.go:195` but not passed to the controller. + - That is 3 GETs per MD per `NodeGroups()` call, and `NodeGroups()` runs about 15× per loop. + - The client uses QPS 5 / burst 10 (`--kube-client-qps` / `--kube-client-burst` defaults). +- There is no upstream fix up to 1.36.1 or master (2026-09-29). +- The owner rejected raising QPS (50/100) and wants fewer requests instead. Fix in + `configs/cluster-autoscaler/values.yaml`: + - `scan-interval: 1m` (was 10s). `loop/trigger.go:74-100`: unschedulable pods and finished + scale-ups/downs still start a loop immediately. Only idle loops are spaced out, so scale-down + takes up to about +1 min. + - `v: 2` (chart default 4). The throttling and "discovered node group" lines are V(3)/V(4). +- Considered and not taken: + - `min-size: 1`: no MD at 0 means no discovery, but two idle VMs. + - `CAPI_VERSION=v1beta1`: v1beta1 refs carry `apiVersion`, so no discovery. But v1beta1 is + deprecated and the path is untested with CAPMOX. + - An upstream patch to pass the cached discovery client (a future task). + +## Rename of the pv1-1 group (owner's request, 2026-09-30) + +This deviates from the plan: the plan kept `lab1-workers`. The owner wants symmetric per-node +names now that node names show the MD name (`lab1-workers-zldvn-*` next to +`lab1-workers-pv1-2-nlv6s-*`). + +Changes in `configs/capi/lab1.yaml` (hand edit #13): +- `MachineDeployment/lab1-workers` → `lab1-workers-pv1-1`. +- `ProxmoxMachineTemplate/lab1-worker-2g` → `lab1-worker-2g-pv1-1`, now with explicit + `allowedNodes: [pv1-1]`. + +A rename creates new objects. `kubectl apply` doesn't prune, so the owner deletes the old MD and PMT +once the old MD has scaled to 0. The timing is right because the CA was already scaling every worker +down (`inflate` at 0). The new MD comes up with 0 replicas: `replicas` is unset, and the CAPI webhook +defaults it to the autoscaler min-size annotation. + +## Step 4 — re-test after the fixes (2026-09-30 07:23 UTC) + +The owner applied the changes in this order: +1. `kubectl apply -f configs/capi/lab1.yaml`: new `lab1-workers-pv1-1` + `lab1-worker-2g-pv1-1`, the + pool label, `maxPods`, os/arch. +2. Deleted the old `MachineDeployment/lab1-workers` and `ProxmoxMachineTemplate/lab1-worker-2g`. +3. `helm upgrade --install ca … --version 9.59.0 -f configs/cluster-autoscaler/values.yaml`: + `scan-interval 1m`, `v 2`, `balancing-label`. +4. Scaled `inflate` 0 → 16. + +The CA log (`--since=15m`): + +``` +orchestrator.go:752] Splitting scale-up between 2 similar node groups: {…/lab1-workers-pv1-2, …/lab1-workers-pv1-1} +orchestrator.go:263] Final scale-up plan: [{…/lab1-workers-pv1-2 0->4 (max: 12)} {…/lab1-workers-pv1-1 0->4 (max: 6)}] +``` + +- **Placement:** ProxmoxMachines `lab1-workers-pv1-1-*` on `pv1-1` (VMIDs 103, 104, 107, 110) and + `lab1-workers-pv1-2-*` on `pv1-2` (105, 106, 108, 109). +- **Nodes:** all 8 Ready about 1–1.5 min after creation, with labels + `node.cluster.x-k8s.io/pool=workers` and `kubernetes.io/os=linux`. +- **Loop timing:** idle loops now run on the `1m0s timer`. +- **One transient warning:** `W clusterstate.go:510 Failed to find acceptable ranges for + MachineDeployment/default/lab1-workers-pv1-1`, once, in the loop during which the new MD was + created. Acceptable ranges are snapshotted at the start of the loop, and the group appeared in a + later `NodeGroups()` call. Harmless; it did not repeat. + +Worth noting: +- `ProxmoxMachineTemplate/lab1-worker` (4 GiB, decision #15) is still in the cluster, and no MD + references it. Deleting it is left to the owner. + +## Step 5 — docs and history (2026-09-30 09:28 CEST) + +- `docs/decisions.md`: #7 → `superseded by #18`. #18 (template copy + MD per PVE node, pool + `.31–.50`, symmetric names) and #19 (CA `balancing-label`, `scan-interval 1m`, `v 2`, QPS left at + the default) are both `active`, since the lab confirmed them in Step 4. +- `docs/phase-1-proxmox-capi.md`: + - intro and topology (PVE cluster, templates, both MDs, pool); + - Step 0: bridge on every node, per-node template; + - Step 5: annotations incl. `maxPods`, `labels`, the pool label; + - Step 6: the settings for two groups, and what a balanced scale-up logs; + - troubleshooting: 7 rows (maxPods, empty os/arch, no balancing, throttling, the + acceptable-ranges warning, cross-node clone, VLAN on a new node). +- `docs/image-builder-proxmox.md`: section "Template on other PVE nodes"; the result line names + 101/9101. +- `configs/capi/clusterctl.yaml.example`: comments on `TEMPLATE_VMID` / `ALLOWED_NODES`. +- `docs/future-tasks.md`: report the three CA clusterapi bugs upstream. The namespace task now uses + the new MD names. +- `README.md` (lab inventory, key decision 7), `CLAUDE.md` / `AGENTS.md` (lab facts): + PVE cluster, `pv1-2`, template 9101. +- `.claude/settings.json`: two inspector allowlist rules `md lab1-workers` → `md lab1-workers-pv1-1`. +- `CHANGELOG.md`: entry added after the owner confirmed the setup works (2026-09-30 11:05 CEST). The owner's pending #16/#17 work went in first as its own commit (`5e5b6c5`). diff --git a/docs/sessions/2026-09-29-2311-pve-two-node-cluster.md b/docs/sessions/2026-09-29-2311-pve-two-node-cluster.md index e83a5a9..8bfae54 100644 --- a/docs/sessions/2026-09-29-2311-pve-two-node-cluster.md +++ b/docs/sessions/2026-09-29-2311-pve-two-node-cluster.md @@ -87,3 +87,14 @@ Conclusion: [2026-09-29-2315-pv1-2-workers](../plans/2026-09-29-2315-pv1-2-workers.md). It covers the bridge fix, the template copy, the second MD, the CA limits and a decision row superseding #7. - Update the "Lab facts" in CLAUDE.md/AGENTS.md: PVE cluster `homelab`, `pv1-2` at 192.168.0.10. + +## Outcome (2026-09-30) + +Done as planned, and confirmed on the lab: +- template 9101 on `pv1-2`; +- MDs `lab1-workers-pv1-1` (max 6) + `lab1-workers-pv1-2` (max 12); +- a balanced scale-up of 4 + 4 across both PVE nodes. + +Two cluster-autoscaler findings from the scale test (no balancing, and throttling from uncached +discovery) led to decision #19. Details are in the +[execution log](../plans-executions/2026-09-29-2315-pv1-2-workers.md).