lab1: 2 GiB workers, CAPMOX memory overcommit, CA install (Step 6), future-tasks
Owner's pending work, committed ahead of the pv1-2 change it underlies: - lab1.yaml: worker template lab1-worker-2g (2048 MiB, decision #16), MD rollout maxSurge 0 / maxUnavailable 1, ProxmoxCluster schedulerHints.memoryAdjustment 160 (#17) - cluster-autoscaler values: RBAC for proxmoxmachinetemplates, aggressive scale-down profile, Cilium startup-taint; runbook Steps 4-7 and troubleshooting rows - inflate test pods 512Mi; decisions #16/#17; session note on the memory overcommit - docs/future-tasks.md convention (CLAUDE.md, README) with the namespace-per-cluster task - AGENTS.md; allowlist entries in .claude/settings.json Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -12,7 +12,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
|
||||
| --- | --- | --- |
|
||||
| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
|
||||
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
|
||||
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 4 GB RAM |
|
||||
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 2 GB RAM |
|
||||
|
||||
## Checklist
|
||||
|
||||
@@ -107,7 +107,10 @@ only when a version changes, then redo these edits:
|
||||
(handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
|
||||
3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
|
||||
4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
|
||||
5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 4096`.
|
||||
5. **Worker size** — rename the worker template to `ProxmoxMachineTemplate/lab1-worker-2g` (and the MD
|
||||
`infrastructureRef`): `numSockets: 1`, `numCores: 4`, `memoryMiB: 2048`. To resize later, add a
|
||||
template with a new name and point the MD at it; CAPI then rolls the workers. Don't edit a
|
||||
template in place.
|
||||
6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.
|
||||
|
||||
```bash
|
||||
@@ -157,7 +160,7 @@ metadata:
|
||||
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
|
||||
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
|
||||
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores
|
||||
capacity.cluster-autoscaler.kubernetes.io/memory: "4Gi" # memoryMiB 4096
|
||||
capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048
|
||||
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
|
||||
capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
|
||||
```
|
||||
@@ -183,20 +186,39 @@ Install into the **same namespace as the `lab1-kubeconfig` secret** (here `defau
|
||||
[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):
|
||||
|
||||
```bash
|
||||
helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update
|
||||
helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \
|
||||
-f configs/cluster-autoscaler/values.yaml
|
||||
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
|
||||
helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler \
|
||||
--version 9.59.0 -n default -f configs/cluster-autoscaler/values.yaml
|
||||
```
|
||||
|
||||
The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5.
|
||||
The values add `rbac.additionalRules` to read `proxmoxmachinetemplates`. The chart's ClusterRole
|
||||
doesn't cover infrastructure templates, and without them the first main loop hangs (see
|
||||
troubleshooting). Health check: the log shows `Starting main loop` followed by regular
|
||||
`Autoscaler loop triggered` lines, and `cluster-autoscaler-status` on lab1 leaves `Initializing`.
|
||||
|
||||
CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`. Its log must
|
||||
show the node group; the line is logged at `v: 4` on every loop (`clusterapi_controller.go`,
|
||||
CA 1.35.2):
|
||||
|
||||
```bash
|
||||
kubectl logs deploy/ca-clusterapi-cluster-autoscaler -n default --kubeconfig ~/.kube/lab-mgmt.kubeconfig \
|
||||
| grep 'discovered node group'
|
||||
# discovered node group: MachineDeployment/default/lab1-workers (min: 0, max: 5, replicas: 1)
|
||||
```
|
||||
|
||||
The status ConfigMap and the leader-election lock live on the workload cluster, not on the management
|
||||
cluster: `kubectl get cm cluster-autoscaler-status -n kube-system --kubeconfig ~/.kube/lab1.kubeconfig`
|
||||
(chart arg `--namespace=kube-system`, because `clusterAPIConfigMapsNamespace` is empty).
|
||||
|
||||
## Step 7 — scale tests
|
||||
|
||||
Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 1Gi each →
|
||||
CPU-bound at 2 pods per 4-core / 4 GiB worker).
|
||||
Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 512Mi each →
|
||||
CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocatable).
|
||||
|
||||
- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
|
||||
- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear.
|
||||
- [ ] Scale-down: `--replicas=0` → after `scale-down-delay-after-add` (2m) + `scale-down-unneeded-time`
|
||||
(1m) Machines and VMs disappear. That's the aggressive lab profile in the CA values; per-MD overrides are
|
||||
`cluster.x-k8s.io/autoscaling-options-{scaledownunneededtime,scaledownutilizationthreshold,…}` annotations.
|
||||
- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
|
||||
- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.
|
||||
|
||||
@@ -207,8 +229,12 @@ CPU-bound at 2 pods per 4-core / 4 GiB worker).
|
||||
| Symptom | Cause | Fix |
|
||||
| --- | --- | --- |
|
||||
| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
|
||||
| CA runs and discovers the node group but never scales; log stops at `waiting for cache sync on infrastructure resource`, `proxmoxmachinetemplates … forbidden` every ~40 s; status ConfigMap stuck on `Initializing` | chart 9.59.0 ClusterRole has no rule for `infrastructure.cluster.x-k8s.io`; CA's first main loop waits for a template informer that can never sync | `rbac.additionalRules` with get/list/watch on `proxmoxmachinetemplates` in the CA values, `helm upgrade` |
|
||||
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
|
||||
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
|
||||
| scale-up adds one node too many (6 pods at 2 per node → 4 workers, second plan `3->4` about 70 s after `1->3`) | a new node registers with Cilium's `node.cilium.io/agent-not-ready:NoSchedule`; once registered it's no longer an "upcoming" template node, and CA sees pending pods that don't fit on it | `extraArgs.startup-taint: node.cilium.io/agent-not-ready` in the CA values (CA treats the node as still starting) |
|
||||
| CA scales the MD up, new Machines stay without a VM; ProxmoxMachine `VMProvisionFailed: cannot reserve 4294967296B of memory on node pv1-1: …B available memory left` although `free` shows GBs available | CAPMOX v0.9.1 scheduler (`GetReservableMemoryBytes`): node total × `schedulerHints.memoryAdjustment` (default 100%) minus the **configured** memory of every non-template VM, **running or stopped**, and every container. Stopped VMs 100/1001/2001 hold 12 GiB | free the reservation (delete unused stopped VMs, or convert them to templates), or set `ProxmoxCluster.spec.schedulerHints.memoryAdjustment` > 100 (overcommit); keep the MD max-size within what the host can actually run |
|
||||
| MD rollout never finishes and CA never scales down: CA logs `Node … will be skipped as it is currently under rollout`, new ProxmoxMachine `cannot reserve …B of memory` | deadlock. Default rollout surge (`maxSurge 1 / maxUnavailable 0`) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (`ScaleDownNodeUpgradeProcessor`) | `spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1}` on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory |
|
||||
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
|
||||
| `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` |
|
||||
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
|
||||
|
||||
Reference in New Issue
Block a user