lab1: 2 GiB workers, CAPMOX memory overcommit, CA install (Step 6), future-tasks
Owner's pending work, committed ahead of the pv1-2 change it underlies: - lab1.yaml: worker template lab1-worker-2g (2048 MiB, decision #16), MD rollout maxSurge 0 / maxUnavailable 1, ProxmoxCluster schedulerHints.memoryAdjustment 160 (#17) - cluster-autoscaler values: RBAC for proxmoxmachinetemplates, aggressive scale-down profile, Cilium startup-taint; runbook Steps 4-7 and troubleshooting rows - inflate test pods 512Mi; decisions #16/#17; session note on the memory overcommit - docs/future-tasks.md convention (CLAUDE.md, README) with the namespace-per-cluster task - AGENTS.md; allowlist entries in .claude/settings.json Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+3
-1
@@ -23,4 +23,6 @@ image-builder fixes log; their dates are when the decision was recorded, not nec
|
||||
| 12 | 2026-09-27 | Management cluster built by one idempotent Ansible playbook, `ansible/mgmt.yml`, run from the laptop ([plan](plans/2026-09-27-0058-mgmt-cluster-ansible.md)) | cloud-init + k3s auto-deploy manifests; bash scripts with guards; OpenTofu + Helm | owner's choice; idempotent modules, one tool; only `ansible.builtin`, charts through k3s's helm-controller so nothing extra on the VM | active |
|
||||
| 13 | 2026-09-27 | CAPI providers installed by cluster-api-operator 0.28.0 (Provider CRs), cert-manager v1.21.2 pre-installed | guarded `clusterctl init` | declarative; re-apply is a no-op; upgrade = `spec.version` bump; 0.28.0 built against CAPI 1.12.10. Consequence: no `clusterctl init/upgrade/move/delete` | active |
|
||||
| 14 | 2026-09-27 | Management VM created by the playbook from the pinned Ubuntu 24.04 cloud image (VM 201, 192.168.4.21) | hand-made VM; the image-builder node template | reproducible from the repo; the image-builder template is a kubeadm node image, not meant for k3s | active |
|
||||
| 15 | 2026-09-27 | `lab1` workers 4 vCPU / **4 GiB** (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker | 8 GiB workers with 2Gi inflate pods | owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape ([configs/capi/lab1.yaml](../configs/capi/lab1.yaml)) | active |
|
||||
| 15 | 2026-09-27 | `lab1` workers 4 vCPU / **4 GiB** (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker | 8 GiB workers with 2Gi inflate pods | owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape ([configs/capi/lab1.yaml](../configs/capi/lab1.yaml)) | superseded by #16 |
|
||||
| 16 | 2026-09-28 | `lab1` workers 4 vCPU / **2 GiB** (new `ProxmoxMachineTemplate/lab1-worker-2g`, rolled in via the MD); inflate pods 1500m / **512Mi**, still CPU-bound at 2 per worker | keep 4 GiB and delete stopped VMs; CAPMOX `memoryAdjustment` overcommit | CAPMOX reserves the configured memory of every VM, stopped ones included, so 4 GiB workers ran out after the first scale-up ([troubleshooting](phase-1-proxmox-capi.md#troubleshooting)); 2 GiB (~1.78 GiB allocatable) still leaves room for kubelet/containerd plus real pods, and max 5 workers now fits the host | proposed |
|
||||
| 17 | 2026-09-28 | CAPMOX memory overcommit on `lab1`: `ProxmoxCluster.spec.schedulerHints.memoryAdjustment: 160` (46.7 GiB reservable on the 29.16 GiB `pv1-1`); MD rollout `maxSurge 0 / maxUnavailable 1` | 100% (default); 120% (minimum sizing for max 5 × 2 GiB workers); 0 (memory check off); deleting stopped VMs | CAPMOX counts the configured memory of stopped VMs (12 GiB here), and no release can use real free memory; 160% keeps a bound, and host-side protection is KSM + swap, not ballooning ([session](sessions/2026-09-28-1007-capmox-memory-overcommit.md)) | proposed |
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Future tasks
|
||||
|
||||
Deferred work, newest first. When a task is picked up it gets a plan in `docs/plans/`; then tick
|
||||
it here and link the plan.
|
||||
|
||||
## - [ ] Namespace per workload cluster (`lab1` out of `default`)
|
||||
|
||||
**Added:** 2026-09-28 09:29 CEST · **Why deferred:** keep phase 1 moving; `default` works for now.
|
||||
|
||||
**Problem.** cluster-autoscaler has to run in the same namespace as the Cluster. The chart mounts
|
||||
`clusterAPIKubeconfigSecret` as a Secret volume, which only works within the pod's namespace, and
|
||||
`lab1-kubeconfig` lives in the Cluster's namespace. `lab1` is in `default`, so CA would be installed
|
||||
into `default` too. That mixes things in `default`, and with more clusters (phase 3) you'd get one
|
||||
CA per cluster sitting side by side in `default`.
|
||||
|
||||
**Options compared:**
|
||||
|
||||
| Option | Verdict |
|
||||
| --- | --- |
|
||||
| CA in `default` (current Step 6) | works; poor hygiene, doesn't scale to more clusters |
|
||||
| CA in its own namespace with a copy of the kubeconfig Secret | rejected: CAPI rewrites `lab1-kubeconfig` when it rotates the client cert, so the copy goes stale and CA silently loses the workload cluster |
|
||||
| **One namespace per cluster** (`lab1` holds the CAPI objects, `InClusterIPPool`, kubeconfig Secret and CA) | preferred: the usual CAPI multi-cluster layout, one CA per workload cluster, RBAC and cleanup per namespace, matches phase 3 |
|
||||
|
||||
**Cost.** Namespace is immutable on CAPI objects, so `lab1` has to be deleted and recreated. That
|
||||
means about 15 minutes of provisioning plus reinstalling Cilium. `clusterctl move` is not an option,
|
||||
since it's forbidden with cluster-api-operator. CAPMOX credentials come from the controller's
|
||||
global Secret, so a new namespace needs no extra credentials. The cost grows once workloads and
|
||||
autoscaler state exist on `lab1`, so do it before the Step 7 results matter, or when creating the
|
||||
next cluster.
|
||||
|
||||
**Changes:**
|
||||
|
||||
- `configs/capi/lab1.yaml`: a `Namespace lab1` object and `metadata.namespace: lab1` on every
|
||||
object, including `InClusterIPPool`.
|
||||
- `configs/cluster-autoscaler/values.yaml`: `autoDiscovery.namespace: lab1`; install with `-n lab1`
|
||||
(`helm uninstall ca -n default` first if it was installed in `default`).
|
||||
- `ansible/group_vars/all.yml`: `inspector_workload_clusters` entry `{ name: lab1, namespace: lab1 }`.
|
||||
- `docs/phase-1-proxmox-capi.md`: Steps 3, 5 and 6 (namespace in commands and log expectations,
|
||||
`MachineDeployment/lab1/lab1-workers`).
|
||||
- `.claude/agents/cluster-inspector.md`: the "Where things live" table says CA runs in `default`.
|
||||
- `docs/decisions.md`: new row (namespace per workload cluster).
|
||||
|
||||
**Run by the owner:** `kubectl delete -f configs/capi/lab1.yaml` (old namespace), wait until the
|
||||
VMs are gone, `kubectl apply` the new manifest, Cilium `helm upgrade --install`,
|
||||
`ansible-playbook mgmt.yml --tags inspector`, then the CA install into `lab1`.
|
||||
@@ -12,7 +12,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
|
||||
| --- | --- | --- |
|
||||
| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
|
||||
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
|
||||
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 4 GB RAM |
|
||||
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 2 GB RAM |
|
||||
|
||||
## Checklist
|
||||
|
||||
@@ -107,7 +107,10 @@ only when a version changes, then redo these edits:
|
||||
(handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
|
||||
3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
|
||||
4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
|
||||
5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 4096`.
|
||||
5. **Worker size** — rename the worker template to `ProxmoxMachineTemplate/lab1-worker-2g` (and the MD
|
||||
`infrastructureRef`): `numSockets: 1`, `numCores: 4`, `memoryMiB: 2048`. To resize later, add a
|
||||
template with a new name and point the MD at it; CAPI then rolls the workers. Don't edit a
|
||||
template in place.
|
||||
6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.
|
||||
|
||||
```bash
|
||||
@@ -157,7 +160,7 @@ metadata:
|
||||
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
|
||||
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
|
||||
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores
|
||||
capacity.cluster-autoscaler.kubernetes.io/memory: "4Gi" # memoryMiB 4096
|
||||
capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048
|
||||
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
|
||||
capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
|
||||
```
|
||||
@@ -183,20 +186,39 @@ Install into the **same namespace as the `lab1-kubeconfig` secret** (here `defau
|
||||
[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):
|
||||
|
||||
```bash
|
||||
helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update
|
||||
helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \
|
||||
-f configs/cluster-autoscaler/values.yaml
|
||||
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
|
||||
helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler \
|
||||
--version 9.59.0 -n default -f configs/cluster-autoscaler/values.yaml
|
||||
```
|
||||
|
||||
The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5.
|
||||
The values add `rbac.additionalRules` to read `proxmoxmachinetemplates`. The chart's ClusterRole
|
||||
doesn't cover infrastructure templates, and without them the first main loop hangs (see
|
||||
troubleshooting). Health check: the log shows `Starting main loop` followed by regular
|
||||
`Autoscaler loop triggered` lines, and `cluster-autoscaler-status` on lab1 leaves `Initializing`.
|
||||
|
||||
CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`. Its log must
|
||||
show the node group; the line is logged at `v: 4` on every loop (`clusterapi_controller.go`,
|
||||
CA 1.35.2):
|
||||
|
||||
```bash
|
||||
kubectl logs deploy/ca-clusterapi-cluster-autoscaler -n default --kubeconfig ~/.kube/lab-mgmt.kubeconfig \
|
||||
| grep 'discovered node group'
|
||||
# discovered node group: MachineDeployment/default/lab1-workers (min: 0, max: 5, replicas: 1)
|
||||
```
|
||||
|
||||
The status ConfigMap and the leader-election lock live on the workload cluster, not on the management
|
||||
cluster: `kubectl get cm cluster-autoscaler-status -n kube-system --kubeconfig ~/.kube/lab1.kubeconfig`
|
||||
(chart arg `--namespace=kube-system`, because `clusterAPIConfigMapsNamespace` is empty).
|
||||
|
||||
## Step 7 — scale tests
|
||||
|
||||
Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 1Gi each →
|
||||
CPU-bound at 2 pods per 4-core / 4 GiB worker).
|
||||
Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 512Mi each →
|
||||
CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocatable).
|
||||
|
||||
- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
|
||||
- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear.
|
||||
- [ ] Scale-down: `--replicas=0` → after `scale-down-delay-after-add` (2m) + `scale-down-unneeded-time`
|
||||
(1m) Machines and VMs disappear. That's the aggressive lab profile in the CA values; per-MD overrides are
|
||||
`cluster.x-k8s.io/autoscaling-options-{scaledownunneededtime,scaledownutilizationthreshold,…}` annotations.
|
||||
- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
|
||||
- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.
|
||||
|
||||
@@ -207,8 +229,12 @@ CPU-bound at 2 pods per 4-core / 4 GiB worker).
|
||||
| Symptom | Cause | Fix |
|
||||
| --- | --- | --- |
|
||||
| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
|
||||
| CA runs and discovers the node group but never scales; log stops at `waiting for cache sync on infrastructure resource`, `proxmoxmachinetemplates … forbidden` every ~40 s; status ConfigMap stuck on `Initializing` | chart 9.59.0 ClusterRole has no rule for `infrastructure.cluster.x-k8s.io`; CA's first main loop waits for a template informer that can never sync | `rbac.additionalRules` with get/list/watch on `proxmoxmachinetemplates` in the CA values, `helm upgrade` |
|
||||
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
|
||||
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
|
||||
| scale-up adds one node too many (6 pods at 2 per node → 4 workers, second plan `3->4` about 70 s after `1->3`) | a new node registers with Cilium's `node.cilium.io/agent-not-ready:NoSchedule`; once registered it's no longer an "upcoming" template node, and CA sees pending pods that don't fit on it | `extraArgs.startup-taint: node.cilium.io/agent-not-ready` in the CA values (CA treats the node as still starting) |
|
||||
| CA scales the MD up, new Machines stay without a VM; ProxmoxMachine `VMProvisionFailed: cannot reserve 4294967296B of memory on node pv1-1: …B available memory left` although `free` shows GBs available | CAPMOX v0.9.1 scheduler (`GetReservableMemoryBytes`): node total × `schedulerHints.memoryAdjustment` (default 100%) minus the **configured** memory of every non-template VM, **running or stopped**, and every container. Stopped VMs 100/1001/2001 hold 12 GiB | free the reservation (delete unused stopped VMs, or convert them to templates), or set `ProxmoxCluster.spec.schedulerHints.memoryAdjustment` > 100 (overcommit); keep the MD max-size within what the host can actually run |
|
||||
| MD rollout never finishes and CA never scales down: CA logs `Node … will be skipped as it is currently under rollout`, new ProxmoxMachine `cannot reserve …B of memory` | deadlock. Default rollout surge (`maxSurge 1 / maxUnavailable 0`) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (`ScaleDownNodeUpgradeProcessor`) | `spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1}` on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory |
|
||||
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
|
||||
| `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` |
|
||||
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Session: CAPMOX memory accounting and overcommit on pv1-1
|
||||
|
||||
**Created:** 2026-09-28 10:07 CEST
|
||||
|
||||
## Question
|
||||
|
||||
> can we configure capmox so that it works with real available memory instead of calculation what
|
||||
> stopped vms would use if somebody would start them? also can we configure memory overprovisioning
|
||||
> on proxmox node?
|
||||
|
||||
Context: during Step 7, new workers failed with `cannot reserve …B of memory on node pv1-1`. The
|
||||
host had about 12.8 GiB actually available, but CAPMOX counted 168 MB. Later this deadlocked the
|
||||
2 GiB worker rollout (see the phase-1 troubleshooting table).
|
||||
|
||||
## What was checked (2026-09-28)
|
||||
|
||||
- CAPMOX source for v0.9.1 (the latest tag) and `main`:
|
||||
- `pkg/proxmox/goproxmox/api_client.go` `GetReservableMemoryBytes`;
|
||||
- `internal/service/scheduler/vmscheduler.go`;
|
||||
- `api/v1alpha2/proxmoxcluster_types.go` `SchedulerHints`;
|
||||
- `api/v1alpha2/proxmoxmachine_types.go`;
|
||||
- `internal/webhook/proxmoxcluster_webhook.go`.
|
||||
- `pv1-1` (read-only):
|
||||
- `qm list`, node memory status;
|
||||
- KSM: `ksmtuned` active, `/sys/kernel/mm/ksm/run` = 0, `pages_sharing` = 0;
|
||||
- 8 GiB swap;
|
||||
- VM 103 has no `balloon:` line.
|
||||
|
||||
## Findings
|
||||
|
||||
- **No "real available memory" mode in any CAPMOX release.** `main` is identical to v0.9.1. The
|
||||
reservable memory is `node total × memoryAdjustment/100` minus the configured `maxmem` of every
|
||||
non-template VM (running or stopped) and every container. That's deliberate: real usage is
|
||||
racy, because VMs grow into their configured memory later. Using free memory would need an
|
||||
upstream change.
|
||||
- **The only knob is `ProxmoxCluster.spec.schedulerHints.memoryAdjustment`** (percent, default 100):
|
||||
- `> 100` allows bounded overcommit;
|
||||
- `0` turns the memory check off completely (it then only checks request ≤ host total).
|
||||
- The ProxmoxCluster update webhook only validates the control-plane endpoint, so the hint can
|
||||
be changed on a live cluster. The scheduler reads it whenever it places a VM, and the stuck
|
||||
ProxmoxMachine retries.
|
||||
- **Sizing for `lab1`:** everything except the workers is 21 GiB configured (VMs 100, 102, 201,
|
||||
1001, 2001, 160549). 5 workers × 2 GiB brings that to 31 GiB, and the host has 29.16 GiB, so
|
||||
`memoryAdjustment ≥ 107`. At 120 there's 35 GiB reservable, which fits max 5 workers with about
|
||||
4 GiB to spare.
|
||||
- **Proxmox itself enforces no memory overcommit limit.** VMs can be started past physical RAM; CAPMOX
|
||||
was the only admission control. On the PVE side, "overprovisioning" is about how the host copes:
|
||||
- **KSM:** `ksmtuned` is active but only merges below 20% free (`KSM_THRES_COEF`). Identical
|
||||
Ubuntu clones deduplicate well. It's currently idle because the host has about 44% free.
|
||||
- **Host swap (8 GiB):** a safety net, but slow.
|
||||
- **Ballooning** (`balloon: <min>` below `memory`, auto-balloon above 80% host use): not
|
||||
exposed by CAPMOX v0.9.1 (`ProxmoxMachineSpec` has no balloon field). It's also a poor fit
|
||||
for Kubernetes nodes: the guest keeps reporting full capacity while the balloon takes memory
|
||||
away, so the kubelet schedules against memory that isn't there.
|
||||
|
||||
## Open questions
|
||||
|
||||
- ~~120 or 0?~~ The owner chose **160** (decision #17): 46.7 GiB reservable, about 17.7 GiB headroom with today's VMs.
|
||||
- Is an upstream feature request worth filing (a free-memory mode, or excluding stopped VMs)?
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- Done: `schedulerHints.memoryAdjustment: 160` on `ProxmoxCluster/lab1` in `configs/capi/lab1.yaml`,
|
||||
decisions #17. Watch real host memory and KSM `pages_sharing` during scale tests.
|
||||
Reference in New Issue
Block a user