cluster-autoscaler-with-capi/docs/sessions/2026-09-28-1007-capmox-memory-overcommit.md

Session: CAPMOX memory accounting and overcommit on pv1-1

Created: 2026-09-28 10:07 CEST

Question

can we configure capmox so that it works with real available memory instead of calculation what stopped vms would use if somebody would start them? also can we configure memory overprovisioning on proxmox node?

Context: during Step 7, new workers failed with cannot reserve …B of memory on node pv1-1. The host had about 12.8 GiB actually available, but CAPMOX counted 168 MB. Later this deadlocked the 2 GiB worker rollout (see the phase-1 troubleshooting table).

What was checked (2026-09-28)

• CAPMOX source for v0.9.1 (the latest tag) and main:
  • pkg/proxmox/goproxmox/api_client.go GetReservableMemoryBytes;
  • internal/service/scheduler/vmscheduler.go;
  • api/v1alpha2/proxmoxcluster_types.go SchedulerHints;
  • api/v1alpha2/proxmoxmachine_types.go;
  • internal/webhook/proxmoxcluster_webhook.go.
• pv1-1 (read-only):
  • qm list, node memory status;
  • KSM: ksmtuned active, /sys/kernel/mm/ksm/run = 0, pages_sharing = 0;
  • 8 GiB swap;
  • VM 103 has no balloon: line.

Findings

• No "real available memory" mode in any CAPMOX release. main is identical to v0.9.1. The reservable memory is node total × memoryAdjustment/100 minus the configured maxmem of every non-template VM (running or stopped) and every container. That's deliberate: real usage is racy, because VMs grow into their configured memory later. Using free memory would need an upstream change.
• The only knob is ProxmoxCluster.spec.schedulerHints.memoryAdjustment (percent, default 100):
  • > 100 allows bounded overcommit;
  • 0 turns the memory check off completely (it then only checks request ≤ host total).
  • The ProxmoxCluster update webhook only validates the control-plane endpoint, so the hint can be changed on a live cluster. The scheduler reads it whenever it places a VM, and the stuck ProxmoxMachine retries.
• Sizing for lab1: everything except the workers is 21 GiB configured (VMs 100, 102, 201, 1001, 2001, 160549). 5 workers × 2 GiB brings that to 31 GiB, and the host has 29.16 GiB, so memoryAdjustment ≥ 107. At 120 there's 35 GiB reservable, which fits max 5 workers with about 4 GiB to spare.
• Proxmox itself enforces no memory overcommit limit. VMs can be started past physical RAM; CAPMOX was the only admission control. On the PVE side, "overprovisioning" is about how the host copes:
  • KSM: ksmtuned is active but only merges below 20% free (KSM_THRES_COEF). Identical Ubuntu clones deduplicate well. It's currently idle because the host has about 44% free.
  • Host swap (8 GiB): a safety net, but slow.
  • Ballooning (balloon: <min> below memory, auto-balloon above 80% host use): not exposed by CAPMOX v0.9.1 (ProxmoxMachineSpec has no balloon field). It's also a poor fit for Kubernetes nodes: the guest keeps reporting full capacity while the balloon takes memory away, so the kubelet schedules against memory that isn't there.

Open questions

• 120 or 0? The owner chose 160 (decision #17): 46.7 GiB reservable, about 17.7 GiB headroom with today's VMs.
• Is an upstream feature request worth filing (a free-memory mode, or excluding stopped VMs)?

Follow-ups

• Done: schedulerHints.memoryAdjustment: 160 on ProxmoxCluster/lab1 in configs/capi/lab1.yaml, decisions #17. Watch real host memory and KSM pages_sharing during scale tests.