# Session: CAPMOX memory accounting and overcommit on pv1-1

**Created:** 2026-09-28 10:07 CEST

## Question

> can we configure capmox so that it works with real available memory instead of calculation what
> stopped vms would use if somebody would start them? also can we configure memory overprovisioning
> on proxmox node?

Context: during Step 7, new workers failed with `cannot reserve …B of memory on node pv1-1`. The
host had about 12.8 GiB actually available, but CAPMOX counted 168 MB. Later this deadlocked the
2 GiB worker rollout (see the phase-1 troubleshooting table).

## What was checked (2026-09-28)

- CAPMOX source for v0.9.1 (the latest tag) and `main`:
  - `pkg/proxmox/goproxmox/api_client.go` `GetReservableMemoryBytes`;
  - `internal/service/scheduler/vmscheduler.go`;
  - `api/v1alpha2/proxmoxcluster_types.go` `SchedulerHints`;
  - `api/v1alpha2/proxmoxmachine_types.go`;
  - `internal/webhook/proxmoxcluster_webhook.go`.
- `pv1-1` (read-only):
  - `qm list`, node memory status;
  - KSM: `ksmtuned` active, `/sys/kernel/mm/ksm/run` = 0, `pages_sharing` = 0;
  - 8 GiB swap;
  - VM 103 has no `balloon:` line.

## Findings

- **No "real available memory" mode in any CAPMOX release.** `main` is identical to v0.9.1. The
  reservable memory is `node total × memoryAdjustment/100` minus the configured `maxmem` of every
  non-template VM (running or stopped) and every container. That's deliberate: real usage is
  racy, because VMs grow into their configured memory later. Using free memory would need an
  upstream change.
- **The only knob is `ProxmoxCluster.spec.schedulerHints.memoryAdjustment`** (percent, default 100):
  - `> 100` allows bounded overcommit;
  - `0` turns the memory check off completely (it then only checks request ≤ host total).
  - The ProxmoxCluster update webhook only validates the control-plane endpoint, so the hint can
    be changed on a live cluster. The scheduler reads it whenever it places a VM, and the stuck
    ProxmoxMachine retries.
- **Sizing for `lab1`:** everything except the workers is 21 GiB configured (VMs 100, 102, 201,
  1001, 2001, 160549). 5 workers × 2 GiB brings that to 31 GiB, and the host has 29.16 GiB, so
  `memoryAdjustment ≥ 107`. At 120 there's 35 GiB reservable, which fits max 5 workers with about
  4 GiB to spare.
- **Proxmox itself enforces no memory overcommit limit.** VMs can be started past physical RAM; CAPMOX
  was the only admission control. On the PVE side, "overprovisioning" is about how the host copes:
  - **KSM:** `ksmtuned` is active but only merges below 20% free (`KSM_THRES_COEF`). Identical
    Ubuntu clones deduplicate well. It's currently idle because the host has about 44% free.
  - **Host swap (8 GiB):** a safety net, but slow.
  - **Ballooning** (`balloon: <min>` below `memory`, auto-balloon above 80% host use): not
    exposed by CAPMOX v0.9.1 (`ProxmoxMachineSpec` has no balloon field). It's also a poor fit
    for Kubernetes nodes: the guest keeps reporting full capacity while the balloon takes memory
    away, so the kubelet schedules against memory that isn't there.

## Open questions

- ~~120 or 0?~~ The owner chose **160** (decision #17): 46.7 GiB reservable, about 17.7 GiB headroom with today's VMs.
- Is an upstream feature request worth filing (a free-memory mode, or excluding stopped VMs)?

## Follow-ups

- Done: `schedulerHints.memoryAdjustment: 160` on `ProxmoxCluster/lab1` in `configs/capi/lab1.yaml`,
  decisions #17. Watch real host memory and KSM `pages_sharing` during scale tests.