kacerrandClaude Opus 5.5
5e5b6c506a
Owner's pending work, committed ahead of the pv1-2 change it underlies: - lab1.yaml: worker template lab1-worker-2g (2048 MiB, decision #16), MD rollout maxSurge 0 / maxUnavailable 1, ProxmoxCluster schedulerHints.memoryAdjustment 160 (#17) - cluster-autoscaler values: RBAC for proxmoxmachinetemplates, aggressive scale-down profile, Cilium startup-taint; runbook Steps 4-7 and troubleshooting rows - inflate test pods 512Mi; decisions #16/#17; session note on the memory overcommit - docs/future-tasks.md convention (CLAUDE.md, README) with the namespace-per-cluster task - AGENTS.md; allowlist entries in .claude/settings.json Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
65 lines
3.5 KiB
Markdown
65 lines
3.5 KiB
Markdown
# Session: CAPMOX memory accounting and overcommit on pv1-1
|
||
|
||
**Created:** 2026-09-28 10:07 CEST
|
||
|
||
## Question
|
||
|
||
> can we configure capmox so that it works with real available memory instead of calculation what
|
||
> stopped vms would use if somebody would start them? also can we configure memory overprovisioning
|
||
> on proxmox node?
|
||
|
||
Context: during Step 7, new workers failed with `cannot reserve …B of memory on node pv1-1`. The
|
||
host had about 12.8 GiB actually available, but CAPMOX counted 168 MB. Later this deadlocked the
|
||
2 GiB worker rollout (see the phase-1 troubleshooting table).
|
||
|
||
## What was checked (2026-09-28)
|
||
|
||
- CAPMOX source for v0.9.1 (the latest tag) and `main`:
|
||
- `pkg/proxmox/goproxmox/api_client.go` `GetReservableMemoryBytes`;
|
||
- `internal/service/scheduler/vmscheduler.go`;
|
||
- `api/v1alpha2/proxmoxcluster_types.go` `SchedulerHints`;
|
||
- `api/v1alpha2/proxmoxmachine_types.go`;
|
||
- `internal/webhook/proxmoxcluster_webhook.go`.
|
||
- `pv1-1` (read-only):
|
||
- `qm list`, node memory status;
|
||
- KSM: `ksmtuned` active, `/sys/kernel/mm/ksm/run` = 0, `pages_sharing` = 0;
|
||
- 8 GiB swap;
|
||
- VM 103 has no `balloon:` line.
|
||
|
||
## Findings
|
||
|
||
- **No "real available memory" mode in any CAPMOX release.** `main` is identical to v0.9.1. The
|
||
reservable memory is `node total × memoryAdjustment/100` minus the configured `maxmem` of every
|
||
non-template VM (running or stopped) and every container. That's deliberate: real usage is
|
||
racy, because VMs grow into their configured memory later. Using free memory would need an
|
||
upstream change.
|
||
- **The only knob is `ProxmoxCluster.spec.schedulerHints.memoryAdjustment`** (percent, default 100):
|
||
- `> 100` allows bounded overcommit;
|
||
- `0` turns the memory check off completely (it then only checks request ≤ host total).
|
||
- The ProxmoxCluster update webhook only validates the control-plane endpoint, so the hint can
|
||
be changed on a live cluster. The scheduler reads it whenever it places a VM, and the stuck
|
||
ProxmoxMachine retries.
|
||
- **Sizing for `lab1`:** everything except the workers is 21 GiB configured (VMs 100, 102, 201,
|
||
1001, 2001, 160549). 5 workers × 2 GiB brings that to 31 GiB, and the host has 29.16 GiB, so
|
||
`memoryAdjustment ≥ 107`. At 120 there's 35 GiB reservable, which fits max 5 workers with about
|
||
4 GiB to spare.
|
||
- **Proxmox itself enforces no memory overcommit limit.** VMs can be started past physical RAM; CAPMOX
|
||
was the only admission control. On the PVE side, "overprovisioning" is about how the host copes:
|
||
- **KSM:** `ksmtuned` is active but only merges below 20% free (`KSM_THRES_COEF`). Identical
|
||
Ubuntu clones deduplicate well. It's currently idle because the host has about 44% free.
|
||
- **Host swap (8 GiB):** a safety net, but slow.
|
||
- **Ballooning** (`balloon: <min>` below `memory`, auto-balloon above 80% host use): not
|
||
exposed by CAPMOX v0.9.1 (`ProxmoxMachineSpec` has no balloon field). It's also a poor fit
|
||
for Kubernetes nodes: the guest keeps reporting full capacity while the balloon takes memory
|
||
away, so the kubelet schedules against memory that isn't there.
|
||
|
||
## Open questions
|
||
|
||
- ~~120 or 0?~~ The owner chose **160** (decision #17): 46.7 GiB reservable, about 17.7 GiB headroom with today's VMs.
|
||
- Is an upstream feature request worth filing (a free-memory mode, or excluding stopped VMs)?
|
||
|
||
## Follow-ups
|
||
|
||
- Done: `schedulerHints.memoryAdjustment: 160` on `ProxmoxCluster/lab1` in `configs/capi/lab1.yaml`,
|
||
decisions #17. Watch real host memory and KSM `pages_sharing` during scale tests.
|