lab1: 2 GiB workers, CAPMOX memory overcommit, CA install (Step 6), future-tasks
Owner's pending work, committed ahead of the pv1-2 change it underlies: - lab1.yaml: worker template lab1-worker-2g (2048 MiB, decision #16), MD rollout maxSurge 0 / maxUnavailable 1, ProxmoxCluster schedulerHints.memoryAdjustment 160 (#17) - cluster-autoscaler values: RBAC for proxmoxmachinetemplates, aggressive scale-down profile, Cilium startup-taint; runbook Steps 4-7 and troubleshooting rows - inflate test pods 512Mi; decisions #16/#17; session note on the memory overcommit - docs/future-tasks.md convention (CLAUDE.md, README) with the namespace-per-cluster task - AGENTS.md; allowlist entries in .claude/settings.json Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+21
-1
@@ -26,7 +26,27 @@
|
||||
"Bash(ansible pve *)",
|
||||
"Read(//Users/jan.novak/**)",
|
||||
"Bash(export KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig)",
|
||||
"Bash(kubectl explain *)"
|
||||
"Bash(kubectl explain *)",
|
||||
"Bash(grep -v '^ *$')",
|
||||
"Bash(git ls-remote *)",
|
||||
"Bash(awk -F/ '{print $3}')",
|
||||
"Bash(curl -fsSL https://kubernetes.github.io/autoscaler/index.yaml)",
|
||||
"Bash(awk '/^ cluster-autoscaler:/{f=1} f&&/appVersion:/{a=$2} f&&/^ version:/{print $2, a; n++} n>=6{exit}')",
|
||||
"Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get md lab1-workers -n default -o json)",
|
||||
"Bash(clusterctl --kubeconfig /Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig describe cluster lab1 -n default --show-conditions all)",
|
||||
"Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get md lab1-workers -n default -o json --show-managed-fields)",
|
||||
"Bash(export KUBECONFIG=$K)",
|
||||
"Bash(helm show *)",
|
||||
"Bash(grep -rniE 'backlog|future tasks|TODO' --include='*.md' . | head; sed -n '/## Key decisions/,/^## /p' README.md | head -5; grep -n 'docs/' README.md | head -20)",
|
||||
"Bash(helm template *)",
|
||||
"Bash(sed -e 's|^\\\\*\\\\*Created:\\\\*\\\\* .*|**Created:** 2026-09-29 23:15 CEST|' /Users/jan.novak/.claude/plans/purrfect-frolicking-lovelace.md > docs/plans/2026-09-29-2315-pv1-2-workers.md && head -3 docs/plans/2026-09-29-2315-pv1-2-workers.md)",
|
||||
"WebSearch",
|
||||
"WebFetch(domain:github.com)",
|
||||
"Bash(git apply *)"
|
||||
],
|
||||
"additionalDirectories": [
|
||||
"/Users/jan.novak/srv/go/cluster-autoscaler-with-capi",
|
||||
"/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/docs/plans"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
# AGENTS.md
|
||||
|
||||
Home-lab research project: cluster-autoscaler + Cluster API on Proxmox VE (phase 1), then a
|
||||
KVM/libvirt provider (phase 2) and Kamaji hosted control planes / "poor man's KaaS" (phase 3).
|
||||
Start with README.md; details live in docs/.
|
||||
|
||||
## Owner context
|
||||
|
||||
Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux, distributed systems).
|
||||
|
||||
## Lab facts (don't re-derive)
|
||||
|
||||
- PVE node `pv1-1`, API `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0).
|
||||
- VM network: `vmbr0` is VLAN-aware, VMs in **VLAN 4** → always `bridge=vmbr0` + tag/`vlan: 4`,
|
||||
never `vmbr0.4@vmbr0` (that is an `ip link` name, not a bridge).
|
||||
- Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2.
|
||||
- Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds).
|
||||
- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55). Login only via cloud-init.
|
||||
- Management cluster: VM 201 `lab-mgmt`, 192.168.4.21 (`ssh ubuntu@`), built by the idempotent
|
||||
`ansible/mgmt.yml` (re-run must be `changed=0`); admin kubeconfig `~/.kube/lab-mgmt.kubeconfig`.
|
||||
- Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`,
|
||||
PVESDNUser on `/sdn/zones/localnetwork`.
|
||||
|
||||
## Version rules
|
||||
|
||||
- `docs/versions.md` is the source of truth. Any version change updates it in the same change.
|
||||
- Before proposing any deployment or upgrade, verify versions are current (use the
|
||||
`k8s-version-check` skill if available). Go module proxy / git ls-remote / Helm index.yaml work
|
||||
even when the GitHub API is unavailable.
|
||||
- CAPI core is pinned to **v1.12.x** because CAPMOX v0.9 is only tested on CAPI 1.11/1.12. This caps
|
||||
workload Kubernetes at 1.35 and cluster-autoscaler at 1.35.x. Don't bump CAPI unless the CAPMOX
|
||||
compatibility table allows it.
|
||||
- Always pin versions explicitly (management cluster: `ansible/group_vars/all.yml`; `helm install
|
||||
--version`); never rely on "latest".
|
||||
- CAPI providers are managed by cluster-api-operator (Provider CRs from `ansible/mgmt.yml`). Never
|
||||
run `clusterctl init/upgrade/move/delete` against the management cluster; `clusterctl` is only for
|
||||
`generate cluster` (always with `--infrastructure proxmox:v0.9.1`), `describe`, `get kubeconfig`.
|
||||
- Check a provider's `metadata.yaml` contract and CAPI version (go.mod) before adding it to the
|
||||
management cluster — all providers share one CAPI core (known skew: Kamaji CP provider v0.21.0 is
|
||||
on CAPI 1.13).
|
||||
|
||||
## Facts about providers: verify in source
|
||||
|
||||
READMEs lag. For "does provider X support Y" read the API types and templates of the exact tagged
|
||||
release (Go proxy zip: `https://proxy.golang.org/<module>/@v/<ver>.zip`). Delegate to the
|
||||
`upstream-researcher` agent for anything non-trivial.
|
||||
|
||||
## Safety
|
||||
|
||||
- Never commit secrets: PVE token secret, kubeconfigs, `clusterctl.yaml` (see .gitignore).
|
||||
- Cluster inspection is read-only via the `cluster-inspector` agent and read-only kubeconfigs
|
||||
(`.Codex/cluster-inspector/`, see its README). Mutations (`kubectl apply/delete/scale`, `qm`, `pveum`)
|
||||
are proposed as commands for the owner to run unless explicitly asked to execute them.
|
||||
- Don't delete Proxmox VMs/templates or CAPI objects on your own initiative.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Docs in English, Markdown, Mermaid for diagrams (Gitea renders it).
|
||||
- Troubleshooting findings go into the relevant doc's troubleshooting/fixes table:
|
||||
symptom → root cause → fix.
|
||||
- Example configs live in `configs/` as `*.example` or non-secret values files.
|
||||
|
||||
## Project history
|
||||
|
||||
Research repo: plans, conversations and decisions are kept in the repo, dated. Timestamps always
|
||||
come from `date` (`+%Y-%m-%d-%H%M` for file names, `+%Y-%m-%d %H:%M %Z` for headings), never guessed.
|
||||
Summaries only, never raw transcripts; mask secrets and check the staged diff before committing.
|
||||
|
||||
| What | Where | When |
|
||||
| --- | --- | --- |
|
||||
| Plans | `docs/plans/YYYY-MM-DD-HHMM-<slug>.md` | every plan-mode plan |
|
||||
| Execution logs | `docs/plans-executions/<plan's file name>` | after each step of an approved plan |
|
||||
| Session notes | `docs/sessions/YYYY-MM-DD-HHMM-<slug>.md` | research/design/debugging conversations without a plan |
|
||||
| Decisions | `docs/decisions.md` (one table) | a direction chosen, rejected or reversed |
|
||||
| Prompts | `docs/prompts/YYYY-MM-DD-HHMM-<slug>.md` | long crafted prompts/specs that kick off work (verbatim) |
|
||||
| Reviews | `docs/reviews/YYYY-MM-DD-HHMM-<slug>.md` | review findings, most severe first, `- [ ]` ticked as processed |
|
||||
| Changelog | `CHANGELOG.md`, newest on top | lab milestone, version change, design direction, new runbook/config — after the owner confirms it works |
|
||||
|
||||
- **Plans:** header `# Plan: <title>` + `**Created:** <timestamp>`. Plan mode can only write to
|
||||
`~/.Codex/plans/`, so right after `ExitPlanMode` the first action is copying the plan into
|
||||
`docs/plans/` and committing it on its own, before any work starts.
|
||||
- **Execution logs:** a `- [ ] Step N — <title>` checklist on top (the authoritative status; a new
|
||||
session resumes from this file alone), then one section per step in chronological order,
|
||||
committed with that step's work. Light prose; exact commands as fenced blocks (`clusterctl`,
|
||||
`helm`, `kubectl apply`, `qm`, Packer), each with one line on why it was run that way, including
|
||||
commands the owner reports running on the lab and outputs that mattered (VM IDs, IPs, resolved
|
||||
versions, errors). Call out deviations from the plan; end with "Worth noting". Problems also go
|
||||
into the relevant fixes table, linked from the log.
|
||||
- **Session notes:** write one unprompted when a conversation produced a finding worth keeping
|
||||
(option compared or rejected, compatibility check, root cause, open question) that no plan
|
||||
covers; mention it in the reply. Sections: Question, What was checked (sources, versions with
|
||||
date checked), Findings, Open questions, Follow-ups. Extend the note if the topic continues.
|
||||
- **Decisions:** never rewrite a row; a reversal is a new row and the old one becomes
|
||||
`superseded by #N`. Design-doc suggestions are `proposed` until lab work confirms them. Link
|
||||
the plan/session/doc with the reasoning. README "Key decisions so far" stays the short summary.
|
||||
|
||||
## Git
|
||||
|
||||
- Remote: Gitea `gitea.home.hrajfrisbee.cz/kacerr/cluster-autoscaler-with-capi`.
|
||||
- Docs, configs and history files go straight to `main`. Phase-3 Go code or anything the owner
|
||||
calls a feature: `feat/<slug>` branch, `git push -u`, MR via
|
||||
`tea pr create --title … --description … --base main --head <branch>` (fallback: print
|
||||
`…/compare/main...<branch>`). Never merge or delete branches from the CLI.
|
||||
- Commit messages end with a `Co-Authored-By:` trailer for Codex. `/ship` commits and pushes.
|
||||
@@ -74,6 +74,7 @@ Summaries only, never raw transcripts; mask secrets and check the staged diff be
|
||||
| Decisions | `docs/decisions.md` (one table) | a direction chosen, rejected or reversed |
|
||||
| Prompts | `docs/prompts/YYYY-MM-DD-HHMM-<slug>.md` | long crafted prompts/specs that kick off work (verbatim) |
|
||||
| Reviews | `docs/reviews/YYYY-MM-DD-HHMM-<slug>.md` | review findings, most severe first, `- [ ]` ticked as processed |
|
||||
| Future tasks | `docs/future-tasks.md`, newest on top | work the owner defers: problem, options, cost, changes, owner-run commands |
|
||||
| Changelog | `CHANGELOG.md`, newest on top | lab milestone, version change, design direction, new runbook/config — after the owner confirms it works |
|
||||
|
||||
- **Plans:** header `# Plan: <title>` + `**Created:** <timestamp>`. Plan mode can only write to
|
||||
|
||||
@@ -82,6 +82,7 @@ Phases 2 and 3 are independent and can be done in either order.
|
||||
│ └── cluster-inspector/ # RBAC, kubeconfig minting script, gitignored kubeconfigs/
|
||||
├── docs/
|
||||
│ ├── decisions.md # decision log: alternatives, why, status
|
||||
│ ├── future-tasks.md # deferred work with context, newest first
|
||||
│ ├── plans/ # plans (plan mode), one file per plan
|
||||
│ ├── plans-executions/ # per-plan execution logs with exact commands
|
||||
│ ├── sessions/ # research/design conversation summaries
|
||||
|
||||
+24
-6
@@ -13,9 +13,13 @@
|
||||
# 2. kube-vip v0.7.1 -> v1.2.4 (kube-vip-prepare.sh kept)
|
||||
# 3. `format: qcow2` removed from both ProxmoxMachineTemplates (local-lvm is LVM-thin, raw only)
|
||||
# 4. `vlan: 4` on net0 of both ProxmoxMachineTemplates
|
||||
# 5. lab1-worker: 1 socket x 4 cores, 4096 MiB (owner's choice; runbook said 8192)
|
||||
# 5. worker template: 1 socket x 4 cores, 2048 MiB, named lab1-worker-2g (owner's choice, decision #16;
|
||||
# first 4096 MiB as lab1-worker). Templates are never edited in place: a new size = a new
|
||||
# template name + MD infrastructureRef, which rolls the workers.
|
||||
# 6. MachineDeployment lab1-workers: `replicas` removed (the autoscaler owns it)
|
||||
# 7. MachineDeployment lab1-workers: autoscaler min/max + capacity annotations (runbook Step 5)
|
||||
# 8. MachineDeployment lab1-workers: rollout maxSurge 0 / maxUnavailable 1 (memory-tight host)
|
||||
# 9. ProxmoxCluster lab1: schedulerHints.memoryAdjustment 160 (CAPMOX memory overcommit, decision #17)
|
||||
apiVersion: cluster.x-k8s.io/v1beta2
|
||||
kind: Cluster
|
||||
metadata:
|
||||
@@ -58,6 +62,11 @@ spec:
|
||||
- 192.168.4.31-192.168.4.39
|
||||
gateway: 192.168.4.8
|
||||
prefix: 24
|
||||
# CAPMOX reserves the configured memory of every non-template VM, stopped ones included, against
|
||||
# host total × memoryAdjustment%. 160% of 29.16 GiB = 46.7 GiB (~17.7 GiB headroom today).
|
||||
# Real overcommit protection on pv1-1: KSM (ksmtuned) + 8 GiB swap.
|
||||
schedulerHints:
|
||||
memoryAdjustment: 160
|
||||
---
|
||||
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
|
||||
kind: KubeadmControlPlane
|
||||
@@ -227,9 +236,9 @@ metadata:
|
||||
annotations:
|
||||
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
|
||||
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
|
||||
# scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker
|
||||
# scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g
|
||||
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets 1 × numCores 4
|
||||
capacity.cluster-autoscaler.kubernetes.io/memory: "4Gi" # memoryMiB 4096
|
||||
capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048
|
||||
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" # bootVolume sizeGb 40
|
||||
capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
|
||||
labels:
|
||||
@@ -238,6 +247,15 @@ metadata:
|
||||
namespace: default
|
||||
spec:
|
||||
clusterName: lab1
|
||||
# Delete an old worker before creating its replacement. The default surge (1/0) deadlocks on a full
|
||||
# host: CAPMOX can't reserve memory for the surge VM, and CA won't scale down the old workers while
|
||||
# the MD is rolling out (clusterapi ScaleDownNodeUpgradeProcessor).
|
||||
rollout:
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxSurge: 0
|
||||
maxUnavailable: 1
|
||||
selector:
|
||||
matchLabels: null
|
||||
template:
|
||||
@@ -255,13 +273,13 @@ spec:
|
||||
infrastructureRef:
|
||||
apiGroup: infrastructure.cluster.x-k8s.io
|
||||
kind: ProxmoxMachineTemplate
|
||||
name: lab1-worker
|
||||
name: lab1-worker-2g
|
||||
version: v1.35.9
|
||||
---
|
||||
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
|
||||
kind: ProxmoxMachineTemplate
|
||||
metadata:
|
||||
name: lab1-worker
|
||||
name: lab1-worker-2g
|
||||
namespace: default
|
||||
spec:
|
||||
template:
|
||||
@@ -271,7 +289,7 @@ spec:
|
||||
disk: scsi0
|
||||
sizeGb: 40
|
||||
full: true
|
||||
memoryMiB: 4096
|
||||
memoryMiB: 2048
|
||||
network:
|
||||
networkDevices:
|
||||
- bridge: vmbr0
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default -f values.yaml
|
||||
# helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler --version 9.59.0 -n default -f values.yaml
|
||||
# must be installed in the same namespace as the <cluster>-kubeconfig secret
|
||||
cloudProvider: clusterapi
|
||||
clusterAPIMode: kubeconfig-incluster # workload via kubeconfig, management in-cluster
|
||||
@@ -8,10 +8,26 @@ autoDiscovery:
|
||||
namespace: default
|
||||
image:
|
||||
tag: v1.35.2 # chart default appVersion is 1.35.0
|
||||
rbac:
|
||||
# CA reads the infrastructure machine template (capacity for scale-from-zero). The chart's
|
||||
# ClusterRole only covers cluster.x-k8s.io; without this the first main loop blocks forever on
|
||||
# "waiting for cache sync on infrastructure resource" and CA never scales.
|
||||
additionalRules:
|
||||
- apiGroups: [infrastructure.cluster.x-k8s.io]
|
||||
resources: [proxmoxmachinetemplates]
|
||||
verbs: [get, list, watch]
|
||||
extraArgs:
|
||||
expander: least-waste
|
||||
scale-down-unneeded-time: 5m
|
||||
scale-down-delay-after-add: 5m
|
||||
scale-down-utilization-threshold: "0.5"
|
||||
# Aggressive lab scale-down profile (CA 1.35 defaults in brackets): idle worker gone in ~3 min.
|
||||
# Keep delay-after-add >= node startup (~2-3 min here) or bursts flap up/down.
|
||||
scale-down-unneeded-time: 1m # [10m]
|
||||
scale-down-delay-after-add: 2m # [10m] cluster-wide cooldown after any scale-up
|
||||
scale-down-delay-after-failure: 1m # [3m]
|
||||
unremovable-node-recheck-timeout: 1m # [5m]
|
||||
scale-down-utilization-threshold: "0.6" # [0.5]
|
||||
max-graceful-termination-sec: "60" # [600] drain wait per pod
|
||||
balance-similar-node-groups: true
|
||||
# Cilium taints new nodes NoSchedule until its agent is up. Without this, CA simulates the freshly
|
||||
# registered (still tainted) node, finds the pending pods don't fit and adds one node too many.
|
||||
startup-taint: node.cilium.io/agent-not-ready
|
||||
v: 4
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# apply to the workload cluster; scale replicas to trigger scale-up
|
||||
# 2 pods per 4 vCPU / 4 GiB worker: CPU-bound (2 × 1500m); 1Gi keeps memory from binding first
|
||||
# (a 4 GiB node has ~3.3–3.6 GiB allocatable, which would fit only one 2Gi pod)
|
||||
# 2 pods per 4 vCPU / 2 GiB worker: CPU-bound (3 × 1500m > 4000m); 512Mi keeps memory from binding
|
||||
# first (a 2 GiB VM has ~1.78 GiB allocatable; measured 3.73 GiB on a 4 GiB VM)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
@@ -17,4 +17,4 @@ spec:
|
||||
- name: pause
|
||||
image: registry.k8s.io/pause:3.10
|
||||
resources:
|
||||
requests: { cpu: "1500m", memory: "1Gi" }
|
||||
requests: { cpu: "1500m", memory: "512Mi" }
|
||||
|
||||
+3
-1
@@ -23,4 +23,6 @@ image-builder fixes log; their dates are when the decision was recorded, not nec
|
||||
| 12 | 2026-09-27 | Management cluster built by one idempotent Ansible playbook, `ansible/mgmt.yml`, run from the laptop ([plan](plans/2026-09-27-0058-mgmt-cluster-ansible.md)) | cloud-init + k3s auto-deploy manifests; bash scripts with guards; OpenTofu + Helm | owner's choice; idempotent modules, one tool; only `ansible.builtin`, charts through k3s's helm-controller so nothing extra on the VM | active |
|
||||
| 13 | 2026-09-27 | CAPI providers installed by cluster-api-operator 0.28.0 (Provider CRs), cert-manager v1.21.2 pre-installed | guarded `clusterctl init` | declarative; re-apply is a no-op; upgrade = `spec.version` bump; 0.28.0 built against CAPI 1.12.10. Consequence: no `clusterctl init/upgrade/move/delete` | active |
|
||||
| 14 | 2026-09-27 | Management VM created by the playbook from the pinned Ubuntu 24.04 cloud image (VM 201, 192.168.4.21) | hand-made VM; the image-builder node template | reproducible from the repo; the image-builder template is a kubeadm node image, not meant for k3s | active |
|
||||
| 15 | 2026-09-27 | `lab1` workers 4 vCPU / **4 GiB** (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker | 8 GiB workers with 2Gi inflate pods | owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape ([configs/capi/lab1.yaml](../configs/capi/lab1.yaml)) | active |
|
||||
| 15 | 2026-09-27 | `lab1` workers 4 vCPU / **4 GiB** (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker | 8 GiB workers with 2Gi inflate pods | owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape ([configs/capi/lab1.yaml](../configs/capi/lab1.yaml)) | superseded by #16 |
|
||||
| 16 | 2026-09-28 | `lab1` workers 4 vCPU / **2 GiB** (new `ProxmoxMachineTemplate/lab1-worker-2g`, rolled in via the MD); inflate pods 1500m / **512Mi**, still CPU-bound at 2 per worker | keep 4 GiB and delete stopped VMs; CAPMOX `memoryAdjustment` overcommit | CAPMOX reserves the configured memory of every VM, stopped ones included, so 4 GiB workers ran out after the first scale-up ([troubleshooting](phase-1-proxmox-capi.md#troubleshooting)); 2 GiB (~1.78 GiB allocatable) still leaves room for kubelet/containerd plus real pods, and max 5 workers now fits the host | proposed |
|
||||
| 17 | 2026-09-28 | CAPMOX memory overcommit on `lab1`: `ProxmoxCluster.spec.schedulerHints.memoryAdjustment: 160` (46.7 GiB reservable on the 29.16 GiB `pv1-1`); MD rollout `maxSurge 0 / maxUnavailable 1` | 100% (default); 120% (minimum sizing for max 5 × 2 GiB workers); 0 (memory check off); deleting stopped VMs | CAPMOX counts the configured memory of stopped VMs (12 GiB here), and no release can use real free memory; 160% keeps a bound, and host-side protection is KSM + swap, not ballooning ([session](sessions/2026-09-28-1007-capmox-memory-overcommit.md)) | proposed |
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Future tasks
|
||||
|
||||
Deferred work, newest first. When a task is picked up it gets a plan in `docs/plans/`; then tick
|
||||
it here and link the plan.
|
||||
|
||||
## - [ ] Namespace per workload cluster (`lab1` out of `default`)
|
||||
|
||||
**Added:** 2026-09-28 09:29 CEST · **Why deferred:** keep phase 1 moving; `default` works for now.
|
||||
|
||||
**Problem.** cluster-autoscaler has to run in the same namespace as the Cluster. The chart mounts
|
||||
`clusterAPIKubeconfigSecret` as a Secret volume, which only works within the pod's namespace, and
|
||||
`lab1-kubeconfig` lives in the Cluster's namespace. `lab1` is in `default`, so CA would be installed
|
||||
into `default` too. That mixes things in `default`, and with more clusters (phase 3) you'd get one
|
||||
CA per cluster sitting side by side in `default`.
|
||||
|
||||
**Options compared:**
|
||||
|
||||
| Option | Verdict |
|
||||
| --- | --- |
|
||||
| CA in `default` (current Step 6) | works; poor hygiene, doesn't scale to more clusters |
|
||||
| CA in its own namespace with a copy of the kubeconfig Secret | rejected: CAPI rewrites `lab1-kubeconfig` when it rotates the client cert, so the copy goes stale and CA silently loses the workload cluster |
|
||||
| **One namespace per cluster** (`lab1` holds the CAPI objects, `InClusterIPPool`, kubeconfig Secret and CA) | preferred: the usual CAPI multi-cluster layout, one CA per workload cluster, RBAC and cleanup per namespace, matches phase 3 |
|
||||
|
||||
**Cost.** Namespace is immutable on CAPI objects, so `lab1` has to be deleted and recreated. That
|
||||
means about 15 minutes of provisioning plus reinstalling Cilium. `clusterctl move` is not an option,
|
||||
since it's forbidden with cluster-api-operator. CAPMOX credentials come from the controller's
|
||||
global Secret, so a new namespace needs no extra credentials. The cost grows once workloads and
|
||||
autoscaler state exist on `lab1`, so do it before the Step 7 results matter, or when creating the
|
||||
next cluster.
|
||||
|
||||
**Changes:**
|
||||
|
||||
- `configs/capi/lab1.yaml`: a `Namespace lab1` object and `metadata.namespace: lab1` on every
|
||||
object, including `InClusterIPPool`.
|
||||
- `configs/cluster-autoscaler/values.yaml`: `autoDiscovery.namespace: lab1`; install with `-n lab1`
|
||||
(`helm uninstall ca -n default` first if it was installed in `default`).
|
||||
- `ansible/group_vars/all.yml`: `inspector_workload_clusters` entry `{ name: lab1, namespace: lab1 }`.
|
||||
- `docs/phase-1-proxmox-capi.md`: Steps 3, 5 and 6 (namespace in commands and log expectations,
|
||||
`MachineDeployment/lab1/lab1-workers`).
|
||||
- `.claude/agents/cluster-inspector.md`: the "Where things live" table says CA runs in `default`.
|
||||
- `docs/decisions.md`: new row (namespace per workload cluster).
|
||||
|
||||
**Run by the owner:** `kubectl delete -f configs/capi/lab1.yaml` (old namespace), wait until the
|
||||
VMs are gone, `kubectl apply` the new manifest, Cilium `helm upgrade --install`,
|
||||
`ansible-playbook mgmt.yml --tags inspector`, then the CA install into `lab1`.
|
||||
@@ -12,7 +12,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
|
||||
| --- | --- | --- |
|
||||
| Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
|
||||
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
|
||||
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 4 GB RAM |
|
||||
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 2 GB RAM |
|
||||
|
||||
## Checklist
|
||||
|
||||
@@ -107,7 +107,10 @@ only when a version changes, then redo these edits:
|
||||
(handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
|
||||
3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
|
||||
4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
|
||||
5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 4096`.
|
||||
5. **Worker size** — rename the worker template to `ProxmoxMachineTemplate/lab1-worker-2g` (and the MD
|
||||
`infrastructureRef`): `numSockets: 1`, `numCores: 4`, `memoryMiB: 2048`. To resize later, add a
|
||||
template with a new name and point the MD at it; CAPI then rolls the workers. Don't edit a
|
||||
template in place.
|
||||
6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.
|
||||
|
||||
```bash
|
||||
@@ -157,7 +160,7 @@ metadata:
|
||||
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
|
||||
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
|
||||
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores
|
||||
capacity.cluster-autoscaler.kubernetes.io/memory: "4Gi" # memoryMiB 4096
|
||||
capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048
|
||||
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
|
||||
capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
|
||||
```
|
||||
@@ -183,20 +186,39 @@ Install into the **same namespace as the `lab1-kubeconfig` secret** (here `defau
|
||||
[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):
|
||||
|
||||
```bash
|
||||
helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update
|
||||
helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \
|
||||
-f configs/cluster-autoscaler/values.yaml
|
||||
export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig
|
||||
helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler \
|
||||
--version 9.59.0 -n default -f configs/cluster-autoscaler/values.yaml
|
||||
```
|
||||
|
||||
The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5.
|
||||
The values add `rbac.additionalRules` to read `proxmoxmachinetemplates`. The chart's ClusterRole
|
||||
doesn't cover infrastructure templates, and without them the first main loop hangs (see
|
||||
troubleshooting). Health check: the log shows `Starting main loop` followed by regular
|
||||
`Autoscaler loop triggered` lines, and `cluster-autoscaler-status` on lab1 leaves `Initializing`.
|
||||
|
||||
CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`. Its log must
|
||||
show the node group; the line is logged at `v: 4` on every loop (`clusterapi_controller.go`,
|
||||
CA 1.35.2):
|
||||
|
||||
```bash
|
||||
kubectl logs deploy/ca-clusterapi-cluster-autoscaler -n default --kubeconfig ~/.kube/lab-mgmt.kubeconfig \
|
||||
| grep 'discovered node group'
|
||||
# discovered node group: MachineDeployment/default/lab1-workers (min: 0, max: 5, replicas: 1)
|
||||
```
|
||||
|
||||
The status ConfigMap and the leader-election lock live on the workload cluster, not on the management
|
||||
cluster: `kubectl get cm cluster-autoscaler-status -n kube-system --kubeconfig ~/.kube/lab1.kubeconfig`
|
||||
(chart arg `--namespace=kube-system`, because `clusterAPIConfigMapsNamespace` is empty).
|
||||
|
||||
## Step 7 — scale tests
|
||||
|
||||
Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 1Gi each →
|
||||
CPU-bound at 2 pods per 4-core / 4 GiB worker).
|
||||
Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 512Mi each →
|
||||
CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocatable).
|
||||
|
||||
- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
|
||||
- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear.
|
||||
- [ ] Scale-down: `--replicas=0` → after `scale-down-delay-after-add` (2m) + `scale-down-unneeded-time`
|
||||
(1m) Machines and VMs disappear. That's the aggressive lab profile in the CA values; per-MD overrides are
|
||||
`cluster.x-k8s.io/autoscaling-options-{scaledownunneededtime,scaledownutilizationthreshold,…}` annotations.
|
||||
- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
|
||||
- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.
|
||||
|
||||
@@ -207,8 +229,12 @@ CPU-bound at 2 pods per 4-core / 4 GiB worker).
|
||||
| Symptom | Cause | Fix |
|
||||
| --- | --- | --- |
|
||||
| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
|
||||
| CA runs and discovers the node group but never scales; log stops at `waiting for cache sync on infrastructure resource`, `proxmoxmachinetemplates … forbidden` every ~40 s; status ConfigMap stuck on `Initializing` | chart 9.59.0 ClusterRole has no rule for `infrastructure.cluster.x-k8s.io`; CA's first main loop waits for a template informer that can never sync | `rbac.additionalRules` with get/list/watch on `proxmoxmachinetemplates` in the CA values, `helm upgrade` |
|
||||
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
|
||||
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
|
||||
| scale-up adds one node too many (6 pods at 2 per node → 4 workers, second plan `3->4` about 70 s after `1->3`) | a new node registers with Cilium's `node.cilium.io/agent-not-ready:NoSchedule`; once registered it's no longer an "upcoming" template node, and CA sees pending pods that don't fit on it | `extraArgs.startup-taint: node.cilium.io/agent-not-ready` in the CA values (CA treats the node as still starting) |
|
||||
| CA scales the MD up, new Machines stay without a VM; ProxmoxMachine `VMProvisionFailed: cannot reserve 4294967296B of memory on node pv1-1: …B available memory left` although `free` shows GBs available | CAPMOX v0.9.1 scheduler (`GetReservableMemoryBytes`): node total × `schedulerHints.memoryAdjustment` (default 100%) minus the **configured** memory of every non-template VM, **running or stopped**, and every container. Stopped VMs 100/1001/2001 hold 12 GiB | free the reservation (delete unused stopped VMs, or convert them to templates), or set `ProxmoxCluster.spec.schedulerHints.memoryAdjustment` > 100 (overcommit); keep the MD max-size within what the host can actually run |
|
||||
| MD rollout never finishes and CA never scales down: CA logs `Node … will be skipped as it is currently under rollout`, new ProxmoxMachine `cannot reserve …B of memory` | deadlock. Default rollout surge (`maxSurge 1 / maxUnavailable 0`) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (`ScaleDownNodeUpgradeProcessor`) | `spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1}` on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory |
|
||||
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
|
||||
| `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` |
|
||||
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Session: CAPMOX memory accounting and overcommit on pv1-1
|
||||
|
||||
**Created:** 2026-09-28 10:07 CEST
|
||||
|
||||
## Question
|
||||
|
||||
> can we configure capmox so that it works with real available memory instead of calculation what
|
||||
> stopped vms would use if somebody would start them? also can we configure memory overprovisioning
|
||||
> on proxmox node?
|
||||
|
||||
Context: during Step 7, new workers failed with `cannot reserve …B of memory on node pv1-1`. The
|
||||
host had about 12.8 GiB actually available, but CAPMOX counted 168 MB. Later this deadlocked the
|
||||
2 GiB worker rollout (see the phase-1 troubleshooting table).
|
||||
|
||||
## What was checked (2026-09-28)
|
||||
|
||||
- CAPMOX source for v0.9.1 (the latest tag) and `main`:
|
||||
- `pkg/proxmox/goproxmox/api_client.go` `GetReservableMemoryBytes`;
|
||||
- `internal/service/scheduler/vmscheduler.go`;
|
||||
- `api/v1alpha2/proxmoxcluster_types.go` `SchedulerHints`;
|
||||
- `api/v1alpha2/proxmoxmachine_types.go`;
|
||||
- `internal/webhook/proxmoxcluster_webhook.go`.
|
||||
- `pv1-1` (read-only):
|
||||
- `qm list`, node memory status;
|
||||
- KSM: `ksmtuned` active, `/sys/kernel/mm/ksm/run` = 0, `pages_sharing` = 0;
|
||||
- 8 GiB swap;
|
||||
- VM 103 has no `balloon:` line.
|
||||
|
||||
## Findings
|
||||
|
||||
- **No "real available memory" mode in any CAPMOX release.** `main` is identical to v0.9.1. The
|
||||
reservable memory is `node total × memoryAdjustment/100` minus the configured `maxmem` of every
|
||||
non-template VM (running or stopped) and every container. That's deliberate: real usage is
|
||||
racy, because VMs grow into their configured memory later. Using free memory would need an
|
||||
upstream change.
|
||||
- **The only knob is `ProxmoxCluster.spec.schedulerHints.memoryAdjustment`** (percent, default 100):
|
||||
- `> 100` allows bounded overcommit;
|
||||
- `0` turns the memory check off completely (it then only checks request ≤ host total).
|
||||
- The ProxmoxCluster update webhook only validates the control-plane endpoint, so the hint can
|
||||
be changed on a live cluster. The scheduler reads it whenever it places a VM, and the stuck
|
||||
ProxmoxMachine retries.
|
||||
- **Sizing for `lab1`:** everything except the workers is 21 GiB configured (VMs 100, 102, 201,
|
||||
1001, 2001, 160549). 5 workers × 2 GiB brings that to 31 GiB, and the host has 29.16 GiB, so
|
||||
`memoryAdjustment ≥ 107`. At 120 there's 35 GiB reservable, which fits max 5 workers with about
|
||||
4 GiB to spare.
|
||||
- **Proxmox itself enforces no memory overcommit limit.** VMs can be started past physical RAM; CAPMOX
|
||||
was the only admission control. On the PVE side, "overprovisioning" is about how the host copes:
|
||||
- **KSM:** `ksmtuned` is active but only merges below 20% free (`KSM_THRES_COEF`). Identical
|
||||
Ubuntu clones deduplicate well. It's currently idle because the host has about 44% free.
|
||||
- **Host swap (8 GiB):** a safety net, but slow.
|
||||
- **Ballooning** (`balloon: <min>` below `memory`, auto-balloon above 80% host use): not
|
||||
exposed by CAPMOX v0.9.1 (`ProxmoxMachineSpec` has no balloon field). It's also a poor fit
|
||||
for Kubernetes nodes: the guest keeps reporting full capacity while the balloon takes memory
|
||||
away, so the kubelet schedules against memory that isn't there.
|
||||
|
||||
## Open questions
|
||||
|
||||
- ~~120 or 0?~~ The owner chose **160** (decision #17): 46.7 GiB reservable, about 17.7 GiB headroom with today's VMs.
|
||||
- Is an upstream feature request worth filing (a free-memory mode, or excluding stopped VMs)?
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- Done: `schedulerHints.memoryAdjustment: 160` on `ProxmoxCluster/lab1` in `configs/capi/lab1.yaml`,
|
||||
decisions #17. Watch real host memory and KSM `pages_sharing` during scale tests.
|
||||
Reference in New Issue
Block a user