diff --git a/.claude/settings.json b/.claude/settings.json index 35f4e75..ee1515e 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -26,7 +26,27 @@ "Bash(ansible pve *)", "Read(//Users/jan.novak/**)", "Bash(export KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig)", - "Bash(kubectl explain *)" + "Bash(kubectl explain *)", + "Bash(grep -v '^ *$')", + "Bash(git ls-remote *)", + "Bash(awk -F/ '{print $3}')", + "Bash(curl -fsSL https://kubernetes.github.io/autoscaler/index.yaml)", + "Bash(awk '/^ cluster-autoscaler:/{f=1} f&&/appVersion:/{a=$2} f&&/^ version:/{print $2, a; n++} n>=6{exit}')", + "Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get md lab1-workers -n default -o json)", + "Bash(clusterctl --kubeconfig /Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig describe cluster lab1 -n default --show-conditions all)", + "Bash(KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig kubectl get md lab1-workers -n default -o json --show-managed-fields)", + "Bash(export KUBECONFIG=$K)", + "Bash(helm show *)", + "Bash(grep -rniE 'backlog|future tasks|TODO' --include='*.md' . | head; sed -n '/## Key decisions/,/^## /p' README.md | head -5; grep -n 'docs/' README.md | head -20)", + "Bash(helm template *)", + "Bash(sed -e 's|^\\\\*\\\\*Created:\\\\*\\\\* .*|**Created:** 2026-09-29 23:15 CEST|' /Users/jan.novak/.claude/plans/purrfect-frolicking-lovelace.md > docs/plans/2026-09-29-2315-pv1-2-workers.md && head -3 docs/plans/2026-09-29-2315-pv1-2-workers.md)", + "WebSearch", + "WebFetch(domain:github.com)", + "Bash(git apply *)" + ], + "additionalDirectories": [ + "/Users/jan.novak/srv/go/cluster-autoscaler-with-capi", + "/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/docs/plans" ] } } diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..275bc26 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,104 @@ +# AGENTS.md + +Home-lab research project: cluster-autoscaler + Cluster API on Proxmox VE (phase 1), then a +KVM/libvirt provider (phase 2) and Kamaji hosted control planes / "poor man's KaaS" (phase 3). +Start with README.md; details live in docs/. + +## Owner context + +Experienced SRE. Be concise and technical; skip basics (k8s internals, Go, Linux, distributed systems). + +## Lab facts (don't re-derive) + +- PVE node `pv1-1`, API `https://192.168.0.8:8006`, token `capmox@pve!capi` (privsep 0). +- VM network: `vmbr0` is VLAN-aware, VMs in **VLAN 4** → always `bridge=vmbr0` + tag/`vlan: 4`, + never `vmbr0.4@vmbr0` (that is an `ip link` name, not a bridge). +- Storage: `local` (iso, dir) and `local-lvm` (LVM-thin) → disks are **raw**, never qcow2. +- Build host for image-builder: 192.168.4.181 (VLAN 4 must reach it on 8000–9000 during builds). +- Node template: `ubuntu-2404-kube-v1.35.9` (image-builder v0.1.55). Login only via cloud-init. +- Management cluster: VM 201 `lab-mgmt`, 192.168.4.21 (`ssh ubuntu@`), built by the idempotent + `ansible/mgmt.yml` (re-run must be `changed=0`); admin kubeconfig `~/.kube/lab-mgmt.kubeconfig`. +- Required PVE ACLs: PVEVMAdmin + PVEAuditor on `/`, PVEDatastoreAdmin on `/storage`, + PVESDNUser on `/sdn/zones/localnetwork`. + +## Version rules + +- `docs/versions.md` is the source of truth. Any version change updates it in the same change. +- Before proposing any deployment or upgrade, verify versions are current (use the + `k8s-version-check` skill if available). Go module proxy / git ls-remote / Helm index.yaml work + even when the GitHub API is unavailable. +- CAPI core is pinned to **v1.12.x** because CAPMOX v0.9 is only tested on CAPI 1.11/1.12. This caps + workload Kubernetes at 1.35 and cluster-autoscaler at 1.35.x. Don't bump CAPI unless the CAPMOX + compatibility table allows it. +- Always pin versions explicitly (management cluster: `ansible/group_vars/all.yml`; `helm install + --version`); never rely on "latest". +- CAPI providers are managed by cluster-api-operator (Provider CRs from `ansible/mgmt.yml`). Never + run `clusterctl init/upgrade/move/delete` against the management cluster; `clusterctl` is only for + `generate cluster` (always with `--infrastructure proxmox:v0.9.1`), `describe`, `get kubeconfig`. +- Check a provider's `metadata.yaml` contract and CAPI version (go.mod) before adding it to the + management cluster — all providers share one CAPI core (known skew: Kamaji CP provider v0.21.0 is + on CAPI 1.13). + +## Facts about providers: verify in source + +READMEs lag. For "does provider X support Y" read the API types and templates of the exact tagged +release (Go proxy zip: `https://proxy.golang.org//@v/.zip`). Delegate to the +`upstream-researcher` agent for anything non-trivial. + +## Safety + +- Never commit secrets: PVE token secret, kubeconfigs, `clusterctl.yaml` (see .gitignore). +- Cluster inspection is read-only via the `cluster-inspector` agent and read-only kubeconfigs + (`.Codex/cluster-inspector/`, see its README). Mutations (`kubectl apply/delete/scale`, `qm`, `pveum`) + are proposed as commands for the owner to run unless explicitly asked to execute them. +- Don't delete Proxmox VMs/templates or CAPI objects on your own initiative. + +## Conventions + +- Docs in English, Markdown, Mermaid for diagrams (Gitea renders it). +- Troubleshooting findings go into the relevant doc's troubleshooting/fixes table: + symptom → root cause → fix. +- Example configs live in `configs/` as `*.example` or non-secret values files. + +## Project history + +Research repo: plans, conversations and decisions are kept in the repo, dated. Timestamps always +come from `date` (`+%Y-%m-%d-%H%M` for file names, `+%Y-%m-%d %H:%M %Z` for headings), never guessed. +Summaries only, never raw transcripts; mask secrets and check the staged diff before committing. + +| What | Where | When | +| --- | --- | --- | +| Plans | `docs/plans/YYYY-MM-DD-HHMM-.md` | every plan-mode plan | +| Execution logs | `docs/plans-executions/` | after each step of an approved plan | +| Session notes | `docs/sessions/YYYY-MM-DD-HHMM-.md` | research/design/debugging conversations without a plan | +| Decisions | `docs/decisions.md` (one table) | a direction chosen, rejected or reversed | +| Prompts | `docs/prompts/YYYY-MM-DD-HHMM-.md` | long crafted prompts/specs that kick off work (verbatim) | +| Reviews | `docs/reviews/YYYY-MM-DD-HHMM-.md` | review findings, most severe first, `- [ ]` ticked as processed | +| Changelog | `CHANGELOG.md`, newest on top | lab milestone, version change, design direction, new runbook/config — after the owner confirms it works | + +- **Plans:** header `# Plan: ` + `**Created:** <timestamp>`. Plan mode can only write to + `~/.Codex/plans/`, so right after `ExitPlanMode` the first action is copying the plan into + `docs/plans/` and committing it on its own, before any work starts. +- **Execution logs:** a `- [ ] Step N — <title>` checklist on top (the authoritative status; a new + session resumes from this file alone), then one section per step in chronological order, + committed with that step's work. Light prose; exact commands as fenced blocks (`clusterctl`, + `helm`, `kubectl apply`, `qm`, Packer), each with one line on why it was run that way, including + commands the owner reports running on the lab and outputs that mattered (VM IDs, IPs, resolved + versions, errors). Call out deviations from the plan; end with "Worth noting". Problems also go + into the relevant fixes table, linked from the log. +- **Session notes:** write one unprompted when a conversation produced a finding worth keeping + (option compared or rejected, compatibility check, root cause, open question) that no plan + covers; mention it in the reply. Sections: Question, What was checked (sources, versions with + date checked), Findings, Open questions, Follow-ups. Extend the note if the topic continues. +- **Decisions:** never rewrite a row; a reversal is a new row and the old one becomes + `superseded by #N`. Design-doc suggestions are `proposed` until lab work confirms them. Link + the plan/session/doc with the reasoning. README "Key decisions so far" stays the short summary. + +## Git + +- Remote: Gitea `gitea.home.hrajfrisbee.cz/kacerr/cluster-autoscaler-with-capi`. +- Docs, configs and history files go straight to `main`. Phase-3 Go code or anything the owner + calls a feature: `feat/<slug>` branch, `git push -u`, MR via + `tea pr create --title … --description … --base main --head <branch>` (fallback: print + `…/compare/main...<branch>`). Never merge or delete branches from the CLI. +- Commit messages end with a `Co-Authored-By:` trailer for Codex. `/ship` commits and pushes. diff --git a/CLAUDE.md b/CLAUDE.md index b6fbd13..5da2645 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -74,6 +74,7 @@ Summaries only, never raw transcripts; mask secrets and check the staged diff be | Decisions | `docs/decisions.md` (one table) | a direction chosen, rejected or reversed | | Prompts | `docs/prompts/YYYY-MM-DD-HHMM-<slug>.md` | long crafted prompts/specs that kick off work (verbatim) | | Reviews | `docs/reviews/YYYY-MM-DD-HHMM-<slug>.md` | review findings, most severe first, `- [ ]` ticked as processed | +| Future tasks | `docs/future-tasks.md`, newest on top | work the owner defers: problem, options, cost, changes, owner-run commands | | Changelog | `CHANGELOG.md`, newest on top | lab milestone, version change, design direction, new runbook/config — after the owner confirms it works | - **Plans:** header `# Plan: <title>` + `**Created:** <timestamp>`. Plan mode can only write to diff --git a/README.md b/README.md index 918c71d..fae1799 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,7 @@ Phases 2 and 3 are independent and can be done in either order. │ └── cluster-inspector/ # RBAC, kubeconfig minting script, gitignored kubeconfigs/ ├── docs/ │ ├── decisions.md # decision log: alternatives, why, status +│ ├── future-tasks.md # deferred work with context, newest first │ ├── plans/ # plans (plan mode), one file per plan │ ├── plans-executions/ # per-plan execution logs with exact commands │ ├── sessions/ # research/design conversation summaries diff --git a/configs/capi/lab1.yaml b/configs/capi/lab1.yaml index 819b81e..afab13b 100644 --- a/configs/capi/lab1.yaml +++ b/configs/capi/lab1.yaml @@ -13,9 +13,13 @@ # 2. kube-vip v0.7.1 -> v1.2.4 (kube-vip-prepare.sh kept) # 3. `format: qcow2` removed from both ProxmoxMachineTemplates (local-lvm is LVM-thin, raw only) # 4. `vlan: 4` on net0 of both ProxmoxMachineTemplates -# 5. lab1-worker: 1 socket x 4 cores, 4096 MiB (owner's choice; runbook said 8192) +# 5. worker template: 1 socket x 4 cores, 2048 MiB, named lab1-worker-2g (owner's choice, decision #16; +# first 4096 MiB as lab1-worker). Templates are never edited in place: a new size = a new +# template name + MD infrastructureRef, which rolls the workers. # 6. MachineDeployment lab1-workers: `replicas` removed (the autoscaler owns it) # 7. MachineDeployment lab1-workers: autoscaler min/max + capacity annotations (runbook Step 5) +# 8. MachineDeployment lab1-workers: rollout maxSurge 0 / maxUnavailable 1 (memory-tight host) +# 9. ProxmoxCluster lab1: schedulerHints.memoryAdjustment 160 (CAPMOX memory overcommit, decision #17) apiVersion: cluster.x-k8s.io/v1beta2 kind: Cluster metadata: @@ -58,6 +62,11 @@ spec: - 192.168.4.31-192.168.4.39 gateway: 192.168.4.8 prefix: 24 + # CAPMOX reserves the configured memory of every non-template VM, stopped ones included, against + # host total × memoryAdjustment%. 160% of 29.16 GiB = 46.7 GiB (~17.7 GiB headroom today). + # Real overcommit protection on pv1-1: KSM (ksmtuned) + 8 GiB swap. + schedulerHints: + memoryAdjustment: 160 --- apiVersion: controlplane.cluster.x-k8s.io/v1beta2 kind: KubeadmControlPlane @@ -227,9 +236,9 @@ metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0" cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5" - # scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker + # scale-from-zero: CAPMOX publishes no status.capacity; must match ProxmoxMachineTemplate/lab1-worker-2g capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets 1 × numCores 4 - capacity.cluster-autoscaler.kubernetes.io/memory: "4Gi" # memoryMiB 4096 + capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048 capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" # bootVolume sizeGb 40 capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110" labels: @@ -238,6 +247,15 @@ metadata: namespace: default spec: clusterName: lab1 + # Delete an old worker before creating its replacement. The default surge (1/0) deadlocks on a full + # host: CAPMOX can't reserve memory for the surge VM, and CA won't scale down the old workers while + # the MD is rolling out (clusterapi ScaleDownNodeUpgradeProcessor). + rollout: + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 selector: matchLabels: null template: @@ -255,13 +273,13 @@ spec: infrastructureRef: apiGroup: infrastructure.cluster.x-k8s.io kind: ProxmoxMachineTemplate - name: lab1-worker + name: lab1-worker-2g version: v1.35.9 --- apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2 kind: ProxmoxMachineTemplate metadata: - name: lab1-worker + name: lab1-worker-2g namespace: default spec: template: @@ -271,7 +289,7 @@ spec: disk: scsi0 sizeGb: 40 full: true - memoryMiB: 4096 + memoryMiB: 2048 network: networkDevices: - bridge: vmbr0 diff --git a/configs/cluster-autoscaler/values.yaml b/configs/cluster-autoscaler/values.yaml index ce94720..9701734 100644 --- a/configs/cluster-autoscaler/values.yaml +++ b/configs/cluster-autoscaler/values.yaml @@ -1,4 +1,4 @@ -# helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default -f values.yaml +# helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler --version 9.59.0 -n default -f values.yaml # must be installed in the same namespace as the <cluster>-kubeconfig secret cloudProvider: clusterapi clusterAPIMode: kubeconfig-incluster # workload via kubeconfig, management in-cluster @@ -8,10 +8,26 @@ autoDiscovery: namespace: default image: tag: v1.35.2 # chart default appVersion is 1.35.0 +rbac: + # CA reads the infrastructure machine template (capacity for scale-from-zero). The chart's + # ClusterRole only covers cluster.x-k8s.io; without this the first main loop blocks forever on + # "waiting for cache sync on infrastructure resource" and CA never scales. + additionalRules: + - apiGroups: [infrastructure.cluster.x-k8s.io] + resources: [proxmoxmachinetemplates] + verbs: [get, list, watch] extraArgs: expander: least-waste - scale-down-unneeded-time: 5m - scale-down-delay-after-add: 5m - scale-down-utilization-threshold: "0.5" + # Aggressive lab scale-down profile (CA 1.35 defaults in brackets): idle worker gone in ~3 min. + # Keep delay-after-add >= node startup (~2-3 min here) or bursts flap up/down. + scale-down-unneeded-time: 1m # [10m] + scale-down-delay-after-add: 2m # [10m] cluster-wide cooldown after any scale-up + scale-down-delay-after-failure: 1m # [3m] + unremovable-node-recheck-timeout: 1m # [5m] + scale-down-utilization-threshold: "0.6" # [0.5] + max-graceful-termination-sec: "60" # [600] drain wait per pod balance-similar-node-groups: true + # Cilium taints new nodes NoSchedule until its agent is up. Without this, CA simulates the freshly + # registered (still tainted) node, finds the pending pods don't fit and adds one node too many. + startup-taint: node.cilium.io/agent-not-ready v: 4 diff --git a/configs/test/inflate.yaml b/configs/test/inflate.yaml index d8b9ada..d9b36ed 100644 --- a/configs/test/inflate.yaml +++ b/configs/test/inflate.yaml @@ -1,6 +1,6 @@ # apply to the workload cluster; scale replicas to trigger scale-up -# 2 pods per 4 vCPU / 4 GiB worker: CPU-bound (2 × 1500m); 1Gi keeps memory from binding first -# (a 4 GiB node has ~3.3–3.6 GiB allocatable, which would fit only one 2Gi pod) +# 2 pods per 4 vCPU / 2 GiB worker: CPU-bound (3 × 1500m > 4000m); 512Mi keeps memory from binding +# first (a 2 GiB VM has ~1.78 GiB allocatable; measured 3.73 GiB on a 4 GiB VM) apiVersion: apps/v1 kind: Deployment metadata: @@ -17,4 +17,4 @@ spec: - name: pause image: registry.k8s.io/pause:3.10 resources: - requests: { cpu: "1500m", memory: "1Gi" } + requests: { cpu: "1500m", memory: "512Mi" } diff --git a/docs/decisions.md b/docs/decisions.md index a957f82..6a1a35f 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -23,4 +23,6 @@ image-builder fixes log; their dates are when the decision was recorded, not nec | 12 | 2026-09-27 | Management cluster built by one idempotent Ansible playbook, `ansible/mgmt.yml`, run from the laptop ([plan](plans/2026-09-27-0058-mgmt-cluster-ansible.md)) | cloud-init + k3s auto-deploy manifests; bash scripts with guards; OpenTofu + Helm | owner's choice; idempotent modules, one tool; only `ansible.builtin`, charts through k3s's helm-controller so nothing extra on the VM | active | | 13 | 2026-09-27 | CAPI providers installed by cluster-api-operator 0.28.0 (Provider CRs), cert-manager v1.21.2 pre-installed | guarded `clusterctl init` | declarative; re-apply is a no-op; upgrade = `spec.version` bump; 0.28.0 built against CAPI 1.12.10. Consequence: no `clusterctl init/upgrade/move/delete` | active | | 14 | 2026-09-27 | Management VM created by the playbook from the pinned Ubuntu 24.04 cloud image (VM 201, 192.168.4.21) | hand-made VM; the image-builder node template | reproducible from the repo; the image-builder template is a kubeadm node image, not meant for k3s | active | -| 15 | 2026-09-27 | `lab1` workers 4 vCPU / **4 GiB** (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker | 8 GiB workers with 2Gi inflate pods | owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape ([configs/capi/lab1.yaml](../configs/capi/lab1.yaml)) | active | +| 15 | 2026-09-27 | `lab1` workers 4 vCPU / **4 GiB** (not 8 GiB); inflate test pods request 1500m / 1Gi so they stay CPU-bound at 2 per worker | 8 GiB workers with 2Gi inflate pods | owner's choice to save hypervisor RAM; scale tests keep the 1 → 3 worker shape ([configs/capi/lab1.yaml](../configs/capi/lab1.yaml)) | superseded by #16 | +| 16 | 2026-09-28 | `lab1` workers 4 vCPU / **2 GiB** (new `ProxmoxMachineTemplate/lab1-worker-2g`, rolled in via the MD); inflate pods 1500m / **512Mi**, still CPU-bound at 2 per worker | keep 4 GiB and delete stopped VMs; CAPMOX `memoryAdjustment` overcommit | CAPMOX reserves the configured memory of every VM, stopped ones included, so 4 GiB workers ran out after the first scale-up ([troubleshooting](phase-1-proxmox-capi.md#troubleshooting)); 2 GiB (~1.78 GiB allocatable) still leaves room for kubelet/containerd plus real pods, and max 5 workers now fits the host | proposed | +| 17 | 2026-09-28 | CAPMOX memory overcommit on `lab1`: `ProxmoxCluster.spec.schedulerHints.memoryAdjustment: 160` (46.7 GiB reservable on the 29.16 GiB `pv1-1`); MD rollout `maxSurge 0 / maxUnavailable 1` | 100% (default); 120% (minimum sizing for max 5 × 2 GiB workers); 0 (memory check off); deleting stopped VMs | CAPMOX counts the configured memory of stopped VMs (12 GiB here), and no release can use real free memory; 160% keeps a bound, and host-side protection is KSM + swap, not ballooning ([session](sessions/2026-09-28-1007-capmox-memory-overcommit.md)) | proposed | diff --git a/docs/future-tasks.md b/docs/future-tasks.md new file mode 100644 index 0000000..1195a0d --- /dev/null +++ b/docs/future-tasks.md @@ -0,0 +1,45 @@ +# Future tasks + +Deferred work, newest first. When a task is picked up it gets a plan in `docs/plans/`; then tick +it here and link the plan. + +## - [ ] Namespace per workload cluster (`lab1` out of `default`) + +**Added:** 2026-09-28 09:29 CEST · **Why deferred:** keep phase 1 moving; `default` works for now. + +**Problem.** cluster-autoscaler has to run in the same namespace as the Cluster. The chart mounts +`clusterAPIKubeconfigSecret` as a Secret volume, which only works within the pod's namespace, and +`lab1-kubeconfig` lives in the Cluster's namespace. `lab1` is in `default`, so CA would be installed +into `default` too. That mixes things in `default`, and with more clusters (phase 3) you'd get one +CA per cluster sitting side by side in `default`. + +**Options compared:** + +| Option | Verdict | +| --- | --- | +| CA in `default` (current Step 6) | works; poor hygiene, doesn't scale to more clusters | +| CA in its own namespace with a copy of the kubeconfig Secret | rejected: CAPI rewrites `lab1-kubeconfig` when it rotates the client cert, so the copy goes stale and CA silently loses the workload cluster | +| **One namespace per cluster** (`lab1` holds the CAPI objects, `InClusterIPPool`, kubeconfig Secret and CA) | preferred: the usual CAPI multi-cluster layout, one CA per workload cluster, RBAC and cleanup per namespace, matches phase 3 | + +**Cost.** Namespace is immutable on CAPI objects, so `lab1` has to be deleted and recreated. That +means about 15 minutes of provisioning plus reinstalling Cilium. `clusterctl move` is not an option, +since it's forbidden with cluster-api-operator. CAPMOX credentials come from the controller's +global Secret, so a new namespace needs no extra credentials. The cost grows once workloads and +autoscaler state exist on `lab1`, so do it before the Step 7 results matter, or when creating the +next cluster. + +**Changes:** + +- `configs/capi/lab1.yaml`: a `Namespace lab1` object and `metadata.namespace: lab1` on every + object, including `InClusterIPPool`. +- `configs/cluster-autoscaler/values.yaml`: `autoDiscovery.namespace: lab1`; install with `-n lab1` + (`helm uninstall ca -n default` first if it was installed in `default`). +- `ansible/group_vars/all.yml`: `inspector_workload_clusters` entry `{ name: lab1, namespace: lab1 }`. +- `docs/phase-1-proxmox-capi.md`: Steps 3, 5 and 6 (namespace in commands and log expectations, + `MachineDeployment/lab1/lab1-workers`). +- `.claude/agents/cluster-inspector.md`: the "Where things live" table says CA runs in `default`. +- `docs/decisions.md`: new row (namespace per workload cluster). + +**Run by the owner:** `kubectl delete -f configs/capi/lab1.yaml` (old namespace), wait until the +VMs are gone, `kubectl apply` the new manifest, Cilium `helm upgrade --install`, +`ansible-playbook mgmt.yml --tags inspector`, then the CA install into `lab1`. diff --git a/docs/phase-1-proxmox-capi.md b/docs/phase-1-proxmox-capi.md index 05766e4..73ca25f 100644 --- a/docs/phase-1-proxmox-capi.md +++ b/docs/phase-1-proxmox-capi.md @@ -12,7 +12,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro | --- | --- | --- | | Management cluster | VM 201 `lab-mgmt` (192.168.4.21) on `pv1-1`, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB | | `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM | -| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 4 GB RAM | +| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 2 GB RAM | ## Checklist @@ -107,7 +107,10 @@ only when a version changes, then redo these edits: (handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes. 3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin). 4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device. -5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 4096`. +5. **Worker size** — rename the worker template to `ProxmoxMachineTemplate/lab1-worker-2g` (and the MD + `infrastructureRef`): `numSockets: 1`, `numCores: 4`, `memoryMiB: 2048`. To resize later, add a + template with a new name and point the MD at it; CAPI then rolls the workers. Don't edit a + template in place. 6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it. ```bash @@ -157,7 +160,7 @@ metadata: cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0" cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5" capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores - capacity.cluster-autoscaler.kubernetes.io/memory: "4Gi" # memoryMiB 4096 + capacity.cluster-autoscaler.kubernetes.io/memory: "2Gi" # memoryMiB 2048 capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi" capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110" ``` @@ -183,20 +186,39 @@ Install into the **same namespace as the `lab1-kubeconfig` secret** (here `defau [configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml): ```bash -helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update -helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \ - -f configs/cluster-autoscaler/values.yaml +export KUBECONFIG=~/.kube/lab-mgmt.kubeconfig +helm upgrade --install ca cluster-autoscaler --repo https://kubernetes.github.io/autoscaler \ + --version 9.59.0 -n default -f configs/cluster-autoscaler/values.yaml ``` -The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5. +The values add `rbac.additionalRules` to read `proxmoxmachinetemplates`. The chart's ClusterRole +doesn't cover infrastructure templates, and without them the first main loop hangs (see +troubleshooting). Health check: the log shows `Starting main loop` followed by regular +`Autoscaler loop triggered` lines, and `cluster-autoscaler-status` on lab1 leaves `Initializing`. + +CA runs on the management cluster as Deployment `ca-clusterapi-cluster-autoscaler`. Its log must +show the node group; the line is logged at `v: 4` on every loop (`clusterapi_controller.go`, +CA 1.35.2): + +```bash +kubectl logs deploy/ca-clusterapi-cluster-autoscaler -n default --kubeconfig ~/.kube/lab-mgmt.kubeconfig \ + | grep 'discovered node group' +# discovered node group: MachineDeployment/default/lab1-workers (min: 0, max: 5, replicas: 1) +``` + +The status ConfigMap and the leader-election lock live on the workload cluster, not on the management +cluster: `kubectl get cm cluster-autoscaler-status -n kube-system --kubeconfig ~/.kube/lab1.kubeconfig` +(chart arg `--namespace=kube-system`, because `clusterAPIConfigMapsNamespace` is empty). ## Step 7 — scale tests -Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 1Gi each → -CPU-bound at 2 pods per 4-core / 4 GiB worker). +Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 512Mi each → +CPU-bound at 2 pods per 4-core / 2 GiB worker; a 2 GiB VM has ~1.78 GiB allocatable). - [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time. -- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear. +- [ ] Scale-down: `--replicas=0` → after `scale-down-delay-after-add` (2m) + `scale-down-unneeded-time` + (1m) Machines and VMs disappear. That's the aggressive lab profile in the CA values; per-MD overrides are + `cluster.x-k8s.io/autoscaling-options-{scaledownunneededtime,scaledownutilizationthreshold,…}` annotations. - [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting). - [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations. @@ -207,8 +229,12 @@ CPU-bound at 2 pods per 4-core / 4 GiB worker). | Symptom | Cause | Fix | | --- | --- | --- | | CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values | +| CA runs and discovers the node group but never scales; log stops at `waiting for cache sync on infrastructure resource`, `proxmoxmachinetemplates … forbidden` every ~40 s; status ConfigMap stuck on `Initializing` | chart 9.59.0 ClusterRole has no rule for `infrastructure.cluster.x-k8s.io`; CA's first main loop waits for a template informer that can never sync | `rbac.additionalRules` with get/list/watch on `proxmoxmachinetemplates` in the CA values, `helm upgrade` | | CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` | | MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` | +| scale-up adds one node too many (6 pods at 2 per node → 4 workers, second plan `3->4` about 70 s after `1->3`) | a new node registers with Cilium's `node.cilium.io/agent-not-ready:NoSchedule`; once registered it's no longer an "upcoming" template node, and CA sees pending pods that don't fit on it | `extraArgs.startup-taint: node.cilium.io/agent-not-ready` in the CA values (CA treats the node as still starting) | +| CA scales the MD up, new Machines stay without a VM; ProxmoxMachine `VMProvisionFailed: cannot reserve 4294967296B of memory on node pv1-1: …B available memory left` although `free` shows GBs available | CAPMOX v0.9.1 scheduler (`GetReservableMemoryBytes`): node total × `schedulerHints.memoryAdjustment` (default 100%) minus the **configured** memory of every non-template VM, **running or stopped**, and every container. Stopped VMs 100/1001/2001 hold 12 GiB | free the reservation (delete unused stopped VMs, or convert them to templates), or set `ProxmoxCluster.spec.schedulerHints.memoryAdjustment` > 100 (overcommit); keep the MD max-size within what the host can actually run | +| MD rollout never finishes and CA never scales down: CA logs `Node … will be skipped as it is currently under rollout`, new ProxmoxMachine `cannot reserve …B of memory` | deadlock. Default rollout surge (`maxSurge 1 / maxUnavailable 0`) needs a new VM first; CAPMOX has no reservable memory for it; CA 1.35 excludes every node of a rolling-out MD from scale-down (`ScaleDownNodeUpgradeProcessor`) | `spec.rollout.strategy.rollingUpdate: {maxSurge: 0, maxUnavailable: 1}` on the MD (delete an old worker, then create the new one), or free CAPMOX-reservable memory | | last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` | | `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` | | scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods | diff --git a/docs/sessions/2026-09-28-1007-capmox-memory-overcommit.md b/docs/sessions/2026-09-28-1007-capmox-memory-overcommit.md new file mode 100644 index 0000000..3760556 --- /dev/null +++ b/docs/sessions/2026-09-28-1007-capmox-memory-overcommit.md @@ -0,0 +1,64 @@ +# Session: CAPMOX memory accounting and overcommit on pv1-1 + +**Created:** 2026-09-28 10:07 CEST + +## Question + +> can we configure capmox so that it works with real available memory instead of calculation what +> stopped vms would use if somebody would start them? also can we configure memory overprovisioning +> on proxmox node? + +Context: during Step 7, new workers failed with `cannot reserve …B of memory on node pv1-1`. The +host had about 12.8 GiB actually available, but CAPMOX counted 168 MB. Later this deadlocked the +2 GiB worker rollout (see the phase-1 troubleshooting table). + +## What was checked (2026-09-28) + +- CAPMOX source for v0.9.1 (the latest tag) and `main`: + - `pkg/proxmox/goproxmox/api_client.go` `GetReservableMemoryBytes`; + - `internal/service/scheduler/vmscheduler.go`; + - `api/v1alpha2/proxmoxcluster_types.go` `SchedulerHints`; + - `api/v1alpha2/proxmoxmachine_types.go`; + - `internal/webhook/proxmoxcluster_webhook.go`. +- `pv1-1` (read-only): + - `qm list`, node memory status; + - KSM: `ksmtuned` active, `/sys/kernel/mm/ksm/run` = 0, `pages_sharing` = 0; + - 8 GiB swap; + - VM 103 has no `balloon:` line. + +## Findings + +- **No "real available memory" mode in any CAPMOX release.** `main` is identical to v0.9.1. The + reservable memory is `node total × memoryAdjustment/100` minus the configured `maxmem` of every + non-template VM (running or stopped) and every container. That's deliberate: real usage is + racy, because VMs grow into their configured memory later. Using free memory would need an + upstream change. +- **The only knob is `ProxmoxCluster.spec.schedulerHints.memoryAdjustment`** (percent, default 100): + - `> 100` allows bounded overcommit; + - `0` turns the memory check off completely (it then only checks request ≤ host total). + - The ProxmoxCluster update webhook only validates the control-plane endpoint, so the hint can + be changed on a live cluster. The scheduler reads it whenever it places a VM, and the stuck + ProxmoxMachine retries. +- **Sizing for `lab1`:** everything except the workers is 21 GiB configured (VMs 100, 102, 201, + 1001, 2001, 160549). 5 workers × 2 GiB brings that to 31 GiB, and the host has 29.16 GiB, so + `memoryAdjustment ≥ 107`. At 120 there's 35 GiB reservable, which fits max 5 workers with about + 4 GiB to spare. +- **Proxmox itself enforces no memory overcommit limit.** VMs can be started past physical RAM; CAPMOX + was the only admission control. On the PVE side, "overprovisioning" is about how the host copes: + - **KSM:** `ksmtuned` is active but only merges below 20% free (`KSM_THRES_COEF`). Identical + Ubuntu clones deduplicate well. It's currently idle because the host has about 44% free. + - **Host swap (8 GiB):** a safety net, but slow. + - **Ballooning** (`balloon: <min>` below `memory`, auto-balloon above 80% host use): not + exposed by CAPMOX v0.9.1 (`ProxmoxMachineSpec` has no balloon field). It's also a poor fit + for Kubernetes nodes: the guest keeps reporting full capacity while the balloon takes memory + away, so the kubelet schedules against memory that isn't there. + +## Open questions + +- ~~120 or 0?~~ The owner chose **160** (decision #17): 46.7 GiB reservable, about 17.7 GiB headroom with today's VMs. +- Is an upstream feature request worth filing (a free-memory mode, or excluding stopped VMs)? + +## Follow-ups + +- Done: `schedulerHints.memoryAdjustment: 160` on `ProxmoxCluster/lab1` in `configs/capi/lab1.yaml`, + decisions #17. Watch real host memory and KSM `pages_sharing` during scale tests.