kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
cluster-autoscaler-with-capi/docs/future-tasks.md
T
kacerrandClaude Opus 5.5 6ef55778ff lab1 workers on both PVE nodes: per-node template + MD, balanced CA 
PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage:
- template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and
  lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50;
  lab1-workers renamed for symmetry (decision #18, #7 superseded)
- CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels
  with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19).
  Scale test: 16 pods -> 4 + 4 across both nodes.
- runbook, image-builder copy procedure, fixes tables, future task to report
  three CA clusterapi bugs upstream, lab facts, changelog, execution log

Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:07:17 +02:00

4.6 KiB
Raw Permalink Blame History

Future tasks

Deferred work, newest first. When a task is picked up it gets a plan in docs/plans/; then tick it here and link the plan.

- [ ] Report three cluster-autoscaler clusterapi-provider bugs upstream

Added: 2026-09-30 09:28 CEST · Why deferred: worked around in config (decision #19); reporting is not blocking.

Problem. Three bugs in CA 1.35.2 cloudprovider/clusterapi, all still on master and 1.36.1 as of 2026-09-29, found in the pv1-2 execution log:

  1. Uncached discovery.
    • clusterapi_provider.go:195 builds a memory.NewMemCacheClient, but passes the raw discovery client to the controller (:209).
    • getKindPreferredVersion therefore does 3 GETs per MD at 0 replicas on every NodeGroups() call, about 15× per loop. The loop took 18 s at 5 QPS.
  2. Labels() reads the wrong path. It reads spec.template.spec.metadata.labels (clusterapi_unstructured.go:207), which doesn't exist in CAPI v1beta2, so MD template labels never reach the from-zero template node. The unit tests use the same wrong path.
  3. Empty os/arch labels. For an infra template without status, as with CAPMOX, InstanceSystemInfo() returns an empty struct. The template node then gets kubernetes.io/os="" and arch="", which override the generic linux/amd64.

Options:

  • Issues only.
  • Issues plus small PRs: pass the cached client, fix the path, skip empty nodeInfo fields.
  • Carry a patched CA image: rejected, versions must stay upstream and pinned.

Cost. About 1–2 h for issues with source references. PRs would add the CLA and a test for each.

Changes when done:

  • Link the issues/PRs in docs/phase-1-proxmox-capi.md troubleshooting.
  • Once fixed in a 1.35.x patch release: bump CA in docs/versions.md, then revisit scan-interval and the capacity…/labels annotation.

Run by the owner: nothing on the lab. Filing on GitHub is the owner's call, since it goes out under their account.

- [ ] Namespace per workload cluster (lab1 out of default)

Added: 2026-09-28 09:29 CEST · Why deferred: keep phase 1 moving; default works for now.

Problem. cluster-autoscaler has to run in the same namespace as the Cluster. The chart mounts clusterAPIKubeconfigSecret as a Secret volume, which only works within the pod's namespace, and lab1-kubeconfig lives in the Cluster's namespace. lab1 is in default, so CA would be installed into default too. That mixes things in default, and with more clusters (phase 3) you'd get one CA per cluster sitting side by side in default.

Options compared:

Option Verdict
CA in default (current Step 6) works; poor hygiene, doesn't scale to more clusters
CA in its own namespace with a copy of the kubeconfig Secret rejected: CAPI rewrites lab1-kubeconfig when it rotates the client cert, so the copy goes stale and CA silently loses the workload cluster
One namespace per cluster (lab1 holds the CAPI objects, InClusterIPPool, kubeconfig Secret and CA) preferred: the usual CAPI multi-cluster layout, one CA per workload cluster, RBAC and cleanup per namespace, matches phase 3

Cost. Namespace is immutable on CAPI objects, so lab1 has to be deleted and recreated. That means about 15 minutes of provisioning plus reinstalling Cilium. clusterctl move is not an option, since it's forbidden with cluster-api-operator. CAPMOX credentials come from the controller's global Secret, so a new namespace needs no extra credentials. The cost grows once workloads and autoscaler state exist on lab1, so do it before the Step 7 results matter, or when creating the next cluster.

Changes:

  • configs/capi/lab1.yaml: a Namespace lab1 object and metadata.namespace: lab1 on every object, including InClusterIPPool.
  • configs/cluster-autoscaler/values.yaml: autoDiscovery.namespace: lab1; install with -n lab1 (helm uninstall ca -n default first if it was installed in default).
  • ansible/group_vars/all.yml: inspector_workload_clusters entry { name: lab1, namespace: lab1 }.
  • docs/phase-1-proxmox-capi.md: Steps 3, 5 and 6 (namespace in commands and log expectations, MachineDeployment/lab1/lab1-workers-pv1-{1,2}).
  • .claude/agents/cluster-inspector.md: the "Where things live" table says CA runs in default.
  • docs/decisions.md: new row (namespace per workload cluster).

Run by the owner: kubectl delete -f configs/capi/lab1.yaml (old namespace), wait until the VMs are gone, kubectl apply the new manifest, Cilium helm upgrade --install, ansible-playbook mgmt.yml --tags inspector, then the CA install into lab1.

Reference in New Issue View Git Blame Copy Permalink