kacerrandClaude Opus 5.5
6ef55778ff
PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage: - template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50; lab1-workers renamed for symmetry (decision #18, #7 superseded) - CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19). Scale test: 16 pods -> 4 + 4 across both nodes. - runbook, image-builder copy procedure, fixes tables, future task to report three CA clusterapi bugs upstream, lab facts, changelog, execution log Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
80 lines
4.6 KiB
Markdown
80 lines
4.6 KiB
Markdown
# Future tasks
|
||
|
||
Deferred work, newest first. When a task is picked up it gets a plan in `docs/plans/`; then tick
|
||
it here and link the plan.
|
||
|
||
## - [ ] Report three cluster-autoscaler clusterapi-provider bugs upstream
|
||
|
||
**Added:** 2026-09-30 09:28 CEST · **Why deferred:** worked around in config (decision #19);
|
||
reporting is not blocking.
|
||
|
||
**Problem.** Three bugs in CA 1.35.2 `cloudprovider/clusterapi`, all still on master and 1.36.1 as
|
||
of 2026-09-29, found in [the pv1-2 execution log](plans-executions/2026-09-29-2315-pv1-2-workers.md):
|
||
1. **Uncached discovery.**
|
||
- `clusterapi_provider.go:195` builds a `memory.NewMemCacheClient`, but passes the raw
|
||
discovery client to the controller (`:209`).
|
||
- `getKindPreferredVersion` therefore does 3 GETs per MD at 0 replicas on every `NodeGroups()`
|
||
call, about 15× per loop. The loop took 18 s at 5 QPS.
|
||
2. **`Labels()` reads the wrong path.** It reads `spec.template.spec.metadata.labels`
|
||
(`clusterapi_unstructured.go:207`), which doesn't exist in CAPI v1beta2, so MD template labels
|
||
never reach the from-zero template node. The unit tests use the same wrong path.
|
||
3. **Empty os/arch labels.** For an infra template without `status`, as with CAPMOX,
|
||
`InstanceSystemInfo()` returns an empty struct. The template node then gets
|
||
`kubernetes.io/os=""` and `arch=""`, which override the generic `linux`/`amd64`.
|
||
|
||
**Options:**
|
||
- Issues only.
|
||
- Issues plus small PRs: pass the cached client, fix the path, skip empty nodeInfo fields.
|
||
- Carry a patched CA image: rejected, versions must stay upstream and pinned.
|
||
|
||
**Cost.** About 1–2 h for issues with source references. PRs would add the CLA and a test for each.
|
||
|
||
**Changes when done:**
|
||
- Link the issues/PRs in `docs/phase-1-proxmox-capi.md` troubleshooting.
|
||
- Once fixed in a 1.35.x patch release: bump CA in `docs/versions.md`, then revisit
|
||
`scan-interval` and the `capacity…/labels` annotation.
|
||
|
||
**Run by the owner:** nothing on the lab. Filing on GitHub is the owner's call, since it goes out
|
||
under their account.
|
||
|
||
## - [ ] Namespace per workload cluster (`lab1` out of `default`)
|
||
|
||
**Added:** 2026-09-28 09:29 CEST · **Why deferred:** keep phase 1 moving; `default` works for now.
|
||
|
||
**Problem.** cluster-autoscaler has to run in the same namespace as the Cluster. The chart mounts
|
||
`clusterAPIKubeconfigSecret` as a Secret volume, which only works within the pod's namespace, and
|
||
`lab1-kubeconfig` lives in the Cluster's namespace. `lab1` is in `default`, so CA would be installed
|
||
into `default` too. That mixes things in `default`, and with more clusters (phase 3) you'd get one
|
||
CA per cluster sitting side by side in `default`.
|
||
|
||
**Options compared:**
|
||
|
||
| Option | Verdict |
|
||
| --- | --- |
|
||
| CA in `default` (current Step 6) | works; poor hygiene, doesn't scale to more clusters |
|
||
| CA in its own namespace with a copy of the kubeconfig Secret | rejected: CAPI rewrites `lab1-kubeconfig` when it rotates the client cert, so the copy goes stale and CA silently loses the workload cluster |
|
||
| **One namespace per cluster** (`lab1` holds the CAPI objects, `InClusterIPPool`, kubeconfig Secret and CA) | preferred: the usual CAPI multi-cluster layout, one CA per workload cluster, RBAC and cleanup per namespace, matches phase 3 |
|
||
|
||
**Cost.** Namespace is immutable on CAPI objects, so `lab1` has to be deleted and recreated. That
|
||
means about 15 minutes of provisioning plus reinstalling Cilium. `clusterctl move` is not an option,
|
||
since it's forbidden with cluster-api-operator. CAPMOX credentials come from the controller's
|
||
global Secret, so a new namespace needs no extra credentials. The cost grows once workloads and
|
||
autoscaler state exist on `lab1`, so do it before the Step 7 results matter, or when creating the
|
||
next cluster.
|
||
|
||
**Changes:**
|
||
|
||
- `configs/capi/lab1.yaml`: a `Namespace lab1` object and `metadata.namespace: lab1` on every
|
||
object, including `InClusterIPPool`.
|
||
- `configs/cluster-autoscaler/values.yaml`: `autoDiscovery.namespace: lab1`; install with `-n lab1`
|
||
(`helm uninstall ca -n default` first if it was installed in `default`).
|
||
- `ansible/group_vars/all.yml`: `inspector_workload_clusters` entry `{ name: lab1, namespace: lab1 }`.
|
||
- `docs/phase-1-proxmox-capi.md`: Steps 3, 5 and 6 (namespace in commands and log expectations,
|
||
`MachineDeployment/lab1/lab1-workers-pv1-{1,2}`).
|
||
- `.claude/agents/cluster-inspector.md`: the "Where things live" table says CA runs in `default`.
|
||
- `docs/decisions.md`: new row (namespace per workload cluster).
|
||
|
||
**Run by the owner:** `kubectl delete -f configs/capi/lab1.yaml` (old namespace), wait until the
|
||
VMs are gone, `kubectl apply` the new manifest, Cilium `helm upgrade --install`,
|
||
`ansible-playbook mgmt.yml --tags inspector`, then the CA install into `lab1`.
|