From 81462e0e0ff54b5045b474f0c753b90d0fa3044f Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Mon, 28 Sep 2026 09:23:41 +0200 Subject: [PATCH] lab1: pin cilium-operator to the control plane; Step 5 verified Re-check of lab1 after Steps 4-5 (inspector on lab-mgmt, read-only gets on lab1): - Cluster/KCP/MD Available for 13h; no controller errors since bootstrap. - All six autoscaler annotations on lab1-workers (copied to the MachineSet). spec.replicas has no owner: kubectl-client-side-apply owns only clusterName/selector/template, so re-applying lab1.yaml won't fight cluster-autoscaler. - Nodes Ready on v1.35.9; Node/Machine providerIDs match. - Cilium, envoy and CoreDNS healthy. No PDBs in the cluster. Found: cilium-operator (replicas: 1) was scheduled onto the only worker. It is a non-DaemonSet kube-system pod without a PDB, so with CA's default skip-nodes-with-system-pods=true the last worker could never scale to zero (Step 7). The chart already tolerates the control-plane taint, so a nodeSelector on node-role.kubernetes.io/control-plane pins it there. - configs/cilium/values.yaml: operator.nodeSelector. - phase-1 doc: Step 5 ticked, Step 4 explains why replicas: 1 isn't enough, new troubleshooting row. - Session note extended with the re-check. Not applied yet: the owner runs helm upgrade with the new values. Co-Authored-By: Claude Opus 5.5 --- configs/cilium/values.yaml | 2 ++ docs/phase-1-proxmox-capi.md | 9 ++++-- .../2026-09-27-1112-lab1-first-review.md | 31 +++++++++++++++++++ 3 files changed, 39 insertions(+), 3 deletions(-) diff --git a/configs/cilium/values.yaml b/configs/cilium/values.yaml index 9d66a38..d2e855f 100644 --- a/configs/cilium/values.yaml +++ b/configs/cilium/values.yaml @@ -4,3 +4,5 @@ ipam: mode: kubernetes # use the node podCIDRs from Cluster.spec.clusterNetwork (10.244.0.0/16), not 10.0.0.0/8 operator: replicas: 1 # default 2 with node anti-affinity would pin one to a worker and block scale-to-zero + nodeSelector: # replicas: 1 alone still let the scheduler pick the worker; chart tolerates the CP taint + node-role.kubernetes.io/control-plane: "" diff --git a/docs/phase-1-proxmox-capi.md b/docs/phase-1-proxmox-capi.md index e6a846f..05766e4 100644 --- a/docs/phase-1-proxmox-capi.md +++ b/docs/phase-1-proxmox-capi.md @@ -21,7 +21,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro - [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers) - [x] Step 3 — workload cluster `lab1` (VMs 102/103, `.31`/`.32`, VIP `.30`) - [x] Step 4 — CNI (Cilium 1.20.2, `configs/cilium/values.yaml`) -- [ ] Step 5 — MachineDeployment autoscaling annotations +- [x] Step 5 — MachineDeployment autoscaling annotations - [ ] Step 6 — cluster-autoscaler - [ ] Step 7 — scale tests @@ -142,8 +142,10 @@ kubectl -n kube-system get pods -o wide # cilium DaemonSet 2/ ``` - `ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default. -- `operator.replicas: 1`: the default 2 replicas with node anti-affinity put one on a worker, and a - kube-system pod without a PDB blocks scale-down to zero. +- `operator.replicas: 1` + `operator.nodeSelector` on `node-role.kubernetes.io/control-plane`: the + default 2 replicas with node anti-affinity put one on a worker, and a kube-system pod without a + PDB blocks scale-down to zero. `replicas: 1` alone isn't enough: the single replica was scheduled + onto the worker (the chart tolerates the control-plane taint, so the selector pins it there). - kube-proxy stays (kubeadm default); no kube-proxy replacement in this phase. ## Step 5 — autoscaling annotations @@ -208,6 +210,7 @@ CPU-bound at 2 pods per 4-core / 4 GiB worker). | CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` | | MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` | | last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` | +| `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` | | scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods | | clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` | | ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range | diff --git a/docs/sessions/2026-09-27-1112-lab1-first-review.md b/docs/sessions/2026-09-27-1112-lab1-first-review.md index aecb02f..0385258 100644 --- a/docs/sessions/2026-09-27-1112-lab1-first-review.md +++ b/docs/sessions/2026-09-27-1112-lab1-first-review.md @@ -41,3 +41,34 @@ creation, Warning events in `default`. `clusterctl describe` failed (403 on CRDs - `.claude/cluster-inspector/rbac.yaml`: CRD read added (for `clusterctl describe`); applied by the next `ansible-playbook mgmt.yml --tags inspector`. - Phase-1 checklist: Step 3 ticked; next Step 4 (Cilium), then Step 5 annotations in `lab1.yaml`. + +## 2026-09-28 00:55 CEST — re-check after Steps 4–5 + +**What was checked:** the `cluster-inspector` agent with `lab-mgmt-ro` covered the MD annotations, +field managers, Cluster/KCP/MD conditions, IPAM and controller logs. For the workload side, Claude +ran read-only `kubectl get` with the owner's admin `~/.kube/lab1.kubeconfig`, because no `lab1-ro` +kubeconfig exists yet. The agent refuses admin kubeconfigs when permission is relayed through +another agent. + +**Findings:** + +- `lab1` has been stable for 13h. Cluster, KCP and MD are Available, and the controllers have + logged no errors since the ~09:00 UTC bootstrap noise. +- All six Step 5 annotations are on `lab1-workers` and copied to its MachineSet. +- Nothing owns `spec.replicas`: `kubectl-client-side-apply` owns only + clusterName/selector/template. +- Both nodes are Ready on v1.35.9, and Node providerIDs match the Machines. +- Cilium 2/2, envoy 2/2, operator 1/1 and CoreDNS 2/2 (both replicas on the CP) are healthy. No + pods are outside Running, and there are no PDBs anywhere. +- **`cilium-operator` runs on the worker** despite `operator.replicas: 1`. With CA's default + `skip-nodes-with-system-pods=true` and no PDB, it would block Step 7's scale-to-zero. Fixed in + `configs/cilium/values.yaml` with a `nodeSelector` on the control-plane role; the owner still + has to apply it with `helm upgrade`. + +**Follow-ups:** + +- Owner: `helm upgrade` Cilium with the new values, then confirm the operator runs on + `lab1-control-plane-*`. +- Owner: mint a `lab1-ro` kubeconfig with `.claude/cluster-inspector/make-kubeconfig.sh`, so that + workload checks can go through the agent. +- After Step 6: check that only CA / the scale subresource owns `spec.replicas`.