kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

lab1: pin cilium-operator to the control plane; Step 5 verified

Browse Source 
Re-check of lab1 after Steps 4-5 (inspector on lab-mgmt, read-only gets
on lab1):
- Cluster/KCP/MD Available for 13h; no controller errors since bootstrap.
- All six autoscaler annotations on lab1-workers (copied to the
  MachineSet). spec.replicas has no owner: kubectl-client-side-apply owns
  only clusterName/selector/template, so re-applying lab1.yaml won't
  fight cluster-autoscaler.
- Nodes Ready on v1.35.9; Node/Machine providerIDs match.
- Cilium, envoy and CoreDNS healthy. No PDBs in the cluster.

Found: cilium-operator (replicas: 1) was scheduled onto the only worker.
It is a non-DaemonSet kube-system pod without a PDB, so with CA's default
skip-nodes-with-system-pods=true the last worker could never scale to
zero (Step 7). The chart already tolerates the control-plane taint, so a
nodeSelector on node-role.kubernetes.io/control-plane pins it there.

- configs/cilium/values.yaml: operator.nodeSelector.
- phase-1 doc: Step 5 ticked, Step 4 explains why replicas: 1 isn't
  enough, new troubleshooting row.
- Session note extended with the re-check.

Not applied yet: the owner runs helm upgrade with the new values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-28 09:23:41 +02:00
co-authored by Claude Opus 5.5
parent 49c548e296
commit 81462e0e0f
3 changed files with 39 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sessions/2026-09-27-1112-lab1-first-review.md
+31
View File
@@ -41,3 +41,34 @@ creation, Warning events in `default`. `clusterctl describe` failed (403 on CRDs
- `.claude/cluster-inspector/rbac.yaml`: CRD read added (for `clusterctl describe`); applied by the
next `ansible-playbook mgmt.yml --tags inspector`.
- Phase-1 checklist: Step 3 ticked; next Step 4 (Cilium), then Step 5 annotations in `lab1.yaml`.
## 2026-09-28 00:55 CEST — re-check after Steps 4–5
**What was checked:** the `cluster-inspector` agent with `lab-mgmt-ro` covered the MD annotations,
field managers, Cluster/KCP/MD conditions, IPAM and controller logs. For the workload side, Claude
ran read-only `kubectl get` with the owner's admin `~/.kube/lab1.kubeconfig`, because no `lab1-ro`
kubeconfig exists yet. The agent refuses admin kubeconfigs when permission is relayed through
another agent.
**Findings:**
- `lab1` has been stable for 13h. Cluster, KCP and MD are Available, and the controllers have
logged no errors since the ~09:00 UTC bootstrap noise.
- All six Step 5 annotations are on `lab1-workers` and copied to its MachineSet.
- Nothing owns `spec.replicas`: `kubectl-client-side-apply` owns only
clusterName/selector/template.
- Both nodes are Ready on v1.35.9, and Node providerIDs match the Machines.
- Cilium 2/2, envoy 2/2, operator 1/1 and CoreDNS 2/2 (both replicas on the CP) are healthy. No
pods are outside Running, and there are no PDBs anywhere.
- **`cilium-operator` runs on the worker** despite `operator.replicas: 1`. With CA's default
`skip-nodes-with-system-pods=true` and no PDB, it would block Step 7's scale-to-zero. Fixed in
`configs/cilium/values.yaml` with a `nodeSelector` on the control-plane role; the owner still
has to apply it with `helm upgrade`.
**Follow-ups:**
- Owner: `helm upgrade` Cilium with the new values, then confirm the operator runs on
`lab1-control-plane-*`.
- Owner: mint a `lab1-ro` kubeconfig with `.claude/cluster-inspector/make-kubeconfig.sh`, so that
workload checks can go through the agent.
- After Step 6: check that only CA / the scale subresource owns `spec.replicas`.