lab1: pin cilium-operator to the control plane; Step 5 verified

Re-check of lab1 after Steps 4-5 (inspector on lab-mgmt, read-only gets
on lab1):
- Cluster/KCP/MD Available for 13h; no controller errors since bootstrap.
- All six autoscaler annotations on lab1-workers (copied to the
  MachineSet). spec.replicas has no owner: kubectl-client-side-apply owns
  only clusterName/selector/template, so re-applying lab1.yaml won't
  fight cluster-autoscaler.
- Nodes Ready on v1.35.9; Node/Machine providerIDs match.
- Cilium, envoy and CoreDNS healthy. No PDBs in the cluster.

Found: cilium-operator (replicas: 1) was scheduled onto the only worker.
It is a non-DaemonSet kube-system pod without a PDB, so with CA's default
skip-nodes-with-system-pods=true the last worker could never scale to
zero (Step 7). The chart already tolerates the control-plane taint, so a
nodeSelector on node-role.kubernetes.io/control-plane pins it there.

- configs/cilium/values.yaml: operator.nodeSelector.
- phase-1 doc: Step 5 ticked, Step 4 explains why replicas: 1 isn't
  enough, new troubleshooting row.
- Session note extended with the re-check.

Not applied yet: the owner runs helm upgrade with the new values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-28 09:23:41 +02:00
co-authored by Claude Opus 5.5
parent 49c548e296
commit 81462e0e0f
3 changed files with 39 additions and 3 deletions
+6 -3
View File
@@ -21,7 +21,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
- [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers)
- [x] Step 3 — workload cluster `lab1` (VMs 102/103, `.31`/`.32`, VIP `.30`)
- [x] Step 4 — CNI (Cilium 1.20.2, `configs/cilium/values.yaml`)
- [ ] Step 5 — MachineDeployment autoscaling annotations
- [x] Step 5 — MachineDeployment autoscaling annotations
- [ ] Step 6 — cluster-autoscaler
- [ ] Step 7 — scale tests
@@ -142,8 +142,10 @@ kubectl -n kube-system get pods -o wide # cilium DaemonSet 2/
```
- `ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default.
- `operator.replicas: 1`: the default 2 replicas with node anti-affinity put one on a worker, and a
kube-system pod without a PDB blocks scale-down to zero.
- `operator.replicas: 1` + `operator.nodeSelector` on `node-role.kubernetes.io/control-plane`: the
default 2 replicas with node anti-affinity put one on a worker, and a kube-system pod without a
PDB blocks scale-down to zero. `replicas: 1` alone isn't enough: the single replica was scheduled
onto the worker (the chart tolerates the control-plane taint, so the selector pins it there).
- kube-proxy stays (kubeadm default); no kube-proxy replacement in this phase.
## Step 5 — autoscaling annotations
@@ -208,6 +210,7 @@ CPU-bound at 2 pods per 4-core / 4 GiB worker).
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
| `cilium-operator` runs on a worker despite `operator.replicas: 1` | chart tolerates the CP taint but doesn't prefer the CP; the scheduler picked the worker | `operator.nodeSelector: {node-role.kubernetes.io/control-plane: ""}` in `configs/cilium/values.yaml`, `helm upgrade` |
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |