kacerrandClaude Opus 5.5
81462e0e0f
Re-check of lab1 after Steps 4-5 (inspector on lab-mgmt, read-only gets on lab1): - Cluster/KCP/MD Available for 13h; no controller errors since bootstrap. - All six autoscaler annotations on lab1-workers (copied to the MachineSet). spec.replicas has no owner: kubectl-client-side-apply owns only clusterName/selector/template, so re-applying lab1.yaml won't fight cluster-autoscaler. - Nodes Ready on v1.35.9; Node/Machine providerIDs match. - Cilium, envoy and CoreDNS healthy. No PDBs in the cluster. Found: cilium-operator (replicas: 1) was scheduled onto the only worker. It is a non-DaemonSet kube-system pod without a PDB, so with CA's default skip-nodes-with-system-pods=true the last worker could never scale to zero (Step 7). The chart already tolerates the control-plane taint, so a nodeSelector on node-role.kubernetes.io/control-plane pins it there. - configs/cilium/values.yaml: operator.nodeSelector. - phase-1 doc: Step 5 ticked, Step 4 explains why replicas: 1 isn't enough, new troubleshooting row. - Session note extended with the re-check. Not applied yet: the owner runs helm upgrade with the new values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
75 lines
3.9 KiB
Markdown
75 lines
3.9 KiB
Markdown
# Session: lab1 created — first review by the cluster-inspector agent
|
||
|
||
**Created:** 2026-09-27 11:12 CEST
|
||
|
||
## Question
|
||
|
||
> lab cluster looks good, can you look at it with inspector agent ?
|
||
|
||
`lab1` had just been applied from `configs/capi/lab1.yaml` (phase-1 Step 3). Only the
|
||
management-side read-only kubeconfig existed, so the review covered what the management cluster sees.
|
||
|
||
## What was checked
|
||
|
||
`cluster-inspector` agent with `lab-mgmt-ro`: Cluster / KCP / MD / MS / Machines / ProxmoxMachines /
|
||
ProxmoxCluster conditions, IPAM claims, MD field managers, logs of all five CAPI controllers since
|
||
creation, Warning events in `default`. `clusterctl describe` failed (403 on CRDs).
|
||
|
||
## Findings
|
||
|
||
- CAPI / CAPMOX side healthy: VMs 102 (CP, 192.168.4.31) and 103 (worker, .32) cloned from
|
||
template 101 with VLAN 4, raw disk, planned sizes; control plane initialised and Available,
|
||
etcd/certs healthy, VIP 192.168.4.30 reachable; providerIDs match across Machine /
|
||
ProxmoxMachine / Node. First real use of the `capmox@pve!capi` token: no Proxmox API errors.
|
||
- Controller errors only in the first ~60 s after creation (no route to VIP, `lab1-kubeconfig`
|
||
not found, `vm not initialized`) — normal bootstrap noise, nothing after 09:01 UTC.
|
||
- MachineDeployment `replicas` defaulted to 1 and is owned by no field manager, so re-applying
|
||
`lab1.yaml` won't fight cluster-autoscaler.
|
||
- Both nodes NotReady (`cni plugin not initialized`) as of ~09:09 UTC → Cilium (Step 4) not
|
||
installed or not starting; the owner's "looks good" didn't hold for node readiness.
|
||
- IPAM pool 2/9 used; 1 CP + 5 workers + 1 surge = 7 fits.
|
||
- On k8s 1.35, `kubectl get ipaddresses` hits the built-in `networking.k8s.io` IPAddress, not
|
||
CAPI's — runbook and agent now use `ipaddresses.ipam.cluster.x-k8s.io`.
|
||
|
||
## Open questions
|
||
|
||
- Why the CNI wasn't up (not installed yet vs. image pull / egress from VLAN 4) — needs the lab1
|
||
read-only kubeconfig or the owner's check.
|
||
|
||
## Follow-ups
|
||
|
||
- `.claude/cluster-inspector/rbac.yaml`: CRD read added (for `clusterctl describe`); applied by the
|
||
next `ansible-playbook mgmt.yml --tags inspector`.
|
||
- Phase-1 checklist: Step 3 ticked; next Step 4 (Cilium), then Step 5 annotations in `lab1.yaml`.
|
||
|
||
## 2026-09-28 00:55 CEST — re-check after Steps 4–5
|
||
|
||
**What was checked:** the `cluster-inspector` agent with `lab-mgmt-ro` covered the MD annotations,
|
||
field managers, Cluster/KCP/MD conditions, IPAM and controller logs. For the workload side, Claude
|
||
ran read-only `kubectl get` with the owner's admin `~/.kube/lab1.kubeconfig`, because no `lab1-ro`
|
||
kubeconfig exists yet. The agent refuses admin kubeconfigs when permission is relayed through
|
||
another agent.
|
||
|
||
**Findings:**
|
||
|
||
- `lab1` has been stable for 13h. Cluster, KCP and MD are Available, and the controllers have
|
||
logged no errors since the ~09:00 UTC bootstrap noise.
|
||
- All six Step 5 annotations are on `lab1-workers` and copied to its MachineSet.
|
||
- Nothing owns `spec.replicas`: `kubectl-client-side-apply` owns only
|
||
clusterName/selector/template.
|
||
- Both nodes are Ready on v1.35.9, and Node providerIDs match the Machines.
|
||
- Cilium 2/2, envoy 2/2, operator 1/1 and CoreDNS 2/2 (both replicas on the CP) are healthy. No
|
||
pods are outside Running, and there are no PDBs anywhere.
|
||
- **`cilium-operator` runs on the worker** despite `operator.replicas: 1`. With CA's default
|
||
`skip-nodes-with-system-pods=true` and no PDB, it would block Step 7's scale-to-zero. Fixed in
|
||
`configs/cilium/values.yaml` with a `nodeSelector` on the control-plane role; the owner still
|
||
has to apply it with `helm upgrade`.
|
||
|
||
**Follow-ups:**
|
||
|
||
- Owner: `helm upgrade` Cilium with the new values, then confirm the operator runs on
|
||
`lab1-control-plane-*`.
|
||
- Owner: mint a `lab1-ro` kubeconfig with `.claude/cluster-inspector/make-kubeconfig.sh`, so that
|
||
workload checks can go through the agent.
|
||
- After Step 6: check that only CA / the scale subresource owns `spec.replicas`.
|