cluster-autoscaler-with-capi/docs/sessions/2026-09-27-1112-lab1-first-review.md

Session: lab1 created — first review by the cluster-inspector agent

Created: 2026-09-27 11:12 CEST

Question

lab cluster looks good, can you look at it with inspector agent ?

lab1 had just been applied from configs/capi/lab1.yaml (phase-1 Step 3). Only the management-side read-only kubeconfig existed, so the review covered what the management cluster sees.

What was checked

cluster-inspector agent with lab-mgmt-ro: Cluster / KCP / MD / MS / Machines / ProxmoxMachines / ProxmoxCluster conditions, IPAM claims, MD field managers, logs of all five CAPI controllers since creation, Warning events in default. clusterctl describe failed (403 on CRDs).

Findings

• CAPI / CAPMOX side healthy: VMs 102 (CP, 192.168.4.31) and 103 (worker, .32) cloned from template 101 with VLAN 4, raw disk, planned sizes; control plane initialised and Available, etcd/certs healthy, VIP 192.168.4.30 reachable; providerIDs match across Machine / ProxmoxMachine / Node. First real use of the capmox@pve!capi token: no Proxmox API errors.
• Controller errors only in the first ~60 s after creation (no route to VIP, lab1-kubeconfig not found, vm not initialized) — normal bootstrap noise, nothing after 09:01 UTC.
• MachineDeployment replicas defaulted to 1 and is owned by no field manager, so re-applying lab1.yaml won't fight cluster-autoscaler.
• Both nodes NotReady (cni plugin not initialized) as of ~09:09 UTC → Cilium (Step 4) not installed or not starting; the owner's "looks good" didn't hold for node readiness.
• IPAM pool 2/9 used; 1 CP + 5 workers + 1 surge = 7 fits.
• On k8s 1.35, kubectl get ipaddresses hits the built-in networking.k8s.io IPAddress, not CAPI's — runbook and agent now use ipaddresses.ipam.cluster.x-k8s.io.

Open questions

• Why the CNI wasn't up (not installed yet vs. image pull / egress from VLAN 4) — needs the lab1 read-only kubeconfig or the owner's check.

Follow-ups

• .claude/cluster-inspector/rbac.yaml: CRD read added (for clusterctl describe); applied by the next ansible-playbook mgmt.yml --tags inspector.
• Phase-1 checklist: Step 3 ticked; next Step 4 (Cilium), then Step 5 annotations in lab1.yaml.

2026-09-28 00:55 CEST — re-check after Steps 4–5

What was checked: the cluster-inspector agent with lab-mgmt-ro covered the MD annotations, field managers, Cluster/KCP/MD conditions, IPAM and controller logs. For the workload side, Claude ran read-only kubectl get with the owner's admin ~/.kube/lab1.kubeconfig, because no lab1-ro kubeconfig exists yet. The agent refuses admin kubeconfigs when permission is relayed through another agent.

Findings:

• lab1 has been stable for 13h. Cluster, KCP and MD are Available, and the controllers have logged no errors since the ~09:00 UTC bootstrap noise.
• All six Step 5 annotations are on lab1-workers and copied to its MachineSet.
• Nothing owns spec.replicas: kubectl-client-side-apply owns only clusterName/selector/template.
• Both nodes are Ready on v1.35.9, and Node providerIDs match the Machines.
• Cilium 2/2, envoy 2/2, operator 1/1 and CoreDNS 2/2 (both replicas on the CP) are healthy. No pods are outside Running, and there are no PDBs anywhere.
• cilium-operator runs on the worker despite operator.replicas: 1. With CA's default skip-nodes-with-system-pods=true and no PDB, it would block Step 7's scale-to-zero. Fixed in configs/cilium/values.yaml with a nodeSelector on the control-plane role; the owner still has to apply it with helm upgrade.

Follow-ups:

• Owner: helm upgrade Cilium with the new values, then confirm the operator runs on lab1-control-plane-*.
• Owner: mint a lab1-ro kubeconfig with .claude/cluster-inspector/make-kubeconfig.sh, so that workload checks can go through the agent.
• After Step 6: check that only CA / the scale subresource owns spec.replicas.