# Session: lab1 created — first review by the cluster-inspector agent

**Created:** 2026-09-27 11:12 CEST

## Question

> lab cluster looks good, can you look at it with inspector agent ?

`lab1` had just been applied from `configs/capi/lab1.yaml` (phase-1 Step 3). Only the
management-side read-only kubeconfig existed, so the review covered what the management cluster sees.

## What was checked

`cluster-inspector` agent with `lab-mgmt-ro`: Cluster / KCP / MD / MS / Machines / ProxmoxMachines /
ProxmoxCluster conditions, IPAM claims, MD field managers, logs of all five CAPI controllers since
creation, Warning events in `default`. `clusterctl describe` failed (403 on CRDs).

## Findings

- CAPI / CAPMOX side healthy: VMs 102 (CP, 192.168.4.31) and 103 (worker, .32) cloned from
  template 101 with VLAN 4, raw disk, planned sizes; control plane initialised and Available,
  etcd/certs healthy, VIP 192.168.4.30 reachable; providerIDs match across Machine /
  ProxmoxMachine / Node. First real use of the `capmox@pve!capi` token: no Proxmox API errors.
- Controller errors only in the first ~60 s after creation (no route to VIP, `lab1-kubeconfig`
  not found, `vm not initialized`) — normal bootstrap noise, nothing after 09:01 UTC.
- MachineDeployment `replicas` defaulted to 1 and is owned by no field manager, so re-applying
  `lab1.yaml` won't fight cluster-autoscaler.
- Both nodes NotReady (`cni plugin not initialized`) as of ~09:09 UTC → Cilium (Step 4) not
  installed or not starting; the owner's "looks good" didn't hold for node readiness.
- IPAM pool 2/9 used; 1 CP + 5 workers + 1 surge = 7 fits.
- On k8s 1.35, `kubectl get ipaddresses` hits the built-in `networking.k8s.io` IPAddress, not
  CAPI's — runbook and agent now use `ipaddresses.ipam.cluster.x-k8s.io`.

## Open questions

- Why the CNI wasn't up (not installed yet vs. image pull / egress from VLAN 4) — needs the lab1
  read-only kubeconfig or the owner's check.

## Follow-ups

- `.claude/cluster-inspector/rbac.yaml`: CRD read added (for `clusterctl describe`); applied by the
  next `ansible-playbook mgmt.yml --tags inspector`.
- Phase-1 checklist: Step 3 ticked; next Step 4 (Cilium), then Step 5 annotations in `lab1.yaml`.

## 2026-09-28 00:55 CEST — re-check after Steps 4–5

**What was checked:** the `cluster-inspector` agent with `lab-mgmt-ro` covered the MD annotations,
field managers, Cluster/KCP/MD conditions, IPAM and controller logs. For the workload side, Claude
ran read-only `kubectl get` with the owner's admin `~/.kube/lab1.kubeconfig`, because no `lab1-ro`
kubeconfig exists yet. The agent refuses admin kubeconfigs when permission is relayed through
another agent.

**Findings:**

- `lab1` has been stable for 13h. Cluster, KCP and MD are Available, and the controllers have
  logged no errors since the ~09:00 UTC bootstrap noise.
- All six Step 5 annotations are on `lab1-workers` and copied to its MachineSet.
- Nothing owns `spec.replicas`: `kubectl-client-side-apply` owns only
  clusterName/selector/template.
- Both nodes are Ready on v1.35.9, and Node providerIDs match the Machines.
- Cilium 2/2, envoy 2/2, operator 1/1 and CoreDNS 2/2 (both replicas on the CP) are healthy. No
  pods are outside Running, and there are no PDBs anywhere.
- **`cilium-operator` runs on the worker** despite `operator.replicas: 1`. With CA's default
  `skip-nodes-with-system-pods=true` and no PDB, it would block Step 7's scale-to-zero. Fixed in
  `configs/cilium/values.yaml` with a `nodeSelector` on the control-plane role; the owner still
  has to apply it with `helm upgrade`.

**Follow-ups:**

- Owner: `helm upgrade` Cilium with the new values, then confirm the operator runs on
  `lab1-control-plane-*`.
- Owner: mint a `lab1-ro` kubeconfig with `.claude/cluster-inspector/make-kubeconfig.sh`, so that
  workload checks can go through the agent.
- After Step 6: check that only CA / the scale subresource owns `spec.replicas`.