lab1 provisioned: fix IPAddress kubectl name, inspector CRD read, review notes

- runbook + inspector agent: ipaddresses.ipam.cluster.x-k8s.io (plain `ipaddresses` is the built-in networking.k8s.io type on k8s 1.35) - inspector RBAC: get/list CRDs so clusterctl describe works read-only - phase-1 Step 3 ticked; session note with the first lab1 review Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 11:14:11 +02:00
parent 57f58f1fc5
commit 6d552ebf33
4 changed files with 52 additions and 3 deletions
@@ -24,7 +24,8 @@ Triage order:
 2. Objects + conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`,
   `kubectl describe` the first non-Ready object.
 3. Controller logs: `capi-system/capi-controller-manager`, `capmox-system/capmox-controller-manager`,
-   `capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses -A`).
+   `capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io -A`; plain
+   `ipaddresses` is the built-in networking.k8s.io type).
 4. Node side: Node vs Machine providerID match
   (`kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID`), kubelet/cloud-init
   status if SSH access is given.
@@ -33,6 +33,10 @@ rules:
  - apiGroups: [""]
    resources: ["nodes", "namespaces"]
    verbs: ["get", "list", "watch"]
+  # clusterctl describe checks the CAPI CRD storage version
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["get", "list", "watch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
@@ -19,7 +19,7 @@ Versions: see [versions.md](versions.md). Template build: see [image-builder-pro
 - [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
 - [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
 - [x] Step 2 — management cluster (`ansible/mgmt.yml`: k3s + cluster-api-operator + providers)
- [ ] Step 3 — workload cluster `lab1`
+- [x] Step 3 — workload cluster `lab1` (VMs 102/103, `.31`/`.32`, VIP `.30`)
 - [ ] Step 4 — CNI (Cilium)
 - [ ] Step 5 — MachineDeployment autoscaling annotations
 - [ ] Step 6 — cluster-autoscaler
@@ -117,7 +117,8 @@ kubectl get proxmoxmachines,machines -w
 clusterctl get kubeconfig lab1 > lab1.kubeconfig
 ```

-Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses`). providerID is set by
+Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io`;
+plain `ipaddresses` resolves to the built-in `networking.k8s.io` type on k8s ≥ 1.33). providerID is set by
 the CAPMOX template as `proxmox://{{ ds.meta_data.instance_id }}` — don't touch it, the autoscaler
 matches Nodes to Machines by it.

@@ -0,0 +1,43 @@
+# Session: lab1 created — first review by the cluster-inspector agent
+
+**Created:** 2026-09-27 11:12 CEST
+
+## Question
+
+> lab cluster looks good, can you look at it with inspector agent ?
+
+`lab1` had just been applied from `configs/capi/lab1.yaml` (phase-1 Step 3). Only the
+management-side read-only kubeconfig existed, so the review covered what the management cluster sees.
+
+## What was checked
+
+`cluster-inspector` agent with `lab-mgmt-ro`: Cluster / KCP / MD / MS / Machines / ProxmoxMachines /
+ProxmoxCluster conditions, IPAM claims, MD field managers, logs of all five CAPI controllers since
+creation, Warning events in `default`. `clusterctl describe` failed (403 on CRDs).
+
+## Findings
+
+- CAPI / CAPMOX side healthy: VMs 102 (CP, 192.168.4.31) and 103 (worker, .32) cloned from
+  template 101 with VLAN 4, raw disk, planned sizes; control plane initialised and Available,
+  etcd/certs healthy, VIP 192.168.4.30 reachable; providerIDs match across Machine /
+  ProxmoxMachine / Node. First real use of the `capmox@pve!capi` token: no Proxmox API errors.
+- Controller errors only in the first ~60 s after creation (no route to VIP, `lab1-kubeconfig`
+  not found, `vm not initialized`) — normal bootstrap noise, nothing after 09:01 UTC.
+- MachineDeployment `replicas` defaulted to 1 and is owned by no field manager, so re-applying
+  `lab1.yaml` won't fight cluster-autoscaler.
+- Both nodes NotReady (`cni plugin not initialized`) as of ~09:09 UTC → Cilium (Step 4) not
+  installed or not starting; the owner's "looks good" didn't hold for node readiness.
+- IPAM pool 2/9 used; 1 CP + 5 workers + 1 surge = 7 fits.
+- On k8s 1.35, `kubectl get ipaddresses` hits the built-in `networking.k8s.io` IPAddress, not
+  CAPI's — runbook and agent now use `ipaddresses.ipam.cluster.x-k8s.io`.
+
+## Open questions
+
+- Why the CNI wasn't up (not installed yet vs. image pull / egress from VLAN 4) — needs the lab1
+  read-only kubeconfig or the owner's check.
+
+## Follow-ups
+
+- `.claude/cluster-inspector/rbac.yaml`: CRD read added (for `clusterctl describe`); applied by the
+  next `ansible-playbook mgmt.yml --tags inspector`.
+- Phase-1 checklist: Step 3 ticked; next Step 4 (Cilium), then Step 5 annotations in `lab1.yaml`.