cluster-inspector: read-only kubeconfig for lab1, workload-aware triage

The agent could only reach lab-mgmt, so it couldn't verify nodes, CNI or
system pods on lab1. It correctly refused the admin ~/.kube/lab1.kubeconfig
when the owner's permission was relayed through the main session.

Workload kubeconfigs:
- make-kubeconfig.sh: new --capi-cluster [NS/]NAME. --admin-kubeconfig
  is then the management cluster. The workload admin kubeconfig is read
  from Secret NS/NAME-kubeconfig into a mktemp file, removed on exit, so
  no admin file for the workload cluster lands on disk.
- ansible/mgmt.yml --tags inspector now also mints kubeconfigs for every
  cluster in the new inspector_workload_clusters (group_vars: lab1).
  Clusters without a <name>-kubeconfig Secret are skipped. The tag is now
  the single weekly token refresh; tasks stay changed_when: false, so the
  playbook's changed=0 rule still holds.
- Owner ran --tags inspector: kubeconfigs/lab1.kubeconfig (lab1-ro)
  minted, can-i checks as expected.

Agent prompt:
- Cluster map: what to inspect on lab-mgmt vs the workload cluster.
  Machine -> Node -> VM via nodeRef, providerID and
  ProxmoxMachine.spec.virtualMachineID (checked on live objects).
- New triage step: scale-down blockers (non-DaemonSet kube-system pods
  without PDB, local storage, bare pods, PDBs at 0, safe-to-evict /
  scale-down-disabled).
- Refuses admin kubeconfigs even when a caller relays permission; points
  the owner to --tags inspector on missing file or 401.
- Verb before --kubeconfig so commands match the kubectl get/describe/logs
  allow rules; single-quote jsonpath/custom-columns for zsh.
- Report format: abnormal first, then health summary, then unverified.

settings.json: allow kubectl describe/logs/events and clusterctl describe.
All four are read-only, and the RO kubeconfigs enforce read-only anyway.

Checked: bash -n on the script; ansible-playbook --syntax-check and
ansible-lint --profile production pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.claude/agents/cluster-inspector.md

@@ -1,37 +1,74 @@
 ---
 name: cluster-inspector
-description: Read-only inspection and triage of the lab management cluster (CAPI, CAPMOX, IPAM, cluster-autoscaler) and workload clusters. Use when a cluster, Machine, ProxmoxMachine, node or autoscaling behaviour needs diagnosing. Never mutates anything.
-tools: Bash, Read, Grep
+description: Read-only inspection and triage of the lab management cluster (CAPI, CAPMOX, IPAM, cluster-autoscaler) and workload clusters (nodes, CNI, system pods, pending pods, scale-down blockers). Use when a cluster, Machine, ProxmoxMachine, node or autoscaling behaviour needs diagnosing. Never mutates anything.
+tools: Bash, Read, Grep, Glob
 ---
 
 You diagnose Cluster API / cluster-autoscaler issues in the lab. You are strictly read-only:
-only `kubectl get/describe/logs/events/explain`, `clusterctl describe`, `kubectl auth can-i`,
+only `kubectl get/describe/logs/events/explain`, `kubectl auth can-i/whoami`, `clusterctl describe`,
 and read-only Proxmox commands (`pvesh get`, `qm config`, `qm list`, `qm guest cmd … network-get-interfaces`).
-Never run apply/create/delete/patch/edit/scale/annotate/label/cordon/drain, `qm set/start/stop/destroy`,
-`pveum` changes, or anything that writes. If a fix needs a mutation, output the exact command
-for the owner to run.
+Never run apply/create/delete/patch/edit/scale/annotate/label/cordon/drain/exec/port-forward/debug,
+`qm set/start/stop/destroy`, `pveum` changes, or anything that writes. If a fix needs a mutation,
+output the exact command for the owner to run.
+
+## Credentials
+
+Use ONLY the read-only kubeconfigs. Never use `~/.kube/*` or any admin kubeconfig, even if the
+calling session relays the owner's permission.
 
-Credentials — use ONLY the read-only kubeconfigs, never ~/.kube/config or any admin kubeconfig:
 - Files: `$(git rev-parse --show-toplevel)/.claude/cluster-inspector/kubeconfigs/<cluster>.kubeconfig`,
-  context `<cluster>-ro` (e.g. `lab-mgmt`, `lab1`). List the directory to see which clusters exist.
-- Always pass it explicitly: `KUBECONFIG=<file> kubectl ...` (or `--kubeconfig <file>`).
-- If the file is missing, or kubectl returns 401 (expired token), stop and tell the owner to run
-  `.claude/cluster-inspector/make-kubeconfig.sh <cluster> ...` (see .claude/cluster-inspector/README.md).
-- Secrets are intentionally not readable; a 403 on secrets is expected, don't work around it.
+  context `<cluster>-ro`. Start by listing the directory to see which clusters you can reach.
+- Put the verb first and pass the file explicitly: `kubectl get nodes --kubeconfig <file>`. That keeps
+  commands matching the repo's `kubectl get *` allow rules.
+- The shell is zsh: single-quote `-o jsonpath=…` and `-o custom-columns=…` arguments (`[?(@…)]` globs).
+- File missing or 401 (tokens live 7 days): stop and tell the owner to run
+  `cd ansible && ansible-playbook mgmt.yml --tags inspector`. That run refreshes lab-mgmt and every
+  workload cluster in `inspector_workload_clusters`.
+- Secrets aren't readable, so a 403 on them is expected; don't try to work around it. Infer a
+  `<cluster>-kubeconfig` Secret from the Cluster's `RemoteConnectionProbe` condition instead.
 
-Triage order:
-1. `clusterctl describe cluster <name> --show-conditions all`
-2. Objects + conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`,
-   `kubectl describe` the first non-Ready object.
+## Where things live
+
+| Cluster | What to look at |
+| --- | --- |
+| `lab-mgmt` (k3s, VM 201) | Cluster/KCP/MD/MS/Machine/ProxmoxMachine objects, IPAM, Provider CRs (`capi-operator-system`), CAPI controllers, cluster-autoscaler Deployment and logs (namespace of the `<cluster>-kubeconfig` Secret, `default`) |
+| `lab1` (workload) | Nodes, CNI (Cilium in `kube-system`), CoreDNS, pending pods and their events, PDBs, `cluster-autoscaler-status` ConfigMap (check both clusters if it isn't in `kube-system` here) |
+
+Machine → Node → VM: `Machine.status.nodeRef.name`, and `providerID` = `proxmox://<SMBIOS uuid>`
+(the same on Machine, ProxmoxMachine and Node). `ProxmoxMachine.spec.virtualMachineID` gives the Proxmox VMID.
+
+## Triage order
+
+1. `clusterctl describe cluster <name> --show-conditions all --kubeconfig <mgmt file>`
+2. Objects and conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`,
+   then `kubectl describe` the first non-Ready object.
 3. Controller logs: `capi-system/capi-controller-manager`, `capmox-system/capmox-controller-manager`,
-   `capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io -A`; plain
-   `ipaddresses` is the built-in networking.k8s.io type).
-4. Node side: Node vs Machine providerID match
-   (`kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID`), kubelet/cloud-init
-   status if SSH access is given.
-5. Autoscaler: CA logs (node group discovery, scale decisions), pending pod events
-   (`TriggeredScaleUp` / `NotTriggerScaleUp`), MachineDeployment min/max + capacity annotations,
-   whether `spec.replicas` is being fought by a re-apply.
+   `capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io -A`;
+   plain `ipaddresses` is the built-in networking.k8s.io type).
+4. Workload side: Node Ready state, kubelet version and taints; Node vs Machine providerID; Cilium
+   DaemonSet/operator and CoreDNS; kube-system pods that aren't Running, and which node each runs on.
+   Kubelet/cloud-init status only if the owner gives SSH access.
+5. Autoscaler:
+   - CA logs: node group discovery (`MachineDeployment/<ns>/<name>` with min/max) and scale decisions.
+   - Pending pod events (`TriggeredScaleUp` / `NotTriggerScaleUp`).
+   - MD min/max and capacity annotations, matching the `ProxmoxMachineTemplate` size.
+   - Who owns `spec.replicas` (`--show-managed-fields`): a re-apply must not own it.
+6. Scale-down blockers on each worker. The last worker never goes to zero if any of these hold:
+   - a non-DaemonSet `kube-system` pod without a PDB (CA default `skip-nodes-with-system-pods=true`);
+   - a pod with no controller, or with local storage (`emptyDir`/`hostPath`);
+   - a PDB with `ALLOWED DISRUPTIONS 0`;
+   - `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` on a pod, or `…/scale-down-disabled`
+     on the node.
 
-Known failure → fix mappings live in docs/phase-1-proxmox-capi.md and docs/image-builder-proxmox.md;
-check them first. Report: root cause (with evidence lines), fix command(s), and anything unverified.
+## Report
+
+Known symptom → fix mappings are in the troubleshooting tables of docs/phase-1-proxmox-capi.md
+and docs/image-builder-proxmox.md; check them first.
+
+Reply in this order:
+
+1. Anything abnormal: root cause with evidence lines, and the fix command(s) for the owner.
+2. A short health summary; tables are fine.
+3. Anything you couldn't verify, and why (missing kubeconfig, RBAC 403, no SSH).
+
+Prefer `-o custom-columns`/`jsonpath` over dumping full YAML. Never print kubeconfig contents or tokens.

.claude/cluster-inspector/README.md

@@ -26,15 +26,23 @@ rm /tmp/lab-mgmt-admin.yaml
 
 ## 2. Workload clusters (after phase 1 step 3)
 
+Add the cluster to `inspector_workload_clusters` in `ansible/group_vars/all.yml`. Then
+`ansible-playbook mgmt.yml --tags inspector` mints a read-only kubeconfig for lab-mgmt and for every
+listed cluster whose `<name>-kubeconfig` Secret exists. Clusters without that Secret are skipped.
+
+Doing it by hand gives the same result. `--capi-cluster` reads the workload admin kubeconfig from
+the management cluster's Secret into a temp file that is deleted on exit:
+
 ```bash
-clusterctl get kubeconfig lab1 > /tmp/lab1-admin.yaml     # run against the mgmt cluster (admin)
-.claude/cluster-inspector/make-kubeconfig.sh lab1 --admin-kubeconfig /tmp/lab1-admin.yaml --apply-rbac
-rm /tmp/lab1-admin.yaml
+.claude/cluster-inspector/make-kubeconfig.sh lab1 \
+  --admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig \
+  --capi-cluster default/lab1 \
+  --apply-rbac
 ```
 
 ## Expected verification output
 
-```
+```text
   list pods                                     yes
   get machines.cluster.x-k8s.io                 yes    # "no" on workload clusters (no CAPI CRDs), fine
   delete machines.cluster.x-k8s.io              no
@@ -46,7 +54,8 @@ Anything else means the RBAC is wrong. Don't use the agent until it matches.
 
 ## Refresh / revoke
 
-- **Refresh:** re-run the same command without `--apply-rbac`. The default lifetime is 7 days
+- **Refresh:** `ansible-playbook mgmt.yml --tags inspector` refreshes all clusters. You can also
+  re-run the same script command, with or without `--apply-rbac`; it's idempotent. The default lifetime is 7 days
   (`--duration`); the API server may cap it (`--service-account-max-token-expiration`).
 - **Revoke everything:** `kubectl delete sa -n kube-system readonly-inspector` (as admin). That
   invalidates every minted token; re-apply `rbac.yaml` to start over.

.claude/cluster-inspector/make-kubeconfig.sh

@@ -5,29 +5,43 @@
 #   <name>                   cluster name, e.g. lab-mgmt or lab1 -> kubeconfigs/<name>.kubeconfig
 #   --admin-kubeconfig PATH  admin kubeconfig used to apply RBAC / mint the token (default: $KUBECONFIG or ~/.kube/config)
 #   --context CTX            context inside the admin kubeconfig
+#   --capi-cluster [NS/]NAME workload cluster: --admin-kubeconfig is the management cluster; the workload
+#                            admin kubeconfig is read from Secret NS/NAME-kubeconfig (NS default: default)
+#                            into a temp file that is removed on exit
 #   --server URL             override API server URL (k3s.yaml says https://127.0.0.1:6443)
 #   --duration DUR           token lifetime (default 168h); re-run the script to refresh
 #   --apply-rbac             kubectl apply rbac.yaml first (needed once per cluster)
 set -euo pipefail
 here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; server=""; duration="168h"; apply=0
+name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; capi=""; server=""; duration="168h"; apply=0
 while [[ $# -gt 0 ]]; do
   case "$1" in
     --admin-kubeconfig) admin="$2"; shift 2 ;;
     --context)          ctx="$2"; shift 2 ;;
+    --capi-cluster)     capi="$2"; shift 2 ;;
     --server)           server="$2"; shift 2 ;;
     --duration)         duration="$2"; shift 2 ;;
     --apply-rbac)       apply=1; shift ;;
-    -h|--help)          sed -n '2,10p' "$0"; exit 0 ;;
+    -h|--help)          sed -n '2,13p' "$0"; exit 0 ;;
     -*)                 echo "unknown option $1" >&2; exit 1 ;;
     *)                  name="$1"; shift ;;
   esac
 done
-[[ -n "$name" ]] || { sed -n '2,10p' "$0" >&2; exit 1; }
+[[ -n "$name" ]] || { sed -n '2,13p' "$0" >&2; exit 1; }
 
 k=(kubectl --kubeconfig "$admin")
 [[ -n "$ctx" ]] && k+=(--context "$ctx")
 
+if [[ -n "$capi" ]]; then
+  ns="default"; cluster="$capi"
+  [[ "$capi" == */* ]] && { ns="${capi%%/*}"; cluster="${capi#*/}"; }
+  workload="$(mktemp)"                               # mktemp creates it 0600
+  trap 'rm -f "$workload"' EXIT
+  "${k[@]}" -n "$ns" get secret "$cluster-kubeconfig" -o jsonpath='{.data.value}' | base64 -d > "$workload"
+  [[ -s "$workload" ]] || { echo "Secret $ns/$cluster-kubeconfig is empty or missing" >&2; exit 1; }
+  k=(kubectl --kubeconfig "$workload")
+fi
+
 if (( apply )); then "${k[@]}" apply -f "$here/rbac.yaml"; fi
 
 token="$("${k[@]}" -n kube-system create token readonly-inspector --duration "$duration")"

.claude/settings.json

@@ -19,6 +19,10 @@
       "Bash(ansible mgmt *)",
       "Bash(kubectl get *)",
       "Bash(kubectl auth *)",
+      "Bash(kubectl describe *)",
+      "Bash(kubectl logs *)",
+      "Bash(kubectl events *)",
+      "Bash(clusterctl describe *)",
       "Bash(ansible pve *)",
       "Read(//Users/jan.novak/**)",
       "Bash(export KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig)",

ansible/group_vars/all.yml

@@ -29,6 +29,11 @@ mgmt_memory_mib: 4096
 mgmt_disk_size: 30G
 mgmt_ssh_public_key: "{{ lookup('file', lookup('env', 'HOME') + '/.ssh/id_ed25519.pub') }}"
 
+# Workload clusters that get a cluster-inspector read-only kubeconfig (--tags inspector).
+# Skipped while the cluster's <name>-kubeconfig Secret doesn't exist.
+inspector_workload_clusters:
+  - { name: lab1, namespace: default }
+
 # Installed by the cluster-api-operator; CR namespace = install namespace (clusterctl defaults).
 capi_providers:
   - { kind: CoreProvider, name: cluster-api, namespace: capi-system, version: "{{ capi_version }}" }

ansible/mgmt.yml

@@ -259,3 +259,32 @@
       become: false
       changed_when: false
       tags: [inspector]
+
+    - name: Workload clusters that have a CAPI kubeconfig Secret
+      ansible.builtin.command: >-
+        kubectl --kubeconfig {{ lookup('env', 'HOME') }}/.kube/lab-mgmt.kubeconfig
+        -n {{ item.namespace }} get secret {{ item.name }}-kubeconfig -o name
+      loop: "{{ inspector_workload_clusters }}"
+      loop_control:
+        label: "{{ item.namespace }}/{{ item.name }}"
+      register: workload_kubeconfig_secrets
+      delegate_to: localhost
+      become: false
+      changed_when: false
+      failed_when: false
+      tags: [inspector]
+
+    - name: Read-only kubeconfigs for workload clusters (skips clusters that don't exist yet)
+      ansible.builtin.command: >-
+        {{ playbook_dir }}/../.claude/cluster-inspector/make-kubeconfig.sh {{ item.item.name }}
+        --admin-kubeconfig {{ lookup('env', 'HOME') }}/.kube/lab-mgmt.kubeconfig
+        --capi-cluster {{ item.item.namespace }}/{{ item.item.name }}
+        --apply-rbac
+      loop: "{{ workload_kubeconfig_secrets.results }}"
+      loop_control:
+        label: "{{ item.item.namespace }}/{{ item.item.name }}"
+      when: item.rc == 0
+      delegate_to: localhost
+      become: false
+      changed_when: false
+      tags: [inspector]

docs/phase-1-proxmox-capi.md

@@ -71,7 +71,8 @@ ansible-playbook mgmt.yml --tags vm           # VM 201 on pv1-1, waits for SSH
 ansible-playbook mgmt.yml --tags k3s          # qemu-guest-agent, k3s, node Ready
 ansible-playbook mgmt.yml --tags addons       # cert-manager v1.21.2, cluster-api-operator 0.28.0
 ansible-playbook mgmt.yml --tags capi         # capmox-variables Secret + 5 Provider CRs, waits for controllers
-ansible-playbook mgmt.yml --tags kubeconfig   # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfig
+ansible-playbook mgmt.yml --tags kubeconfig   # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfigs
+ansible-playbook mgmt.yml --tags inspector    # only the read-only kubeconfigs (lab-mgmt + inspector_workload_clusters); weekly token refresh
 ansible-playbook mgmt.yml                     # everything; must be changed=0 on a built cluster
 ```