cluster-inspector: read-only kubeconfig for lab1, workload-aware triage

The agent could only reach lab-mgmt, so it couldn't verify nodes, CNI or
system pods on lab1. It correctly refused the admin ~/.kube/lab1.kubeconfig
when the owner's permission was relayed through the main session.

Workload kubeconfigs:
- make-kubeconfig.sh: new --capi-cluster [NS/]NAME. --admin-kubeconfig
  is then the management cluster. The workload admin kubeconfig is read
  from Secret NS/NAME-kubeconfig into a mktemp file, removed on exit, so
  no admin file for the workload cluster lands on disk.
- ansible/mgmt.yml --tags inspector now also mints kubeconfigs for every
  cluster in the new inspector_workload_clusters (group_vars: lab1).
  Clusters without a <name>-kubeconfig Secret are skipped. The tag is now
  the single weekly token refresh; tasks stay changed_when: false, so the
  playbook's changed=0 rule still holds.
- Owner ran --tags inspector: kubeconfigs/lab1.kubeconfig (lab1-ro)
  minted, can-i checks as expected.

Agent prompt:
- Cluster map: what to inspect on lab-mgmt vs the workload cluster.
  Machine -> Node -> VM via nodeRef, providerID and
  ProxmoxMachine.spec.virtualMachineID (checked on live objects).
- New triage step: scale-down blockers (non-DaemonSet kube-system pods
  without PDB, local storage, bare pods, PDBs at 0, safe-to-evict /
  scale-down-disabled).
- Refuses admin kubeconfigs even when a caller relays permission; points
  the owner to --tags inspector on missing file or 401.
- Verb before --kubeconfig so commands match the kubectl get/describe/logs
  allow rules; single-quote jsonpath/custom-columns for zsh.
- Report format: abnormal first, then health summary, then unverified.

settings.json: allow kubectl describe/logs/events and clusterctl describe.
All four are read-only, and the RO kubeconfigs enforce read-only anyway.

Checked: bash -n on the script; ansible-playbook --syntax-check and
ansible-lint --profile production pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ansible/group_vars/all.yml

@@ -29,6 +29,11 @@ mgmt_memory_mib: 4096
 mgmt_disk_size: 30G
 mgmt_ssh_public_key: "{{ lookup('file', lookup('env', 'HOME') + '/.ssh/id_ed25519.pub') }}"
 
+# Workload clusters that get a cluster-inspector read-only kubeconfig (--tags inspector).
+# Skipped while the cluster's <name>-kubeconfig Secret doesn't exist.
+inspector_workload_clusters:
+  - { name: lab1, namespace: default }
+
 # Installed by the cluster-api-operator; CR namespace = install namespace (clusterctl defaults).
 capi_providers:
   - { kind: CoreProvider, name: cluster-api, namespace: capi-system, version: "{{ capi_version }}" }

ansible/mgmt.yml

@@ -259,3 +259,32 @@
       become: false
       changed_when: false
       tags: [inspector]
+
+    - name: Workload clusters that have a CAPI kubeconfig Secret
+      ansible.builtin.command: >-
+        kubectl --kubeconfig {{ lookup('env', 'HOME') }}/.kube/lab-mgmt.kubeconfig
+        -n {{ item.namespace }} get secret {{ item.name }}-kubeconfig -o name
+      loop: "{{ inspector_workload_clusters }}"
+      loop_control:
+        label: "{{ item.namespace }}/{{ item.name }}"
+      register: workload_kubeconfig_secrets
+      delegate_to: localhost
+      become: false
+      changed_when: false
+      failed_when: false
+      tags: [inspector]
+
+    - name: Read-only kubeconfigs for workload clusters (skips clusters that don't exist yet)
+      ansible.builtin.command: >-
+        {{ playbook_dir }}/../.claude/cluster-inspector/make-kubeconfig.sh {{ item.item.name }}
+        --admin-kubeconfig {{ lookup('env', 'HOME') }}/.kube/lab-mgmt.kubeconfig
+        --capi-cluster {{ item.item.namespace }}/{{ item.item.name }}
+        --apply-rbac
+      loop: "{{ workload_kubeconfig_secrets.results }}"
+      loop_control:
+        label: "{{ item.item.namespace }}/{{ item.item.name }}"
+      when: item.rc == 0
+      delegate_to: localhost
+      become: false
+      changed_when: false
+      tags: [inspector]