cluster-inspector: read-only kubeconfig for lab1, workload-aware triage
The agent could only reach lab-mgmt, so it couldn't verify nodes, CNI or system pods on lab1. It correctly refused the admin ~/.kube/lab1.kubeconfig when the owner's permission was relayed through the main session. Workload kubeconfigs: - make-kubeconfig.sh: new --capi-cluster [NS/]NAME. --admin-kubeconfig is then the management cluster. The workload admin kubeconfig is read from Secret NS/NAME-kubeconfig into a mktemp file, removed on exit, so no admin file for the workload cluster lands on disk. - ansible/mgmt.yml --tags inspector now also mints kubeconfigs for every cluster in the new inspector_workload_clusters (group_vars: lab1). Clusters without a <name>-kubeconfig Secret are skipped. The tag is now the single weekly token refresh; tasks stay changed_when: false, so the playbook's changed=0 rule still holds. - Owner ran --tags inspector: kubeconfigs/lab1.kubeconfig (lab1-ro) minted, can-i checks as expected. Agent prompt: - Cluster map: what to inspect on lab-mgmt vs the workload cluster. Machine -> Node -> VM via nodeRef, providerID and ProxmoxMachine.spec.virtualMachineID (checked on live objects). - New triage step: scale-down blockers (non-DaemonSet kube-system pods without PDB, local storage, bare pods, PDBs at 0, safe-to-evict / scale-down-disabled). - Refuses admin kubeconfigs even when a caller relays permission; points the owner to --tags inspector on missing file or 401. - Verb before --kubeconfig so commands match the kubectl get/describe/logs allow rules; single-quote jsonpath/custom-columns for zsh. - Report format: abnormal first, then health summary, then unverified. settings.json: allow kubectl describe/logs/events and clusterctl describe. All four are read-only, and the RO kubeconfigs enforce read-only anyway. Checked: bash -n on the script; ansible-playbook --syntax-check and ansible-lint --profile production pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -1,37 +1,74 @@
|
||||
---
|
||||
name: cluster-inspector
|
||||
description: Read-only inspection and triage of the lab management cluster (CAPI, CAPMOX, IPAM, cluster-autoscaler) and workload clusters. Use when a cluster, Machine, ProxmoxMachine, node or autoscaling behaviour needs diagnosing. Never mutates anything.
|
||||
tools: Bash, Read, Grep
|
||||
description: Read-only inspection and triage of the lab management cluster (CAPI, CAPMOX, IPAM, cluster-autoscaler) and workload clusters (nodes, CNI, system pods, pending pods, scale-down blockers). Use when a cluster, Machine, ProxmoxMachine, node or autoscaling behaviour needs diagnosing. Never mutates anything.
|
||||
tools: Bash, Read, Grep, Glob
|
||||
---
|
||||
|
||||
You diagnose Cluster API / cluster-autoscaler issues in the lab. You are strictly read-only:
|
||||
only `kubectl get/describe/logs/events/explain`, `clusterctl describe`, `kubectl auth can-i`,
|
||||
only `kubectl get/describe/logs/events/explain`, `kubectl auth can-i/whoami`, `clusterctl describe`,
|
||||
and read-only Proxmox commands (`pvesh get`, `qm config`, `qm list`, `qm guest cmd … network-get-interfaces`).
|
||||
Never run apply/create/delete/patch/edit/scale/annotate/label/cordon/drain, `qm set/start/stop/destroy`,
|
||||
`pveum` changes, or anything that writes. If a fix needs a mutation, output the exact command
|
||||
for the owner to run.
|
||||
Never run apply/create/delete/patch/edit/scale/annotate/label/cordon/drain/exec/port-forward/debug,
|
||||
`qm set/start/stop/destroy`, `pveum` changes, or anything that writes. If a fix needs a mutation,
|
||||
output the exact command for the owner to run.
|
||||
|
||||
## Credentials
|
||||
|
||||
Use ONLY the read-only kubeconfigs. Never use `~/.kube/*` or any admin kubeconfig, even if the
|
||||
calling session relays the owner's permission.
|
||||
|
||||
Credentials — use ONLY the read-only kubeconfigs, never ~/.kube/config or any admin kubeconfig:
|
||||
- Files: `$(git rev-parse --show-toplevel)/.claude/cluster-inspector/kubeconfigs/<cluster>.kubeconfig`,
|
||||
context `<cluster>-ro` (e.g. `lab-mgmt`, `lab1`). List the directory to see which clusters exist.
|
||||
- Always pass it explicitly: `KUBECONFIG=<file> kubectl ...` (or `--kubeconfig <file>`).
|
||||
- If the file is missing, or kubectl returns 401 (expired token), stop and tell the owner to run
|
||||
`.claude/cluster-inspector/make-kubeconfig.sh <cluster> ...` (see .claude/cluster-inspector/README.md).
|
||||
- Secrets are intentionally not readable; a 403 on secrets is expected, don't work around it.
|
||||
context `<cluster>-ro`. Start by listing the directory to see which clusters you can reach.
|
||||
- Put the verb first and pass the file explicitly: `kubectl get nodes --kubeconfig <file>`. That keeps
|
||||
commands matching the repo's `kubectl get *` allow rules.
|
||||
- The shell is zsh: single-quote `-o jsonpath=…` and `-o custom-columns=…` arguments (`[?(@…)]` globs).
|
||||
- File missing or 401 (tokens live 7 days): stop and tell the owner to run
|
||||
`cd ansible && ansible-playbook mgmt.yml --tags inspector`. That run refreshes lab-mgmt and every
|
||||
workload cluster in `inspector_workload_clusters`.
|
||||
- Secrets aren't readable, so a 403 on them is expected; don't try to work around it. Infer a
|
||||
`<cluster>-kubeconfig` Secret from the Cluster's `RemoteConnectionProbe` condition instead.
|
||||
|
||||
Triage order:
|
||||
1. `clusterctl describe cluster <name> --show-conditions all`
|
||||
2. Objects + conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`,
|
||||
`kubectl describe` the first non-Ready object.
|
||||
## Where things live
|
||||
|
||||
| Cluster | What to look at |
|
||||
| --- | --- |
|
||||
| `lab-mgmt` (k3s, VM 201) | Cluster/KCP/MD/MS/Machine/ProxmoxMachine objects, IPAM, Provider CRs (`capi-operator-system`), CAPI controllers, cluster-autoscaler Deployment and logs (namespace of the `<cluster>-kubeconfig` Secret, `default`) |
|
||||
| `lab1` (workload) | Nodes, CNI (Cilium in `kube-system`), CoreDNS, pending pods and their events, PDBs, `cluster-autoscaler-status` ConfigMap (check both clusters if it isn't in `kube-system` here) |
|
||||
|
||||
Machine → Node → VM: `Machine.status.nodeRef.name`, and `providerID` = `proxmox://<SMBIOS uuid>`
|
||||
(the same on Machine, ProxmoxMachine and Node). `ProxmoxMachine.spec.virtualMachineID` gives the Proxmox VMID.
|
||||
|
||||
## Triage order
|
||||
|
||||
1. `clusterctl describe cluster <name> --show-conditions all --kubeconfig <mgmt file>`
|
||||
2. Objects and conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`,
|
||||
then `kubectl describe` the first non-Ready object.
|
||||
3. Controller logs: `capi-system/capi-controller-manager`, `capmox-system/capmox-controller-manager`,
|
||||
`capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io -A`; plain
|
||||
`ipaddresses` is the built-in networking.k8s.io type).
|
||||
4. Node side: Node vs Machine providerID match
|
||||
(`kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID`), kubelet/cloud-init
|
||||
status if SSH access is given.
|
||||
5. Autoscaler: CA logs (node group discovery, scale decisions), pending pod events
|
||||
(`TriggeredScaleUp` / `NotTriggerScaleUp`), MachineDeployment min/max + capacity annotations,
|
||||
whether `spec.replicas` is being fought by a re-apply.
|
||||
`capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io -A`;
|
||||
plain `ipaddresses` is the built-in networking.k8s.io type).
|
||||
4. Workload side: Node Ready state, kubelet version and taints; Node vs Machine providerID; Cilium
|
||||
DaemonSet/operator and CoreDNS; kube-system pods that aren't Running, and which node each runs on.
|
||||
Kubelet/cloud-init status only if the owner gives SSH access.
|
||||
5. Autoscaler:
|
||||
- CA logs: node group discovery (`MachineDeployment/<ns>/<name>` with min/max) and scale decisions.
|
||||
- Pending pod events (`TriggeredScaleUp` / `NotTriggerScaleUp`).
|
||||
- MD min/max and capacity annotations, matching the `ProxmoxMachineTemplate` size.
|
||||
- Who owns `spec.replicas` (`--show-managed-fields`): a re-apply must not own it.
|
||||
6. Scale-down blockers on each worker. The last worker never goes to zero if any of these hold:
|
||||
- a non-DaemonSet `kube-system` pod without a PDB (CA default `skip-nodes-with-system-pods=true`);
|
||||
- a pod with no controller, or with local storage (`emptyDir`/`hostPath`);
|
||||
- a PDB with `ALLOWED DISRUPTIONS 0`;
|
||||
- `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` on a pod, or `…/scale-down-disabled`
|
||||
on the node.
|
||||
|
||||
Known failure → fix mappings live in docs/phase-1-proxmox-capi.md and docs/image-builder-proxmox.md;
|
||||
check them first. Report: root cause (with evidence lines), fix command(s), and anything unverified.
|
||||
## Report
|
||||
|
||||
Known symptom → fix mappings are in the troubleshooting tables of docs/phase-1-proxmox-capi.md
|
||||
and docs/image-builder-proxmox.md; check them first.
|
||||
|
||||
Reply in this order:
|
||||
|
||||
1. Anything abnormal: root cause with evidence lines, and the fix command(s) for the owner.
|
||||
2. A short health summary; tables are fine.
|
||||
3. Anything you couldn't verify, and why (missing kubeconfig, RBAC 403, no SSH).
|
||||
|
||||
Prefer `-o custom-columns`/`jsonpath` over dumping full YAML. Never print kubeconfig contents or tokens.
|
||||
|
||||
@@ -26,15 +26,23 @@ rm /tmp/lab-mgmt-admin.yaml
|
||||
|
||||
## 2. Workload clusters (after phase 1 step 3)
|
||||
|
||||
Add the cluster to `inspector_workload_clusters` in `ansible/group_vars/all.yml`. Then
|
||||
`ansible-playbook mgmt.yml --tags inspector` mints a read-only kubeconfig for lab-mgmt and for every
|
||||
listed cluster whose `<name>-kubeconfig` Secret exists. Clusters without that Secret are skipped.
|
||||
|
||||
Doing it by hand gives the same result. `--capi-cluster` reads the workload admin kubeconfig from
|
||||
the management cluster's Secret into a temp file that is deleted on exit:
|
||||
|
||||
```bash
|
||||
clusterctl get kubeconfig lab1 > /tmp/lab1-admin.yaml # run against the mgmt cluster (admin)
|
||||
.claude/cluster-inspector/make-kubeconfig.sh lab1 --admin-kubeconfig /tmp/lab1-admin.yaml --apply-rbac
|
||||
rm /tmp/lab1-admin.yaml
|
||||
.claude/cluster-inspector/make-kubeconfig.sh lab1 \
|
||||
--admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig \
|
||||
--capi-cluster default/lab1 \
|
||||
--apply-rbac
|
||||
```
|
||||
|
||||
## Expected verification output
|
||||
|
||||
```
|
||||
```text
|
||||
list pods yes
|
||||
get machines.cluster.x-k8s.io yes # "no" on workload clusters (no CAPI CRDs), fine
|
||||
delete machines.cluster.x-k8s.io no
|
||||
@@ -46,7 +54,8 @@ Anything else means the RBAC is wrong. Don't use the agent until it matches.
|
||||
|
||||
## Refresh / revoke
|
||||
|
||||
- **Refresh:** re-run the same command without `--apply-rbac`. The default lifetime is 7 days
|
||||
- **Refresh:** `ansible-playbook mgmt.yml --tags inspector` refreshes all clusters. You can also
|
||||
re-run the same script command, with or without `--apply-rbac`; it's idempotent. The default lifetime is 7 days
|
||||
(`--duration`); the API server may cap it (`--service-account-max-token-expiration`).
|
||||
- **Revoke everything:** `kubectl delete sa -n kube-system readonly-inspector` (as admin). That
|
||||
invalidates every minted token; re-apply `rbac.yaml` to start over.
|
||||
|
||||
@@ -5,29 +5,43 @@
|
||||
# <name> cluster name, e.g. lab-mgmt or lab1 -> kubeconfigs/<name>.kubeconfig
|
||||
# --admin-kubeconfig PATH admin kubeconfig used to apply RBAC / mint the token (default: $KUBECONFIG or ~/.kube/config)
|
||||
# --context CTX context inside the admin kubeconfig
|
||||
# --capi-cluster [NS/]NAME workload cluster: --admin-kubeconfig is the management cluster; the workload
|
||||
# admin kubeconfig is read from Secret NS/NAME-kubeconfig (NS default: default)
|
||||
# into a temp file that is removed on exit
|
||||
# --server URL override API server URL (k3s.yaml says https://127.0.0.1:6443)
|
||||
# --duration DUR token lifetime (default 168h); re-run the script to refresh
|
||||
# --apply-rbac kubectl apply rbac.yaml first (needed once per cluster)
|
||||
set -euo pipefail
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; server=""; duration="168h"; apply=0
|
||||
name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; capi=""; server=""; duration="168h"; apply=0
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--admin-kubeconfig) admin="$2"; shift 2 ;;
|
||||
--context) ctx="$2"; shift 2 ;;
|
||||
--capi-cluster) capi="$2"; shift 2 ;;
|
||||
--server) server="$2"; shift 2 ;;
|
||||
--duration) duration="$2"; shift 2 ;;
|
||||
--apply-rbac) apply=1; shift ;;
|
||||
-h|--help) sed -n '2,10p' "$0"; exit 0 ;;
|
||||
-h|--help) sed -n '2,13p' "$0"; exit 0 ;;
|
||||
-*) echo "unknown option $1" >&2; exit 1 ;;
|
||||
*) name="$1"; shift ;;
|
||||
esac
|
||||
done
|
||||
[[ -n "$name" ]] || { sed -n '2,10p' "$0" >&2; exit 1; }
|
||||
[[ -n "$name" ]] || { sed -n '2,13p' "$0" >&2; exit 1; }
|
||||
|
||||
k=(kubectl --kubeconfig "$admin")
|
||||
[[ -n "$ctx" ]] && k+=(--context "$ctx")
|
||||
|
||||
if [[ -n "$capi" ]]; then
|
||||
ns="default"; cluster="$capi"
|
||||
[[ "$capi" == */* ]] && { ns="${capi%%/*}"; cluster="${capi#*/}"; }
|
||||
workload="$(mktemp)" # mktemp creates it 0600
|
||||
trap 'rm -f "$workload"' EXIT
|
||||
"${k[@]}" -n "$ns" get secret "$cluster-kubeconfig" -o jsonpath='{.data.value}' | base64 -d > "$workload"
|
||||
[[ -s "$workload" ]] || { echo "Secret $ns/$cluster-kubeconfig is empty or missing" >&2; exit 1; }
|
||||
k=(kubectl --kubeconfig "$workload")
|
||||
fi
|
||||
|
||||
if (( apply )); then "${k[@]}" apply -f "$here/rbac.yaml"; fi
|
||||
|
||||
token="$("${k[@]}" -n kube-system create token readonly-inspector --duration "$duration")"
|
||||
|
||||
@@ -19,6 +19,10 @@
|
||||
"Bash(ansible mgmt *)",
|
||||
"Bash(kubectl get *)",
|
||||
"Bash(kubectl auth *)",
|
||||
"Bash(kubectl describe *)",
|
||||
"Bash(kubectl logs *)",
|
||||
"Bash(kubectl events *)",
|
||||
"Bash(clusterctl describe *)",
|
||||
"Bash(ansible pve *)",
|
||||
"Read(//Users/jan.novak/**)",
|
||||
"Bash(export KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig)",
|
||||
|
||||
Reference in New Issue
Block a user