kacerr/cluster-autoscaler-with-capi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cluster-inspector: read-only kubeconfig for lab1, workload-aware triage

Browse Source 
The agent could only reach lab-mgmt, so it couldn't verify nodes, CNI or
system pods on lab1. It correctly refused the admin ~/.kube/lab1.kubeconfig
when the owner's permission was relayed through the main session.

Workload kubeconfigs:
- make-kubeconfig.sh: new --capi-cluster [NS/]NAME. --admin-kubeconfig
  is then the management cluster. The workload admin kubeconfig is read
  from Secret NS/NAME-kubeconfig into a mktemp file, removed on exit, so
  no admin file for the workload cluster lands on disk.
- ansible/mgmt.yml --tags inspector now also mints kubeconfigs for every
  cluster in the new inspector_workload_clusters (group_vars: lab1).
  Clusters without a <name>-kubeconfig Secret are skipped. The tag is now
  the single weekly token refresh; tasks stay changed_when: false, so the
  playbook's changed=0 rule still holds.
- Owner ran --tags inspector: kubeconfigs/lab1.kubeconfig (lab1-ro)
  minted, can-i checks as expected.

Agent prompt:
- Cluster map: what to inspect on lab-mgmt vs the workload cluster.
  Machine -> Node -> VM via nodeRef, providerID and
  ProxmoxMachine.spec.virtualMachineID (checked on live objects).
- New triage step: scale-down blockers (non-DaemonSet kube-system pods
  without PDB, local storage, bare pods, PDBs at 0, safe-to-evict /
  scale-down-disabled).
- Refuses admin kubeconfigs even when a caller relays permission; points
  the owner to --tags inspector on missing file or 401.
- Verb before --kubeconfig so commands match the kubectl get/describe/logs
  allow rules; single-quote jsonpath/custom-columns for zsh.
- Report format: abnormal first, then health summary, then unverified.

settings.json: allow kubectl describe/logs/events and clusterctl describe.
All four are read-only, and the RO kubeconfigs enforce read-only anyway.

Checked: bash -n on the script; ansible-playbook --syntax-check and
ansible-lint --profile production pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-28 09:23:41 +02:00
co-authored by Claude Opus 5.5
parent 1d0b669d07
commit 49c548e296
7 changed files with 134 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/cluster-inspector/README.md
+14 -5
View File
@@ -26,15 +26,23 @@ rm /tmp/lab-mgmt-admin.yaml
## 2. Workload clusters (after phase 1 step 3)
Add the cluster to `inspector_workload_clusters` in `ansible/group_vars/all.yml`. Then
`ansible-playbook mgmt.yml --tags inspector` mints a read-only kubeconfig for lab-mgmt and for every
listed cluster whose `<name>-kubeconfig` Secret exists. Clusters without that Secret are skipped.
Doing it by hand gives the same result. `--capi-cluster` reads the workload admin kubeconfig from
the management cluster's Secret into a temp file that is deleted on exit:
```bash
clusterctl get kubeconfig lab1 > /tmp/lab1-admin.yaml # run against the mgmt cluster (admin)
.claude/cluster-inspector/make-kubeconfig.sh lab1 --admin-kubeconfig /tmp/lab1-admin.yaml --apply-rbac
rm /tmp/lab1-admin.yaml
.claude/cluster-inspector/make-kubeconfig.sh lab1 \
--admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig \
--capi-cluster default/lab1 \
--apply-rbac
```
## Expected verification output
```
```text
list pods yes
get machines.cluster.x-k8s.io yes # "no" on workload clusters (no CAPI CRDs), fine
delete machines.cluster.x-k8s.io no
@@ -46,7 +54,8 @@ Anything else means the RBAC is wrong. Don't use the agent until it matches.
## Refresh / revoke
- **Refresh:** re-run the same command without `--apply-rbac`. The default lifetime is 7 days
- **Refresh:** `ansible-playbook mgmt.yml --tags inspector` refreshes all clusters. You can also
re-run the same script command, with or without `--apply-rbac`; it's idempotent. The default lifetime is 7 days
(`--duration`); the API server may cap it (`--service-account-max-token-expiration`).
- **Revoke everything:** `kubectl delete sa -n kube-system readonly-inspector` (as admin). That
invalidates every minted token; re-apply `rbac.yaml` to start over.
.claude/cluster-inspector/make-kubeconfig.sh
+17 -3
View File
@@ -5,29 +5,43 @@
# <name> cluster name, e.g. lab-mgmt or lab1 -> kubeconfigs/<name>.kubeconfig
# --admin-kubeconfig PATH admin kubeconfig used to apply RBAC / mint the token (default: $KUBECONFIG or ~/.kube/config)
# --context CTX context inside the admin kubeconfig
# --capi-cluster [NS/]NAME workload cluster: --admin-kubeconfig is the management cluster; the workload
# admin kubeconfig is read from Secret NS/NAME-kubeconfig (NS default: default)
# into a temp file that is removed on exit
# --server URL override API server URL (k3s.yaml says https://127.0.0.1:6443)
# --duration DUR token lifetime (default 168h); re-run the script to refresh
# --apply-rbac kubectl apply rbac.yaml first (needed once per cluster)
set -euo pipefail
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; server=""; duration="168h"; apply=0
name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; capi=""; server=""; duration="168h"; apply=0
while [[ $# -gt 0 ]]; do
case "$1" in
--admin-kubeconfig) admin="$2"; shift 2 ;;
--context) ctx="$2"; shift 2 ;;
--capi-cluster) capi="$2"; shift 2 ;;
--server) server="$2"; shift 2 ;;
--duration) duration="$2"; shift 2 ;;
--apply-rbac) apply=1; shift ;;
-h|--help) sed -n '2,10p' "$0"; exit 0 ;;
-h|--help) sed -n '2,13p' "$0"; exit 0 ;;
-*) echo "unknown option $1" >&2; exit 1 ;;
*) name="$1"; shift ;;
esac
done
[[ -n "$name" ]] || { sed -n '2,10p' "$0" >&2; exit 1; }
[[ -n "$name" ]] || { sed -n '2,13p' "$0" >&2; exit 1; }
k=(kubectl --kubeconfig "$admin")
[[ -n "$ctx" ]] && k+=(--context "$ctx")
if [[ -n "$capi" ]]; then
ns="default"; cluster="$capi"
[[ "$capi" == */* ]] && { ns="${capi%%/*}"; cluster="${capi#*/}"; }
workload="$(mktemp)" # mktemp creates it 0600
trap 'rm -f "$workload"' EXIT
"${k[@]}" -n "$ns" get secret "$cluster-kubeconfig" -o jsonpath='{.data.value}' | base64 -d > "$workload"
[[ -s "$workload" ]] || { echo "Secret $ns/$cluster-kubeconfig is empty or missing" >&2; exit 1; }
k=(kubectl --kubeconfig "$workload")
fi
if (( apply )); then "${k[@]}" apply -f "$here/rbac.yaml"; fi
token="$("${k[@]}" -n kube-system create token readonly-inspector --duration "$duration")"