Files

T

kacerrandClaude Opus 5.5 49c548e296 cluster-inspector: read-only kubeconfig for lab1, workload-aware triage

The agent could only reach lab-mgmt, so it couldn't verify nodes, CNI or
system pods on lab1. It correctly refused the admin ~/.kube/lab1.kubeconfig
when the owner's permission was relayed through the main session.

Workload kubeconfigs:
- make-kubeconfig.sh: new --capi-cluster [NS/]NAME. --admin-kubeconfig
  is then the management cluster. The workload admin kubeconfig is read
  from Secret NS/NAME-kubeconfig into a mktemp file, removed on exit, so
  no admin file for the workload cluster lands on disk.
- ansible/mgmt.yml --tags inspector now also mints kubeconfigs for every
  cluster in the new inspector_workload_clusters (group_vars: lab1).
  Clusters without a <name>-kubeconfig Secret are skipped. The tag is now
  the single weekly token refresh; tasks stay changed_when: false, so the
  playbook's changed=0 rule still holds.
- Owner ran --tags inspector: kubeconfigs/lab1.kubeconfig (lab1-ro)
  minted, can-i checks as expected.

Agent prompt:
- Cluster map: what to inspect on lab-mgmt vs the workload cluster.
  Machine -> Node -> VM via nodeRef, providerID and
  ProxmoxMachine.spec.virtualMachineID (checked on live objects).
- New triage step: scale-down blockers (non-DaemonSet kube-system pods
  without PDB, local storage, bare pods, PDBs at 0, safe-to-evict /
  scale-down-disabled).
- Refuses admin kubeconfigs even when a caller relays permission; points
  the owner to --tags inspector on missing file or 401.
- Verb before --kubeconfig so commands match the kubectl get/describe/logs
  allow rules; single-quote jsonpath/custom-columns for zsh.
- Report format: abnormal first, then health summary, then unverified.

settings.json: allow kubectl describe/logs/events and clusterctl describe.
All four are read-only, and the RO kubeconfigs enforce read-only anyway.

Checked: bash -n on the script; ansible-playbook --syntax-check and
ansible-lint --profile production pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-09-28 09:23:41 +02:00

3.5 KiB

Raw Permalink Blame History

cluster-inspector: credentials setup

The cluster-inspector agent (.claude/agents/cluster-inspector.md) only ever talks to clusters through the read-only kubeconfigs in kubeconfigs/ here. They're gitignored and minted by make-kubeconfig.sh from a short-lived ServiceAccount token.

File	Purpose
`rbac.yaml`	ServiceAccount `kube-system/readonly-inspector` + `view` + read on CAPI CRDs and pod logs (no Secrets)
`make-kubeconfig.sh`	applies RBAC (once), mints a TokenRequest token, writes `kubeconfigs/<name>.kubeconfig`, runs `can-i` checks
`kubeconfigs/`	output, gitignored; one file per cluster, context `<name>-ro`

1. Management cluster (after phase 1 step 2)

k3s writes server: https://127.0.0.1:6443, so pass the VM address. k3s puts node IPs into the API cert SANs, so the IP works; add --tls-san for a DNS name.

scp ubuntu@<mgmt-vm>:/etc/rancher/k3s/k3s.yaml /tmp/lab-mgmt-admin.yaml   # or use your existing admin kubeconfig
.claude/cluster-inspector/make-kubeconfig.sh lab-mgmt \
  --admin-kubeconfig /tmp/lab-mgmt-admin.yaml \
  --server https://<mgmt-vm-ip>:6443 \
  --apply-rbac
rm /tmp/lab-mgmt-admin.yaml

2. Workload clusters (after phase 1 step 3)

Add the cluster to inspector_workload_clusters in ansible/group_vars/all.yml. Then ansible-playbook mgmt.yml --tags inspector mints a read-only kubeconfig for lab-mgmt and for every listed cluster whose <name>-kubeconfig Secret exists. Clusters without that Secret are skipped.

Doing it by hand gives the same result. --capi-cluster reads the workload admin kubeconfig from the management cluster's Secret into a temp file that is deleted on exit:

.claude/cluster-inspector/make-kubeconfig.sh lab1 \
  --admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig \
  --capi-cluster default/lab1 \
  --apply-rbac

Expected verification output

  list pods                                     yes
  get machines.cluster.x-k8s.io                 yes    # "no" on workload clusters (no CAPI CRDs), fine
  delete machines.cluster.x-k8s.io              no
  patch machinedeployments.cluster.x-k8s.io     no
  get secrets                                   no

Anything else means the RBAC is wrong. Don't use the agent until it matches.

Refresh / revoke

Refresh: ansible-playbook mgmt.yml --tags inspector refreshes all clusters. You can also re-run the same script command, with or without --apply-rbac; it's idempotent. The default lifetime is 7 days (--duration); the API server may cap it (--service-account-max-token-expiration).
Revoke everything: kubectl delete sa -n kube-system readonly-inspector (as admin). That invalidates every minted token; re-apply rbac.yaml to start over.

Making read-only the default for this repo (recommended)

RBAC only protects you when the read-only kubeconfig is actually used. Your admin ~/.kube/config is still reachable from any shell. To make Claude Code sessions in this repo default to the read-only files, add this to .claude/settings.local.json (personal, not committed):

{
  "env": {
    "KUBECONFIG": "/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig:/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab1.kubeconfig"
  }
}

Then plain kubectl --context lab-mgmt-ro … / --context lab1-ro … works everywhere in the session, and mutations fail on RBAC even if something tries them.

3.5 KiB Raw Permalink Blame History