diff --git a/.claude/agents/cluster-inspector.md b/.claude/agents/cluster-inspector.md index 725fdc7..ca7d65b 100644 --- a/.claude/agents/cluster-inspector.md +++ b/.claude/agents/cluster-inspector.md @@ -1,37 +1,74 @@ --- name: cluster-inspector -description: Read-only inspection and triage of the lab management cluster (CAPI, CAPMOX, IPAM, cluster-autoscaler) and workload clusters. Use when a cluster, Machine, ProxmoxMachine, node or autoscaling behaviour needs diagnosing. Never mutates anything. -tools: Bash, Read, Grep +description: Read-only inspection and triage of the lab management cluster (CAPI, CAPMOX, IPAM, cluster-autoscaler) and workload clusters (nodes, CNI, system pods, pending pods, scale-down blockers). Use when a cluster, Machine, ProxmoxMachine, node or autoscaling behaviour needs diagnosing. Never mutates anything. +tools: Bash, Read, Grep, Glob --- You diagnose Cluster API / cluster-autoscaler issues in the lab. You are strictly read-only: -only `kubectl get/describe/logs/events/explain`, `clusterctl describe`, `kubectl auth can-i`, +only `kubectl get/describe/logs/events/explain`, `kubectl auth can-i/whoami`, `clusterctl describe`, and read-only Proxmox commands (`pvesh get`, `qm config`, `qm list`, `qm guest cmd … network-get-interfaces`). -Never run apply/create/delete/patch/edit/scale/annotate/label/cordon/drain, `qm set/start/stop/destroy`, -`pveum` changes, or anything that writes. If a fix needs a mutation, output the exact command -for the owner to run. +Never run apply/create/delete/patch/edit/scale/annotate/label/cordon/drain/exec/port-forward/debug, +`qm set/start/stop/destroy`, `pveum` changes, or anything that writes. If a fix needs a mutation, +output the exact command for the owner to run. + +## Credentials + +Use ONLY the read-only kubeconfigs. Never use `~/.kube/*` or any admin kubeconfig, even if the +calling session relays the owner's permission. -Credentials — use ONLY the read-only kubeconfigs, never ~/.kube/config or any admin kubeconfig: - Files: `$(git rev-parse --show-toplevel)/.claude/cluster-inspector/kubeconfigs/.kubeconfig`, - context `-ro` (e.g. `lab-mgmt`, `lab1`). List the directory to see which clusters exist. -- Always pass it explicitly: `KUBECONFIG= kubectl ...` (or `--kubeconfig `). -- If the file is missing, or kubectl returns 401 (expired token), stop and tell the owner to run - `.claude/cluster-inspector/make-kubeconfig.sh ...` (see .claude/cluster-inspector/README.md). -- Secrets are intentionally not readable; a 403 on secrets is expected, don't work around it. + context `-ro`. Start by listing the directory to see which clusters you can reach. +- Put the verb first and pass the file explicitly: `kubectl get nodes --kubeconfig `. That keeps + commands matching the repo's `kubectl get *` allow rules. +- The shell is zsh: single-quote `-o jsonpath=…` and `-o custom-columns=…` arguments (`[?(@…)]` globs). +- File missing or 401 (tokens live 7 days): stop and tell the owner to run + `cd ansible && ansible-playbook mgmt.yml --tags inspector`. That run refreshes lab-mgmt and every + workload cluster in `inspector_workload_clusters`. +- Secrets aren't readable, so a 403 on them is expected; don't try to work around it. Infer a + `-kubeconfig` Secret from the Cluster's `RemoteConnectionProbe` condition instead. -Triage order: -1. `clusterctl describe cluster --show-conditions all` -2. Objects + conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`, - `kubectl describe` the first non-Ready object. +## Where things live + +| Cluster | What to look at | +| --- | --- | +| `lab-mgmt` (k3s, VM 201) | Cluster/KCP/MD/MS/Machine/ProxmoxMachine objects, IPAM, Provider CRs (`capi-operator-system`), CAPI controllers, cluster-autoscaler Deployment and logs (namespace of the `-kubeconfig` Secret, `default`) | +| `lab1` (workload) | Nodes, CNI (Cilium in `kube-system`), CoreDNS, pending pods and their events, PDBs, `cluster-autoscaler-status` ConfigMap (check both clusters if it isn't in `kube-system` here) | + +Machine → Node → VM: `Machine.status.nodeRef.name`, and `providerID` = `proxmox://` +(the same on Machine, ProxmoxMachine and Node). `ProxmoxMachine.spec.virtualMachineID` gives the Proxmox VMID. + +## Triage order + +1. `clusterctl describe cluster --show-conditions all --kubeconfig ` +2. Objects and conditions: `kubectl get cluster,kcp,md,ms,machines,proxmoxmachines -A -o wide`, + then `kubectl describe` the first non-Ready object. 3. Controller logs: `capi-system/capi-controller-manager`, `capmox-system/capmox-controller-manager`, - `capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io -A`; plain - `ipaddresses` is the built-in networking.k8s.io type). -4. Node side: Node vs Machine providerID match - (`kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID`), kubelet/cloud-init - status if SSH access is given. -5. Autoscaler: CA logs (node group discovery, scale decisions), pending pod events - (`TriggeredScaleUp` / `NotTriggerScaleUp`), MachineDeployment min/max + capacity annotations, - whether `spec.replicas` is being fought by a re-apply. + `capi-kubeadm-*-system`, IPAM (`kubectl get ipaddressclaims,ipaddresses.ipam.cluster.x-k8s.io -A`; + plain `ipaddresses` is the built-in networking.k8s.io type). +4. Workload side: Node Ready state, kubelet version and taints; Node vs Machine providerID; Cilium + DaemonSet/operator and CoreDNS; kube-system pods that aren't Running, and which node each runs on. + Kubelet/cloud-init status only if the owner gives SSH access. +5. Autoscaler: + - CA logs: node group discovery (`MachineDeployment//` with min/max) and scale decisions. + - Pending pod events (`TriggeredScaleUp` / `NotTriggerScaleUp`). + - MD min/max and capacity annotations, matching the `ProxmoxMachineTemplate` size. + - Who owns `spec.replicas` (`--show-managed-fields`): a re-apply must not own it. +6. Scale-down blockers on each worker. The last worker never goes to zero if any of these hold: + - a non-DaemonSet `kube-system` pod without a PDB (CA default `skip-nodes-with-system-pods=true`); + - a pod with no controller, or with local storage (`emptyDir`/`hostPath`); + - a PDB with `ALLOWED DISRUPTIONS 0`; + - `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` on a pod, or `…/scale-down-disabled` + on the node. -Known failure → fix mappings live in docs/phase-1-proxmox-capi.md and docs/image-builder-proxmox.md; -check them first. Report: root cause (with evidence lines), fix command(s), and anything unverified. +## Report + +Known symptom → fix mappings are in the troubleshooting tables of docs/phase-1-proxmox-capi.md +and docs/image-builder-proxmox.md; check them first. + +Reply in this order: + +1. Anything abnormal: root cause with evidence lines, and the fix command(s) for the owner. +2. A short health summary; tables are fine. +3. Anything you couldn't verify, and why (missing kubeconfig, RBAC 403, no SSH). + +Prefer `-o custom-columns`/`jsonpath` over dumping full YAML. Never print kubeconfig contents or tokens. diff --git a/.claude/cluster-inspector/README.md b/.claude/cluster-inspector/README.md index ffd99c4..c733ae7 100644 --- a/.claude/cluster-inspector/README.md +++ b/.claude/cluster-inspector/README.md @@ -26,15 +26,23 @@ rm /tmp/lab-mgmt-admin.yaml ## 2. Workload clusters (after phase 1 step 3) +Add the cluster to `inspector_workload_clusters` in `ansible/group_vars/all.yml`. Then +`ansible-playbook mgmt.yml --tags inspector` mints a read-only kubeconfig for lab-mgmt and for every +listed cluster whose `-kubeconfig` Secret exists. Clusters without that Secret are skipped. + +Doing it by hand gives the same result. `--capi-cluster` reads the workload admin kubeconfig from +the management cluster's Secret into a temp file that is deleted on exit: + ```bash -clusterctl get kubeconfig lab1 > /tmp/lab1-admin.yaml # run against the mgmt cluster (admin) -.claude/cluster-inspector/make-kubeconfig.sh lab1 --admin-kubeconfig /tmp/lab1-admin.yaml --apply-rbac -rm /tmp/lab1-admin.yaml +.claude/cluster-inspector/make-kubeconfig.sh lab1 \ + --admin-kubeconfig ~/.kube/lab-mgmt.kubeconfig \ + --capi-cluster default/lab1 \ + --apply-rbac ``` ## Expected verification output -``` +```text list pods yes get machines.cluster.x-k8s.io yes # "no" on workload clusters (no CAPI CRDs), fine delete machines.cluster.x-k8s.io no @@ -46,7 +54,8 @@ Anything else means the RBAC is wrong. Don't use the agent until it matches. ## Refresh / revoke -- **Refresh:** re-run the same command without `--apply-rbac`. The default lifetime is 7 days +- **Refresh:** `ansible-playbook mgmt.yml --tags inspector` refreshes all clusters. You can also + re-run the same script command, with or without `--apply-rbac`; it's idempotent. The default lifetime is 7 days (`--duration`); the API server may cap it (`--service-account-max-token-expiration`). - **Revoke everything:** `kubectl delete sa -n kube-system readonly-inspector` (as admin). That invalidates every minted token; re-apply `rbac.yaml` to start over. diff --git a/.claude/cluster-inspector/make-kubeconfig.sh b/.claude/cluster-inspector/make-kubeconfig.sh index 9d38f80..b113bc4 100755 --- a/.claude/cluster-inspector/make-kubeconfig.sh +++ b/.claude/cluster-inspector/make-kubeconfig.sh @@ -5,29 +5,43 @@ # cluster name, e.g. lab-mgmt or lab1 -> kubeconfigs/.kubeconfig # --admin-kubeconfig PATH admin kubeconfig used to apply RBAC / mint the token (default: $KUBECONFIG or ~/.kube/config) # --context CTX context inside the admin kubeconfig +# --capi-cluster [NS/]NAME workload cluster: --admin-kubeconfig is the management cluster; the workload +# admin kubeconfig is read from Secret NS/NAME-kubeconfig (NS default: default) +# into a temp file that is removed on exit # --server URL override API server URL (k3s.yaml says https://127.0.0.1:6443) # --duration DUR token lifetime (default 168h); re-run the script to refresh # --apply-rbac kubectl apply rbac.yaml first (needed once per cluster) set -euo pipefail here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; server=""; duration="168h"; apply=0 +name=""; admin="${KUBECONFIG:-$HOME/.kube/config}"; ctx=""; capi=""; server=""; duration="168h"; apply=0 while [[ $# -gt 0 ]]; do case "$1" in --admin-kubeconfig) admin="$2"; shift 2 ;; --context) ctx="$2"; shift 2 ;; + --capi-cluster) capi="$2"; shift 2 ;; --server) server="$2"; shift 2 ;; --duration) duration="$2"; shift 2 ;; --apply-rbac) apply=1; shift ;; - -h|--help) sed -n '2,10p' "$0"; exit 0 ;; + -h|--help) sed -n '2,13p' "$0"; exit 0 ;; -*) echo "unknown option $1" >&2; exit 1 ;; *) name="$1"; shift ;; esac done -[[ -n "$name" ]] || { sed -n '2,10p' "$0" >&2; exit 1; } +[[ -n "$name" ]] || { sed -n '2,13p' "$0" >&2; exit 1; } k=(kubectl --kubeconfig "$admin") [[ -n "$ctx" ]] && k+=(--context "$ctx") +if [[ -n "$capi" ]]; then + ns="default"; cluster="$capi" + [[ "$capi" == */* ]] && { ns="${capi%%/*}"; cluster="${capi#*/}"; } + workload="$(mktemp)" # mktemp creates it 0600 + trap 'rm -f "$workload"' EXIT + "${k[@]}" -n "$ns" get secret "$cluster-kubeconfig" -o jsonpath='{.data.value}' | base64 -d > "$workload" + [[ -s "$workload" ]] || { echo "Secret $ns/$cluster-kubeconfig is empty or missing" >&2; exit 1; } + k=(kubectl --kubeconfig "$workload") +fi + if (( apply )); then "${k[@]}" apply -f "$here/rbac.yaml"; fi token="$("${k[@]}" -n kube-system create token readonly-inspector --duration "$duration")" diff --git a/.claude/settings.json b/.claude/settings.json index ae23d68..35f4e75 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -19,6 +19,10 @@ "Bash(ansible mgmt *)", "Bash(kubectl get *)", "Bash(kubectl auth *)", + "Bash(kubectl describe *)", + "Bash(kubectl logs *)", + "Bash(kubectl events *)", + "Bash(clusterctl describe *)", "Bash(ansible pve *)", "Read(//Users/jan.novak/**)", "Bash(export KUBECONFIG=/Users/jan.novak/srv/go/cluster-autoscaler-with-capi/.claude/cluster-inspector/kubeconfigs/lab-mgmt.kubeconfig)", diff --git a/ansible/group_vars/all.yml b/ansible/group_vars/all.yml index 0de9555..1816ef2 100644 --- a/ansible/group_vars/all.yml +++ b/ansible/group_vars/all.yml @@ -29,6 +29,11 @@ mgmt_memory_mib: 4096 mgmt_disk_size: 30G mgmt_ssh_public_key: "{{ lookup('file', lookup('env', 'HOME') + '/.ssh/id_ed25519.pub') }}" +# Workload clusters that get a cluster-inspector read-only kubeconfig (--tags inspector). +# Skipped while the cluster's -kubeconfig Secret doesn't exist. +inspector_workload_clusters: + - { name: lab1, namespace: default } + # Installed by the cluster-api-operator; CR namespace = install namespace (clusterctl defaults). capi_providers: - { kind: CoreProvider, name: cluster-api, namespace: capi-system, version: "{{ capi_version }}" } diff --git a/ansible/mgmt.yml b/ansible/mgmt.yml index 89daf3b..0943e49 100644 --- a/ansible/mgmt.yml +++ b/ansible/mgmt.yml @@ -259,3 +259,32 @@ become: false changed_when: false tags: [inspector] + + - name: Workload clusters that have a CAPI kubeconfig Secret + ansible.builtin.command: >- + kubectl --kubeconfig {{ lookup('env', 'HOME') }}/.kube/lab-mgmt.kubeconfig + -n {{ item.namespace }} get secret {{ item.name }}-kubeconfig -o name + loop: "{{ inspector_workload_clusters }}" + loop_control: + label: "{{ item.namespace }}/{{ item.name }}" + register: workload_kubeconfig_secrets + delegate_to: localhost + become: false + changed_when: false + failed_when: false + tags: [inspector] + + - name: Read-only kubeconfigs for workload clusters (skips clusters that don't exist yet) + ansible.builtin.command: >- + {{ playbook_dir }}/../.claude/cluster-inspector/make-kubeconfig.sh {{ item.item.name }} + --admin-kubeconfig {{ lookup('env', 'HOME') }}/.kube/lab-mgmt.kubeconfig + --capi-cluster {{ item.item.namespace }}/{{ item.item.name }} + --apply-rbac + loop: "{{ workload_kubeconfig_secrets.results }}" + loop_control: + label: "{{ item.item.namespace }}/{{ item.item.name }}" + when: item.rc == 0 + delegate_to: localhost + become: false + changed_when: false + tags: [inspector] diff --git a/docs/phase-1-proxmox-capi.md b/docs/phase-1-proxmox-capi.md index db86698..e6a846f 100644 --- a/docs/phase-1-proxmox-capi.md +++ b/docs/phase-1-proxmox-capi.md @@ -71,7 +71,8 @@ ansible-playbook mgmt.yml --tags vm # VM 201 on pv1-1, waits for SSH ansible-playbook mgmt.yml --tags k3s # qemu-guest-agent, k3s, node Ready ansible-playbook mgmt.yml --tags addons # cert-manager v1.21.2, cluster-api-operator 0.28.0 ansible-playbook mgmt.yml --tags capi # capmox-variables Secret + 5 Provider CRs, waits for controllers -ansible-playbook mgmt.yml --tags kubeconfig # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfig +ansible-playbook mgmt.yml --tags kubeconfig # ~/.kube/lab-mgmt.kubeconfig + cluster-inspector read-only kubeconfigs +ansible-playbook mgmt.yml --tags inspector # only the read-only kubeconfigs (lab-mgmt + inspector_workload_clusters); weekly token refresh ansible-playbook mgmt.yml # everything; must be changed=0 on a built cluster ```