PVE is now the 2-node cluster homelab (pv1-1 + pv1-2), no shared storage:
- template 101 copied to pv1-2 as 9101; MDs lab1-workers-pv1-1 (max 6) and
lab1-workers-pv1-2 (max 12), PMTs pinned with allowedNodes; pool .31-.50;
lab1-workers renamed for symmetry (decision #18, #7 superseded)
- CA: balancing-label node.cluster.x-k8s.io/pool (+ pool label, capacity labels
with os/arch, maxPods on the MDs), scan-interval 1m, v 2, default QPS (#19).
Scale test: 16 pods -> 4 + 4 across both nodes.
- runbook, image-builder copy procedure, fixes tables, future task to report
three CA clusterapi bugs upstream, lab facts, changelog, execution log
Plan: docs/plans/2026-09-29-2315-pv1-2-workers.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-check of lab1 after Steps 4-5 (inspector on lab-mgmt, read-only gets
on lab1):
- Cluster/KCP/MD Available for 13h; no controller errors since bootstrap.
- All six autoscaler annotations on lab1-workers (copied to the
MachineSet). spec.replicas has no owner: kubectl-client-side-apply owns
only clusterName/selector/template, so re-applying lab1.yaml won't
fight cluster-autoscaler.
- Nodes Ready on v1.35.9; Node/Machine providerIDs match.
- Cilium, envoy and CoreDNS healthy. No PDBs in the cluster.
Found: cilium-operator (replicas: 1) was scheduled onto the only worker.
It is a non-DaemonSet kube-system pod without a PDB, so with CA's default
skip-nodes-with-system-pods=true the last worker could never scale to
zero (Step 7). The chart already tolerates the control-plane taint, so a
nodeSelector on node-role.kubernetes.io/control-plane pins it there.
- configs/cilium/values.yaml: operator.nodeSelector.
- phase-1 doc: Step 5 ticked, Step 4 explains why replicas: 1 isn't
enough, new troubleshooting row.
- Session note extended with the re-check.
Not applied yet: the owner runs helm upgrade with the new values.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent could only reach lab-mgmt, so it couldn't verify nodes, CNI or
system pods on lab1. It correctly refused the admin ~/.kube/lab1.kubeconfig
when the owner's permission was relayed through the main session.
Workload kubeconfigs:
- make-kubeconfig.sh: new --capi-cluster [NS/]NAME. --admin-kubeconfig
is then the management cluster. The workload admin kubeconfig is read
from Secret NS/NAME-kubeconfig into a mktemp file, removed on exit, so
no admin file for the workload cluster lands on disk.
- ansible/mgmt.yml --tags inspector now also mints kubeconfigs for every
cluster in the new inspector_workload_clusters (group_vars: lab1).
Clusters without a <name>-kubeconfig Secret are skipped. The tag is now
the single weekly token refresh; tasks stay changed_when: false, so the
playbook's changed=0 rule still holds.
- Owner ran --tags inspector: kubeconfigs/lab1.kubeconfig (lab1-ro)
minted, can-i checks as expected.
Agent prompt:
- Cluster map: what to inspect on lab-mgmt vs the workload cluster.
Machine -> Node -> VM via nodeRef, providerID and
ProxmoxMachine.spec.virtualMachineID (checked on live objects).
- New triage step: scale-down blockers (non-DaemonSet kube-system pods
without PDB, local storage, bare pods, PDBs at 0, safe-to-evict /
scale-down-disabled).
- Refuses admin kubeconfigs even when a caller relays permission; points
the owner to --tags inspector on missing file or 401.
- Verb before --kubeconfig so commands match the kubectl get/describe/logs
allow rules; single-quote jsonpath/custom-columns for zsh.
- Report format: abnormal first, then health summary, then unverified.
settings.json: allow kubectl describe/logs/events and clusterctl describe.
All four are read-only, and the RO kubeconfigs enforce read-only anyway.
Checked: bash -n on the script; ansible-playbook --syntax-check and
ansible-lint --profile production pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipam.mode=kubernetes, operator.replicas=1 (default 2 would pin one on a
worker and block scale-to-zero). Keys checked against chart 1.20.2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- runbook + inspector agent: ipaddresses.ipam.cluster.x-k8s.io (plain
`ipaddresses` is the built-in networking.k8s.io type on k8s 1.35)
- inspector RBAC: get/list CRDs so clusterctl describe works read-only
- phase-1 Step 3 ticked; session note with the first lab1 review
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated with clusterctl v1.12.11 + the six runbook edits (pod CIDR,
kube-vip v1.2.4, no qcow2, VLAN 4, worker size, no MD replicas); passes
server-side dry run. Inflate test pods lowered to 1Gi to stay CPU-bound,
capacity annotation 4Gi, decision #15.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase-1 Step 2 rewritten around ansible/mgmt.yml and cluster-api-operator,
version pins for operator/cert-manager/cloud image, decisions #12-14,
execution log and changelog. Phase-1 Step 2 checked off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stringData is never stored, so kubectl apply reported "configured" on
every run and the capi play was never changed=0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PVE's cloud-init user-data triggers a deprecation warning that makes
`cloud-init status --wait` exit 2 even though provisioning succeeded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VM from Ubuntu cloud image on pv1-1, pinned k3s, cert-manager and
cluster-api-operator via k3s HelmChart CRs, CAPI providers as operator CRs,
kubeconfig fetch + cluster-inspector kubeconfig. Lint passes (production).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kacerrandClaude Opus 5.5