Initial import: phase docs, configs, Claude conventions
Phase 1-3 runbooks and design notes, version pins, config templates, project-history conventions, cluster-inspector read-only credentials setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -0,0 +1,173 @@
|
||||
# Phase 1 — Proxmox VMs via Cluster API + cluster-autoscaler
|
||||
|
||||
Goal: a manually installed k3s management cluster runs CAPI + CAPMOX and creates workload
|
||||
cluster `lab1` as Proxmox VMs; cluster-autoscaler scales its workers 0–5, including scale-from-zero.
|
||||
No custom code in this phase — only assembling upstream components.
|
||||
|
||||
Versions: see [versions.md](versions.md). Template build: see [image-builder-proxmox.md](image-builder-proxmox.md).
|
||||
|
||||
## Topology
|
||||
|
||||
| Role | Where | Size |
|
||||
| --- | --- | --- |
|
||||
| Management cluster | 1 VM on Proxmox, k3s v1.35.8+k3s1 | 2 vCPU, 4 GB RAM, 30 GB |
|
||||
| `lab1` control plane | 1 VM, VIP via kube-vip | 2 vCPU, 4 GB RAM |
|
||||
| `lab1` workers | MachineDeployment `lab1-workers`, min 0 / max 5 | 4 vCPU, 8 GB RAM |
|
||||
|
||||
## Checklist
|
||||
|
||||
- [x] Step 0 — Proxmox prep (API token, ACLs, bridge/VLAN, IP ranges)
|
||||
- [x] Step 1 — VM template `ubuntu-2404-kube-v1.35.9` built and smoke-tested
|
||||
- [ ] Step 2 — management cluster + `clusterctl init`
|
||||
- [ ] Step 3 — workload cluster `lab1`
|
||||
- [ ] Step 4 — CNI (Cilium)
|
||||
- [ ] Step 5 — MachineDeployment autoscaling annotations
|
||||
- [ ] Step 6 — cluster-autoscaler
|
||||
- [ ] Step 7 — scale tests
|
||||
|
||||
## Step 0 — Proxmox prep
|
||||
|
||||
```bash
|
||||
pveum user add capmox@pve
|
||||
pveum user token add capmox@pve capi -privsep 0 # token inherits user ACLs
|
||||
|
||||
pveum aclmod / -user capmox@pve -role PVEVMAdmin
|
||||
pveum aclmod / -user capmox@pve -role PVEAuditor # Sys.Audit etc.
|
||||
pveum aclmod /storage -user capmox@pve -role PVEDatastoreAdmin # ISO + cloud-init ISO upload, disks
|
||||
pveum aclmod /sdn/zones/localnetwork -user capmox@pve -role PVESDNUser # NIC on any local bridge (PVE 8+)
|
||||
```
|
||||
|
||||
Least-privilege variant (dedicated pools, specific bridge/VLAN): CAPMOX `docs/advanced-setups.md`.
|
||||
|
||||
Network: `vmbr0` is VLAN-aware, VMs live in VLAN 4. Reserve a node IP range outside DHCP and one VIP
|
||||
for the control plane endpoint in the same subnet.
|
||||
|
||||
Storage: `local-lvm` (LVM-thin) → disks must be **raw**, never qcow2. Cloning to another PVE node
|
||||
requires shared storage; with local storage keep `ALLOWED_NODES` to the node holding the template.
|
||||
|
||||
## Step 2 — management cluster
|
||||
|
||||
```bash
|
||||
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.8+k3s1 sh -s - --disable traefik
|
||||
mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config
|
||||
|
||||
curl -L -o clusterctl https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.11/clusterctl-linux-amd64
|
||||
sudo install -m 0755 clusterctl /usr/local/bin/clusterctl
|
||||
```
|
||||
|
||||
`~/.cluster-api/clusterctl.yaml` — start from [configs/capi/clusterctl.yaml.example](../configs/capi/clusterctl.yaml.example).
|
||||
|
||||
```bash
|
||||
clusterctl init \
|
||||
--core cluster-api:v1.12.11 \
|
||||
--bootstrap kubeadm:v1.12.11 \
|
||||
--control-plane kubeadm:v1.12.11 \
|
||||
--infrastructure proxmox:v0.9.1 \
|
||||
--ipam in-cluster:v1.1.1
|
||||
```
|
||||
|
||||
Always pin versions explicitly — without them clusterctl installs the newest core (1.14.x) and
|
||||
breaks the tested CAPMOX combination.
|
||||
|
||||
## Step 3 — workload cluster `lab1`
|
||||
|
||||
```bash
|
||||
clusterctl generate cluster lab1 \
|
||||
--infrastructure proxmox:v0.9.1 \
|
||||
--kubernetes-version v1.35.9 \
|
||||
--control-plane-machine-count 1 \
|
||||
--worker-machine-count 1 > lab1.yaml
|
||||
```
|
||||
|
||||
Edit `lab1.yaml` before applying:
|
||||
|
||||
1. **Pod CIDR** — `Cluster.spec.clusterNetwork.pods.cidrBlocks` defaults to `192.168.0.0/16`, which
|
||||
overlaps the lab network → use e.g. `10.244.0.0/16`.
|
||||
2. **kube-vip** — bump `ghcr.io/kube-vip/kube-vip:v0.7.1` → `v1.2.4`; keep `kube-vip-prepare.sh`
|
||||
(handles `super-admin.conf` for kubeadm ≥ 1.29). Fall back to the template's version if the VIP flakes.
|
||||
3. **Disk format** — delete `format: "qcow2"` from both `ProxmoxMachineTemplate`s (LVM-thin).
|
||||
4. **Network** — `bridge: vmbr0` + `vlan: 4` on the network device.
|
||||
5. **Worker size** — `ProxmoxMachineTemplate/lab1-worker`: `numSockets: 1`, `numCores: 4`, `memoryMiB: 8192`.
|
||||
6. **Replicas** — remove `spec.replicas` from `MachineDeployment/lab1-workers`; the autoscaler owns it.
|
||||
|
||||
```bash
|
||||
kubectl apply -f lab1.yaml
|
||||
clusterctl describe cluster lab1 --show-conditions all
|
||||
kubectl get proxmoxmachines,machines -w
|
||||
clusterctl get kubeconfig lab1 > lab1.kubeconfig
|
||||
```
|
||||
|
||||
Node IPs come from the IPAM pool (`kubectl get ipaddressclaims,ipaddresses`). providerID is set by
|
||||
the CAPMOX template as `proxmox://{{ ds.meta_data.instance_id }}` — don't touch it, the autoscaler
|
||||
matches Nodes to Machines by it.
|
||||
|
||||
## Step 4 — Cilium
|
||||
|
||||
```bash
|
||||
export KUBECONFIG=$PWD/lab1.kubeconfig
|
||||
helm repo add cilium https://helm.cilium.io && helm repo update
|
||||
helm install cilium cilium/cilium --version 1.20.2 -n kube-system --set ipam.mode=kubernetes
|
||||
```
|
||||
|
||||
`ipam.mode=kubernetes` uses the pod CIDR from `clusterNetwork` instead of Cilium's `10.0.0.0/8` default.
|
||||
|
||||
## Step 5 — autoscaling annotations
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
name: lab1-workers
|
||||
annotations:
|
||||
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
|
||||
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
|
||||
capacity.cluster-autoscaler.kubernetes.io/cpu: "4" # numSockets × numCores
|
||||
capacity.cluster-autoscaler.kubernetes.io/memory: "8Gi"
|
||||
capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "40Gi"
|
||||
capacity.cluster-autoscaler.kubernetes.io/maximum-pods: "110"
|
||||
```
|
||||
|
||||
Capacity annotations are required for scale-from-zero because CAPMOX does not publish
|
||||
`status.capacity`. They must match the `ProxmoxMachineTemplate`; add `.../labels` and `.../taints`
|
||||
annotations if pods target the group by selector/toleration.
|
||||
|
||||
## Step 6 — cluster-autoscaler
|
||||
|
||||
Install into the **same namespace as the `lab1-kubeconfig` secret** (here `default`), values in
|
||||
[configs/cluster-autoscaler/values.yaml](../configs/cluster-autoscaler/values.yaml):
|
||||
|
||||
```bash
|
||||
helm repo add autoscaler https://kubernetes.github.io/autoscaler && helm repo update
|
||||
helm install ca autoscaler/cluster-autoscaler --version 9.59.0 -n default \
|
||||
-f configs/cluster-autoscaler/values.yaml
|
||||
```
|
||||
|
||||
The log must show node group `MachineDeployment/default/lab1-workers` with min 0 / max 5.
|
||||
|
||||
## Step 7 — scale tests
|
||||
|
||||
Workload: [configs/test/inflate.yaml](../configs/test/inflate.yaml) (pause pods, 1500m / 2Gi each).
|
||||
|
||||
- [ ] Scale-up: `kubectl scale deploy/inflate --replicas=6` → workers 1 → 3; record pending → Running time.
|
||||
- [ ] Scale-down: `--replicas=0` → after `scale-down-unneeded-time` (5m) Machines and VMs disappear.
|
||||
- [ ] Scale-to-zero: last worker goes away (watch CoreDNS — see troubleshooting).
|
||||
- [ ] Scale-from-zero: `--replicas=2` → CA provisions from capacity annotations.
|
||||
|
||||
`kubectl describe pod <pending>` shows `TriggeredScaleUp` / `NotTriggerScaleUp` events — fastest diagnostic.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
| --- | --- | --- |
|
||||
| CA sees no node group | missing min/max annotations or wrong `autoDiscovery` | check MD annotations, clusterName/namespace in values |
|
||||
| CA never scales down | Node `providerID` ≠ Machine `providerID` | compare `kubectl get nodes -o custom-columns=N:.metadata.name,P:.spec.providerID` with `kubectl get machines` |
|
||||
| MD jumps back to old count | `spec.replicas` in manifest re-applied | remove `replicas` |
|
||||
| last worker won't go to 0 | kube-system pods without PDB (`skip-nodes-with-system-pods=true`) | PDB for CoreDNS or `skip-nodes-with-system-pods: false` |
|
||||
| scale-from-zero does nothing | capacity annotations missing/wrong | fix cpu/memory/maximum-pods |
|
||||
| clone fails `unsupported format 'qcow2'` | qcow2 on LVM-thin | remove `format` |
|
||||
| ProxmoxMachine stuck without IP | IPAM pool exhausted | `kubectl get ipaddressclaims`, widen range |
|
||||
| `SDN.Use` 403 | missing `PVESDNUser` | ACL on `/sdn/zones/localnetwork` |
|
||||
|
||||
```bash
|
||||
kubectl -n capmox-system logs deploy/capmox-controller-manager -f
|
||||
kubectl -n capi-system logs deploy/capi-controller-manager -f
|
||||
ssh <user>@<node-ip> 'cloud-init status --long; journalctl -u kubelet -n 100'
|
||||
```
|
||||
Reference in New Issue
Block a user