Initial import: phase docs, configs, Claude conventions

Phase 1-3 runbooks and design notes, version pins, config templates, project-history conventions, cluster-inspector read-only credentials setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 00:58:52 +02:00
commit 5fd998c5f9
26 changed files with 1124 additions and 0 deletions
@@ -0,0 +1,62 @@
+# Phase 2 — CAPI provider for vanilla KVM/libvirt hosts
+
+Goal: autoscaled worker VMs on the Ubuntu 24.04 KVM/libvirt hypervisors, managed by the same
+management cluster as phase 1. Status: **design / evaluation**.
+
+## Problem
+
+There is no mature, SIG-maintained CAPI infrastructure provider for plain libvirt. Proxmox does not
+use libvirt (it drives QEMU via `qm`), so one libvirt provider cannot cover both hypervisor types.
+
+## Options
+
+| Option | How it works | Pros | Cons |
+| --- | --- | --- | --- |
+| **A. CAPLV** ([joshuagrisham/cluster-api-provider-libvirt](https://github.com/joshuagrisham/cluster-api-provider-libvirt), v0.2.1, CAPI v1.12.1) | talks libvirt RPC via pure-Go `digitalocean/go-libvirt` | same CAPI core as CAPMOX; direct fit for our hosts | young, single maintainer; feature gaps unknown (IPAM, placement, capacity) |
+| **B. CAPK + KubeVirt** (CAPK v0.11.2 on CAPI v1.11.4, KubeVirt v1.9.0) | Ubuntu hosts become a Kubernetes "infra cluster" running KubeVirt; CAPK creates `VirtualMachine`s there | kubernetes-sigs project, mature, live migration, CDI for images | replaces plain libvirt management with a whole k8s + KubeVirt stack on the hypervisors; nested networking design (bridge/Multus) |
+| **C. Metal3** (CAPM3 v1.14.1) + sushy-tools | VMs pre-defined in libvirt, exposed as "bare metal" via Redfish emulator, provisioned by Ironic | production-grade bare-metal flow | Ironic is heavy; VM pool is fixed → not real elasticity |
+| **D. Own provider** | kubebuilder project, `LibvirtCluster/LibvirtMachine/LibvirtMachineTemplate` | full control, best learning value; can add multi-backend and `status.capacity` | we own maintenance |
+
+## Suggested approach
+
+1. **Spike A (≈ 1 week):** deploy CAPLV next to CAPMOX (both on CAPI 1.12), one MachineDeployment
+   on one Ubuntu host. Check: cloud-init/bootstrap handling, IP assignment (DHCP vs CAPI IPAM),
+   `providerID` format, delete/cleanup of volumes, behaviour under cluster-autoscaler.
+2. If A is insufficient → **D**, reusing CAPLV's patterns, or contribute upstream.
+3. Keep **B** as the alternative if we're willing to turn the Ubuntu hosts into a KubeVirt cluster
+   (it may pay off together with phase 3).
+
+## Own provider sketch (option D)
+
+Reconcile of `LibvirtMachine`:
+
+1. Wait for `Machine.spec.bootstrap.dataSecretName`.
+2. Pick a host (placement: one MachineDeployment per host, or least-loaded by free memory).
+3. Create a qcow2 overlay volume on the base image (backing file) in the host's storage pool.
+4. Build a NoCloud ISO from the bootstrap secret (+ network-config from CAPI IPAM `IPAddressClaim`).
+5. `virDomainDefineXML` + start; wait for IP (guest agent / DHCP leases).
+6. Set `spec.providerID = libvirt://<host>/<domain-uuid>`, `status.addresses`, `ready`.
+7. Finalizer: destroy domain, delete overlay + ISO.
+
+`LibvirtMachineTemplate.status.capacity` (cpu/memory) → scale-from-zero without annotations.
+Connection: `qemu+ssh://` or `qemu+tls://` per host, credentials in a Secret.
+
+## Mixing providers in one workload cluster
+
+`Cluster.spec.infrastructureRef` points to **one** InfraCluster kind, and most providers' machine
+reconcilers look up *their own* InfraCluster kind through it — a `ProxmoxMachine` in a cluster whose
+InfraCluster is a `LibvirtCluster` won't reconcile. Choices:
+
+- **Separate workload clusters per provider** — simplest, no hacks.
+- **Tolerant own provider** — our `LibvirtMachine` works regardless of the InfraCluster kind, so a
+  CAPMOX-based cluster can also get libvirt worker MachineDeployments.
+- **Multi-backend provider** — one `HypervisorCluster` kind with Proxmox API + libvirt backends.
+
+cluster-autoscaler is indifferent: it scales any MachineDeployment carrying the min/max annotations,
+whatever its infrastructure kind. The priority expander can prefer one hypervisor pool over another.
+
+## Exit criteria
+
+- [ ] Worker MachineDeployment on an Ubuntu KVM host joins a cluster managed from the phase-1 mgmt cluster.
+- [ ] cluster-autoscaler scales it 0 → N → 0.
+- [ ] VM and volumes fully cleaned up on scale-down.