kacerrandClaude Opus 5.5
5fd998c5f9
Phase 1-3 runbooks and design notes, version pins, config templates, project-history conventions, cluster-inspector read-only credentials setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4.0 KiB
Phase 2 — CAPI provider for vanilla KVM/libvirt hosts
Goal: autoscaled worker VMs on the Ubuntu 24.04 KVM/libvirt hypervisors, managed by the same management cluster as phase 1. Status: design / evaluation.
Problem
There is no mature, SIG-maintained CAPI infrastructure provider for plain libvirt. Proxmox does not
use libvirt (it drives QEMU via qm), so one libvirt provider cannot cover both hypervisor types.
Options
| Option | How it works | Pros | Cons |
|---|---|---|---|
| A. CAPLV (joshuagrisham/cluster-api-provider-libvirt, v0.2.1, CAPI v1.12.1) | talks libvirt RPC via pure-Go digitalocean/go-libvirt |
same CAPI core as CAPMOX; direct fit for our hosts | young, single maintainer; feature gaps unknown (IPAM, placement, capacity) |
| B. CAPK + KubeVirt (CAPK v0.11.2 on CAPI v1.11.4, KubeVirt v1.9.0) | Ubuntu hosts become a Kubernetes "infra cluster" running KubeVirt; CAPK creates VirtualMachines there |
kubernetes-sigs project, mature, live migration, CDI for images | replaces plain libvirt management with a whole k8s + KubeVirt stack on the hypervisors; nested networking design (bridge/Multus) |
| C. Metal3 (CAPM3 v1.14.1) + sushy-tools | VMs pre-defined in libvirt, exposed as "bare metal" via Redfish emulator, provisioned by Ironic | production-grade bare-metal flow | Ironic is heavy; VM pool is fixed → not real elasticity |
| D. Own provider | kubebuilder project, LibvirtCluster/LibvirtMachine/LibvirtMachineTemplate |
full control, best learning value; can add multi-backend and status.capacity |
we own maintenance |
Suggested approach
- Spike A (≈ 1 week): deploy CAPLV next to CAPMOX (both on CAPI 1.12), one MachineDeployment
on one Ubuntu host. Check: cloud-init/bootstrap handling, IP assignment (DHCP vs CAPI IPAM),
providerIDformat, delete/cleanup of volumes, behaviour under cluster-autoscaler. - If A is insufficient → D, reusing CAPLV's patterns, or contribute upstream.
- Keep B as the alternative if we're willing to turn the Ubuntu hosts into a KubeVirt cluster (it may pay off together with phase 3).
Own provider sketch (option D)
Reconcile of LibvirtMachine:
- Wait for
Machine.spec.bootstrap.dataSecretName. - Pick a host (placement: one MachineDeployment per host, or least-loaded by free memory).
- Create a qcow2 overlay volume on the base image (backing file) in the host's storage pool.
- Build a NoCloud ISO from the bootstrap secret (+ network-config from CAPI IPAM
IPAddressClaim). virDomainDefineXML+ start; wait for IP (guest agent / DHCP leases).- Set
spec.providerID = libvirt://<host>/<domain-uuid>,status.addresses,ready. - Finalizer: destroy domain, delete overlay + ISO.
LibvirtMachineTemplate.status.capacity (cpu/memory) → scale-from-zero without annotations.
Connection: qemu+ssh:// or qemu+tls:// per host, credentials in a Secret.
Mixing providers in one workload cluster
Cluster.spec.infrastructureRef points to one InfraCluster kind, and most providers' machine
reconcilers look up their own InfraCluster kind through it — a ProxmoxMachine in a cluster whose
InfraCluster is a LibvirtCluster won't reconcile. Choices:
- Separate workload clusters per provider — simplest, no hacks.
- Tolerant own provider — our
LibvirtMachineworks regardless of the InfraCluster kind, so a CAPMOX-based cluster can also get libvirt worker MachineDeployments. - Multi-backend provider — one
HypervisorClusterkind with Proxmox API + libvirt backends.
cluster-autoscaler is indifferent: it scales any MachineDeployment carrying the min/max annotations, whatever its infrastructure kind. The priority expander can prefer one hypervisor pool over another.
Exit criteria
- Worker MachineDeployment on an Ubuntu KVM host joins a cluster managed from the phase-1 mgmt cluster.
- cluster-autoscaler scales it 0 → N → 0.
- VM and volumes fully cleaned up on scale-down.