kacerrandClaude Opus 5.5
5fd998c5f9
Phase 1-3 runbooks and design notes, version pins, config templates, project-history conventions, cluster-inspector read-only credentials setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
63 lines
4.0 KiB
Markdown
63 lines
4.0 KiB
Markdown
# Phase 2 — CAPI provider for vanilla KVM/libvirt hosts
|
|
|
|
Goal: autoscaled worker VMs on the Ubuntu 24.04 KVM/libvirt hypervisors, managed by the same
|
|
management cluster as phase 1. Status: **design / evaluation**.
|
|
|
|
## Problem
|
|
|
|
There is no mature, SIG-maintained CAPI infrastructure provider for plain libvirt. Proxmox does not
|
|
use libvirt (it drives QEMU via `qm`), so one libvirt provider cannot cover both hypervisor types.
|
|
|
|
## Options
|
|
|
|
| Option | How it works | Pros | Cons |
|
|
| --- | --- | --- | --- |
|
|
| **A. CAPLV** ([joshuagrisham/cluster-api-provider-libvirt](https://github.com/joshuagrisham/cluster-api-provider-libvirt), v0.2.1, CAPI v1.12.1) | talks libvirt RPC via pure-Go `digitalocean/go-libvirt` | same CAPI core as CAPMOX; direct fit for our hosts | young, single maintainer; feature gaps unknown (IPAM, placement, capacity) |
|
|
| **B. CAPK + KubeVirt** (CAPK v0.11.2 on CAPI v1.11.4, KubeVirt v1.9.0) | Ubuntu hosts become a Kubernetes "infra cluster" running KubeVirt; CAPK creates `VirtualMachine`s there | kubernetes-sigs project, mature, live migration, CDI for images | replaces plain libvirt management with a whole k8s + KubeVirt stack on the hypervisors; nested networking design (bridge/Multus) |
|
|
| **C. Metal3** (CAPM3 v1.14.1) + sushy-tools | VMs pre-defined in libvirt, exposed as "bare metal" via Redfish emulator, provisioned by Ironic | production-grade bare-metal flow | Ironic is heavy; VM pool is fixed → not real elasticity |
|
|
| **D. Own provider** | kubebuilder project, `LibvirtCluster/LibvirtMachine/LibvirtMachineTemplate` | full control, best learning value; can add multi-backend and `status.capacity` | we own maintenance |
|
|
|
|
## Suggested approach
|
|
|
|
1. **Spike A (≈ 1 week):** deploy CAPLV next to CAPMOX (both on CAPI 1.12), one MachineDeployment
|
|
on one Ubuntu host. Check: cloud-init/bootstrap handling, IP assignment (DHCP vs CAPI IPAM),
|
|
`providerID` format, delete/cleanup of volumes, behaviour under cluster-autoscaler.
|
|
2. If A is insufficient → **D**, reusing CAPLV's patterns, or contribute upstream.
|
|
3. Keep **B** as the alternative if we're willing to turn the Ubuntu hosts into a KubeVirt cluster
|
|
(it may pay off together with phase 3).
|
|
|
|
## Own provider sketch (option D)
|
|
|
|
Reconcile of `LibvirtMachine`:
|
|
|
|
1. Wait for `Machine.spec.bootstrap.dataSecretName`.
|
|
2. Pick a host (placement: one MachineDeployment per host, or least-loaded by free memory).
|
|
3. Create a qcow2 overlay volume on the base image (backing file) in the host's storage pool.
|
|
4. Build a NoCloud ISO from the bootstrap secret (+ network-config from CAPI IPAM `IPAddressClaim`).
|
|
5. `virDomainDefineXML` + start; wait for IP (guest agent / DHCP leases).
|
|
6. Set `spec.providerID = libvirt://<host>/<domain-uuid>`, `status.addresses`, `ready`.
|
|
7. Finalizer: destroy domain, delete overlay + ISO.
|
|
|
|
`LibvirtMachineTemplate.status.capacity` (cpu/memory) → scale-from-zero without annotations.
|
|
Connection: `qemu+ssh://` or `qemu+tls://` per host, credentials in a Secret.
|
|
|
|
## Mixing providers in one workload cluster
|
|
|
|
`Cluster.spec.infrastructureRef` points to **one** InfraCluster kind, and most providers' machine
|
|
reconcilers look up *their own* InfraCluster kind through it — a `ProxmoxMachine` in a cluster whose
|
|
InfraCluster is a `LibvirtCluster` won't reconcile. Choices:
|
|
|
|
- **Separate workload clusters per provider** — simplest, no hacks.
|
|
- **Tolerant own provider** — our `LibvirtMachine` works regardless of the InfraCluster kind, so a
|
|
CAPMOX-based cluster can also get libvirt worker MachineDeployments.
|
|
- **Multi-backend provider** — one `HypervisorCluster` kind with Proxmox API + libvirt backends.
|
|
|
|
cluster-autoscaler is indifferent: it scales any MachineDeployment carrying the min/max annotations,
|
|
whatever its infrastructure kind. The priority expander can prefer one hypervisor pool over another.
|
|
|
|
## Exit criteria
|
|
|
|
- [ ] Worker MachineDeployment on an Ubuntu KVM host joins a cluster managed from the phase-1 mgmt cluster.
|
|
- [ ] cluster-autoscaler scales it 0 → N → 0.
|
|
- [ ] VM and volumes fully cleaned up on scale-down.
|