# Phase 2 — CAPI provider for vanilla KVM/libvirt hosts

Goal: autoscaled worker VMs on the Ubuntu 24.04 KVM/libvirt hypervisors, managed by the same
management cluster as phase 1. Status: **design / evaluation**.

## Problem

There is no mature, SIG-maintained CAPI infrastructure provider for plain libvirt. Proxmox does not
use libvirt (it drives QEMU via `qm`), so one libvirt provider cannot cover both hypervisor types.

## Options

| Option | How it works | Pros | Cons |
| --- | --- | --- | --- |
| **A. CAPLV** ([joshuagrisham/cluster-api-provider-libvirt](https://github.com/joshuagrisham/cluster-api-provider-libvirt), v0.2.1, CAPI v1.12.1) | talks libvirt RPC via pure-Go `digitalocean/go-libvirt` | same CAPI core as CAPMOX; direct fit for our hosts | young, single maintainer; feature gaps unknown (IPAM, placement, capacity) |
| **B. CAPK + KubeVirt** (CAPK v0.11.2 on CAPI v1.11.4, KubeVirt v1.9.0) | Ubuntu hosts become a Kubernetes "infra cluster" running KubeVirt; CAPK creates `VirtualMachine`s there | kubernetes-sigs project, mature, live migration, CDI for images | replaces plain libvirt management with a whole k8s + KubeVirt stack on the hypervisors; nested networking design (bridge/Multus) |
| **C. Metal3** (CAPM3 v1.14.1) + sushy-tools | VMs pre-defined in libvirt, exposed as "bare metal" via Redfish emulator, provisioned by Ironic | production-grade bare-metal flow | Ironic is heavy; VM pool is fixed → not real elasticity |
| **D. Own provider** | kubebuilder project, `LibvirtCluster/LibvirtMachine/LibvirtMachineTemplate` | full control, best learning value; can add multi-backend and `status.capacity` | we own maintenance |

## Suggested approach

1. **Spike A (≈ 1 week):** deploy CAPLV next to CAPMOX (both on CAPI 1.12), one MachineDeployment
   on one Ubuntu host. Check: cloud-init/bootstrap handling, IP assignment (DHCP vs CAPI IPAM),
   `providerID` format, delete/cleanup of volumes, behaviour under cluster-autoscaler.
2. If A is insufficient → **D**, reusing CAPLV's patterns, or contribute upstream.
3. Keep **B** as the alternative if we're willing to turn the Ubuntu hosts into a KubeVirt cluster
   (it may pay off together with phase 3).

## Own provider sketch (option D)

Reconcile of `LibvirtMachine`:

1. Wait for `Machine.spec.bootstrap.dataSecretName`.
2. Pick a host (placement: one MachineDeployment per host, or least-loaded by free memory).
3. Create a qcow2 overlay volume on the base image (backing file) in the host's storage pool.
4. Build a NoCloud ISO from the bootstrap secret (+ network-config from CAPI IPAM `IPAddressClaim`).
5. `virDomainDefineXML` + start; wait for IP (guest agent / DHCP leases).
6. Set `spec.providerID = libvirt://<host>/<domain-uuid>`, `status.addresses`, `ready`.
7. Finalizer: destroy domain, delete overlay + ISO.

`LibvirtMachineTemplate.status.capacity` (cpu/memory) → scale-from-zero without annotations.
Connection: `qemu+ssh://` or `qemu+tls://` per host, credentials in a Secret.

## Mixing providers in one workload cluster

`Cluster.spec.infrastructureRef` points to **one** InfraCluster kind, and most providers' machine
reconcilers look up *their own* InfraCluster kind through it — a `ProxmoxMachine` in a cluster whose
InfraCluster is a `LibvirtCluster` won't reconcile. Choices:

- **Separate workload clusters per provider** — simplest, no hacks.
- **Tolerant own provider** — our `LibvirtMachine` works regardless of the InfraCluster kind, so a
  CAPMOX-based cluster can also get libvirt worker MachineDeployments.
- **Multi-backend provider** — one `HypervisorCluster` kind with Proxmox API + libvirt backends.

cluster-autoscaler is indifferent: it scales any MachineDeployment carrying the min/max annotations,
whatever its infrastructure kind. The priority expander can prefer one hypervisor pool over another.

## Exit criteria

- [ ] Worker MachineDeployment on an Ubuntu KVM host joins a cluster managed from the phase-1 mgmt cluster.
- [ ] cluster-autoscaler scales it 0 → N → 0.
- [ ] VM and volumes fully cleaned up on scale-down.