diff --git a/CHANGELOG.md b/CHANGELOG.md index 4fe773f..e82db70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,21 @@ # Changelog +## 2026-09-08 11:00 UTC — KVM/libvirt provider + +- New `kvm` provider type: Managed proxies as VMs on your own libvirt/KVM hosts. Each + host is reached over SSH with the libvirt unix socket forwarded through the + connection (pure Go, nothing to expose on the hypervisors beyond sshd); one provider + instance can span several hosts, selected by `placement.zone`. +- Base images (`images:` map or a direct URL in `placement.image`) are downloaded by the + operator into each host's storage pool on first use, verified by sha256 when configured, + and cached there. Per-VM disks are qcow2 overlays; cloud-init arrives on a NoCloud seed + ISO built in-process. GC identity lives in the domain's `` element. +- Wiring: `"kvm": kvm.New` in `cmd/main.go`; optional `kvm-ssh` Secret mounted at + `/etc/proxy-operator/kvm`; samples `config/samples/proxy_kvm.yaml` and a `kvm-home` + block in `providers-config.yaml`; `docs/kvm-provider.md`; opt-in integration test + (`make test-integration-kvm`) against a real hypervisor. +- Not yet validated on a real host — that is the next step on the branch. + ## 2026-08-10 09:34 CEST — kind e2e verified; fix Squid OOM in containers; quickstart goes in-cluster - Full end-to-end pass on a throwaway kind cluster: Squid pod Ready with a real CONNECT diff --git a/README.md b/README.md index a3a80dd..8dc4df5 100644 --- a/README.md +++ b/README.md @@ -15,9 +15,10 @@ itself**, and hands out to crawler clients via an HTTP list/lease API. cattle**: any meaningful spec change (placement, cloud-init, port) deletes and recreates the VM — never in-place mutation. - **Providers** behind one minimal interface: `kubernetes` (a real Squid - pod in this cluster — local dev/CI) and `gcp` (Compute Engine VMs with - ephemeral external IPs — the real egress fleet). Config is a YAML file - (`--providers-config`) with named instances (`gcp-eu`, `gcp-us`, ...). + pod in this cluster — local dev/CI), `gcp` (Compute Engine VMs with + ephemeral external IPs — the real egress fleet) and `kvm` (VMs on your + own libvirt hypervisors, reached over SSH). Config is a YAML file + (`--providers-config`) with named instances (`gcp-eu`, `kvm-home`, ...). - **Health engine** probes every proxy by fetching a URL *through* it (a real CONNECT tunnel — a proxy that accepts TCP but can't egress goes Unhealthy), with threshold logic and transition-only status writes. @@ -131,6 +132,29 @@ make deploy IMG=/egress-proxies-operator:dev which also installs Squid via cloud-init). A missing field fails the Proxy with a message naming it. +## KVM setup + +Managed proxies as VMs on libvirt/KVM hosts you own. Full guide: +[docs/kvm-provider.md](docs/kvm-provider.md). + +1. Each host needs `libvirtd`, an SSH user in the `libvirt` group, a + bridge to a network with DHCP, and a storage pool. +2. Create the `kvm-ssh` Secret (private key + `known_hosts`); the manager + mounts it at `/etc/proxy-operator/kvm`: + + ```sh + kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \ + --from-file=id_ed25519= --from-file=known_hosts= + ``` + +3. Add a `kvm` entry to the providers config with `images`, + `machineTypes` and `hosts` (see + [config/samples/providers-config.yaml](config/samples/providers-config.yaml)). + Base images are downloaded into each host's pool on first use. +4. A Proxy names a host (`placement.zone`, optional with one host), a + size (`placement.machineType`) and an image (`placement.image`) — see + [config/samples/proxy_kvm.yaml](config/samples/proxy_kvm.yaml). + Cloud-init from a Secret: the Secret **must** carry the label `crawl.example.com/cloud-init: "true"` — the operator's cache only holds labelled Secrets, so an unlabelled one is invisible (the Proxy reports diff --git a/cmd/main.go b/cmd/main.go index 2b85b59..18d76cf 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -58,6 +58,7 @@ import ( "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/gcp" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kubernetes" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/registry" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/version" @@ -187,6 +188,7 @@ func main() { "gcp": func(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, error) { return gcp.NewWithWireOptions(ctx, pc, gcp.WireLogOptions{FullPayloads: gcpWireFullPayloads}) }, + "kvm": kvm.New, }) if err != nil { setupLog.Error(err, "Failed to build providers") diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml index 5676abb..e115f10 100644 --- a/config/manager/manager.yaml +++ b/config/manager/manager.yaml @@ -145,9 +145,22 @@ spec: - name: providers-config mountPath: /etc/proxy-operator readOnly: true + # SSH material for kvm providers (sshKeyFile / knownHostsFile in the + # providers config point here). Optional: the mount is empty when the + # Secret does not exist, and only kvm entries need it. Create it with: + # kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \ + # --from-file=id_ed25519= \ + # --from-file=known_hosts= + - name: kvm-ssh + mountPath: /etc/proxy-operator/kvm + readOnly: true volumes: - name: providers-config configMap: name: providers-config + - name: kvm-ssh + secret: + secretName: kvm-ssh + optional: true serviceAccountName: controller-manager terminationGracePeriodSeconds: 10 diff --git a/config/manager/providers_config.yaml b/config/manager/providers_config.yaml index 6c383c0..916ac14 100644 --- a/config/manager/providers_config.yaml +++ b/config/manager/providers_config.yaml @@ -9,8 +9,9 @@ metadata: data: # Mounted at /etc/proxy-operator/providers.yaml (--providers-config). # The default ships only the kubernetes-pod provider so the operator runs - # out of the box; add gcp entries (type: gcp, gcp.project: ...) for real - # egress fleets — see config/samples/providers-config.yaml. + # out of the box; add gcp entries (type: gcp, gcp.project: ...) or kvm + # entries (type: kvm, hosts reached over SSH) for real egress fleets — + # see config/samples/providers-config.yaml. providers.yaml: | providers: - name: kubernetes diff --git a/config/samples/kustomization.yaml b/config/samples/kustomization.yaml index d1c025f..4545440 100644 --- a/config/samples/kustomization.yaml +++ b/config/samples/kustomization.yaml @@ -4,5 +4,6 @@ resources: - proxy_kubernetes.yaml - proxy_gcp.yaml +- proxy_kvm.yaml - proxy_external.yaml # +kubebuilder:scaffold:manifestskustomizesamples diff --git a/config/samples/providers-config.yaml b/config/samples/providers-config.yaml index 34220d8..af9e575 100644 --- a/config/samples/providers-config.yaml +++ b/config/samples/providers-config.yaml @@ -19,3 +19,36 @@ providers: # diskSizeGb: 10 # auth: Application Default Credentials (workload identity # in-cluster, gcloud ADC locally). No key-file plumbing. + - name: kvm-home + type: kvm + kvm: + # SSH to each host with the libvirt unix socket forwarded through the + # connection; nothing to open on the hypervisors beyond sshd. Both + # files come from the optional kvm-ssh Secret mounted by + # config/manager/manager.yaml — see docs/kvm-provider.md. + sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 + knownHostsFile: /etc/proxy-operator/kvm/known_hosts + # insecureIgnoreHostKey: true # skip host-key checks (not recommended) + # diskSizeGb: 10 # per-VM overlay; cloud-init grows the root fs into it + # vendorData: | # default installs qemu-guest-agent for IP discovery + # #cloud-config + # ... + images: + # placement.image names a key here (or is an https:// URL directly). + # Fetched into each host's pool on first use and cached there as + # base-.qcow2; do not change an entry while Proxies use it. + debian-13: + url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 + # sha256: "<64 hex>" # optional; verified after download + ubuntu-24.04: + url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img + machineTypes: # placement.machineType names a key here + small: {vcpus: 1, memoryMiB: 1024} + medium: {vcpus: 2, memoryMiB: 2048} + hosts: # placement.zone names a host; optional with one host + - name: nas1 + address: nas1.home.lan:22 + user: virt # must be in the libvirt group + bridge: br0 # to a LAN with DHCP; the VM's IP comes from there + # socket: /var/run/libvirt/libvirt-sock + # pool: default diff --git a/config/samples/proxy_kvm.yaml b/config/samples/proxy_kvm.yaml new file mode 100644 index 0000000..e84e4d6 --- /dev/null +++ b/config/samples/proxy_kvm.yaml @@ -0,0 +1,43 @@ +# A Managed proxy backed by a libvirt/KVM hypervisor on the home network: +# the operator boots a stock Debian cloud image as a VM on the named host, +# with a NIC on the host's bridge (IP from the LAN's DHCP), and installs +# Squid via cloud-init. Requires a providers-config entry named "kvm-home" +# (see providers-config.yaml) and the kvm-ssh Secret (docs/kvm-provider.md). +# +# placement.zone is the host name from kvm.hosts (optional when only one +# host is configured); machineType and image are keys in the provider's +# machineTypes and images maps. The first Proxy for an image on a host +# waits for the image download; later ones reuse the cached copy. +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: proxy-kvm-sample +spec: + mode: Managed + provider: kvm-home + placement: + zone: nas1 + machineType: small + image: debian-13 + port: 3128 + cloudInit: + inline: | + #cloud-config + packages: + - squid + write_files: + - path: /etc/squid/conf.d/proxy-operator.conf + content: | + http_port 3128 + http_access allow all + via off + forwarded_for off + runcmd: + - systemctl restart squid + # Optional: log in with `ssh debian@` (the Debian image's default + # user) to look around; `virsh console ` on the host works too. + # ssh_authorized_keys: + # - ssh-ed25519 AAAA... you@laptop + attributes: + geo: home + purpose: crawl diff --git a/docs/architecture.md b/docs/architecture.md index 5c1053b..328c069 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -9,6 +9,7 @@ | Provider contract | `internal/provider` | library | — | `Provider` interface, error taxonomy, deterministic naming, config, metrics decorator | | kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) | | gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops | +| kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds | | Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events | | Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex | | Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` | @@ -147,6 +148,25 @@ prov.ListByTag ─► AggregatedList(label filter, ─┘ prov ReturnPartialSuccess: true) providerID = zones//instances/ — zone-qualified, so Get/Delete stay correct even mid-replacement after a zone edit + +kvm provider (internal/provider/kvm/) — libvirt RPC over an SSH-forwarded unix socket, per host +prov.Create ──► DomainLookupByName: exists → start if needed, done ─┐ every step tolerates + else: ensure base image in pool (download on first use) │ the previous attempt + overlay volume (exists = reuse) │ having got that far; + seed ISO (always rebuilt — a cut-off upload must │ the seed is the one + never be trusted) │ thing redone + DomainDefineXML + DomainCreate ─┘ +prov.Get ──► DomainGetState + DomainGetMetadata + DomainInterfaceAddresses + (ARP, then guest agent; first routable IPv4) + running without IP = Provisioning; nostate = Provisioning +prov.Delete ──► destroy (not running = fine) → undefine → delete overlay + seed by name +prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose + carries managed=true; one host down fails the sweep, + one unreadable domain is skipped +providerID = hosts//domains/; GC identity lives in the domain's + element (libvirt has no labels). All RPCs run under a context- +bounded call: go-libvirt never times out on its own, so a cancelled reconcile +closes the SSH connection to unblock it. ``` The reconciler never watches provider-side resources (Pods or GCP VMs). @@ -415,6 +435,30 @@ they read as choices, not accidents. Chronological by build step. - **Unknown GCP instance statuses map to `Stopped`:** the reconciler's answer to Stopped is delete-and-recreate, the always-safe move for cattle when the API grows a new state. +- **KVM hosts are reached over SSH with the libvirt socket forwarded,** + not libvirtd's TCP/TLS listener: nothing to configure or expose on the + hypervisors beyond sshd, pure Go on the operator side (no cgo, no + libvirt libraries in the image), and the hosts list can grow by a + config edit because the providerID carries the host name. +- **KVM base images are fetched by the operator into the host's pool** + through libvirt's upload stream, rather than expected to be pre-placed: + the operator then needs no filesystem access on the host and a new + host is usable with zero manual staging. The cost is a synchronous + first-time download inside Create. +- **The KVM seed ISO is rebuilt on every Create attempt** before the + domain exists, instead of "exists → skip": a reconcile cancelled mid + upload leaves a truncated seed, and the compensating delete cannot run + because the connection was already closed to unblock the RPC. Skipping + it would boot a VM with no cloud-init and no retry would ever repair it. +- **KVM `nostate` maps to `Provisioning`, not `Stopped`:** unlike GCP's + unknown statuses, `nostate` means libvirt could not determine the + state at all; answering that with delete-and-recreate is too drastic, + so the next poll decides. +- **A KVM domain GC cannot read is skipped, a host it cannot reach fails + the sweep.** GC never deletes what it does not see, so leaving one + domain out is safe; one corrupt domain blocking GC for every other one + is not. A whole host missing from the view would hide many, so that + still aborts. - **Kubernetes 403s classify as `ErrPermanent`** even though quota exhaustion also surfaces as 403 (indistinguishable from RBAC denial in `apierrors`): not hammering an API server that may never allow the diff --git a/docs/kvm-provider.md b/docs/kvm-provider.md new file mode 100644 index 0000000..c79899e --- /dev/null +++ b/docs/kvm-provider.md @@ -0,0 +1,214 @@ +# The KVM provider + +Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. The +operator reaches each host over SSH, forwards the libvirt unix socket +through that connection, and talks libvirt's own RPC over it — so the +hosts need nothing beyond `sshd` and `libvirtd`, and the operator image +carries no libvirt libraries. VMs boot a stock distro cloud image that the +operator downloads into the host's storage pool on first use, with the +Proxy's cloud-init delivered on a NoCloud seed ISO built in-process. + +## Hypervisor prerequisites + +Per host, once: + +1. **libvirt + KVM** (`libvirtd` running, `/dev/kvm` present). +2. **An SSH user in the `libvirt` group**, so it may open + `/var/run/libvirt/libvirt-sock`. The operator authenticates with a + private key; nothing else on the host is touched. +3. **A bridge to a network with DHCP** (e.g. `br0` bridged onto the LAN). + Each VM gets one virtio NIC on it and takes its IP from that network's + DHCP; the cluster must be able to reach those IPs for health probes + and for crawlers to use the proxy. +4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`) + with room for one base image per distro (a few hundred MB) plus the + per-VM overlays (`diskSizeGb` each, thin-provisioned). +5. Optionally, **outbound internet from the VMs**: the default + vendor-data installs `qemu-guest-agent` from the distro mirrors on + first boot. Without it the VM still works — the operator falls back to + the host's ARP table to find the IP, which fills in once the guest + sends traffic. + +Check from the operator's point of view before wiring it in: + +```sh +ssh virt@nas1.home.lan virsh -c qemu:///system list --all # user can drive libvirt +ssh virt@nas1.home.lan virsh -c qemu:///system pool-info default +ssh virt@nas1.home.lan ip link show br0 +``` + +## Cluster-side setup + +1. Create the SSH Secret. `known_hosts` must contain every host's key + (`ssh-keyscan -H nas1.home.lan >> known_hosts`); without it, set + `insecureIgnoreHostKey: true` and accept that a MITM could hand you a + fake hypervisor. + + ```sh + kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \ + --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \ + --from-file=known_hosts=./known_hosts + ``` + + `config/manager/manager.yaml` mounts it (optional) at + `/etc/proxy-operator/kvm`; the deployment starts fine without it as + long as no `kvm` provider is configured. + +2. Add a provider entry to the providers config + (`config/manager/providers_config.yaml` in-cluster; full sample in + `config/samples/providers-config.yaml`): + + ```yaml + - name: kvm-home + type: kvm + kvm: + sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 + knownHostsFile: /etc/proxy-operator/kvm/known_hosts + images: + debian-13: + url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 + machineTypes: + small: {vcpus: 1, memoryMiB: 1024} + hosts: + - name: nas1 + address: nas1.home.lan:22 + user: virt + bridge: br0 + ``` + + The operator reads the key and known_hosts at startup and refuses to + start if either is missing — a bad path fails fast, not on the first + Proxy. + +3. Apply a Proxy (`config/samples/proxy_kvm.yaml`) and watch it: + + ```sh + kubectl get px -w + # NAME MODE PROVIDER PHASE IP HEALTHY + # proxy-kvm-sample Managed kvm-home Provisioning + # proxy-kvm-sample Managed kvm-home Running 192.168.1.73 + # proxy-kvm-sample Managed kvm-home Ready 192.168.1.73 True + ``` + + The first Proxy for a given image on a given host sits in + `Provisioning` for the length of the image download; the operator log + shows `fetching base image into host pool` and `base image cached` + with the size and duration. + +## Config reference + +| Field | Required | Default | Meaning | +|---|---|---|---| +| `sshKeyFile` | yes | | private key used for every host | +| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against | +| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely | +| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it | +| `vendorData` | | installs `qemu-guest-agent` | cloud-init vendor-data placed on every seed next to the Proxy's user-data | +| `images..url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) | +| `images..sha256` | | | expected digest; verified after download, a change triggers a re-download | +| `images..format` | | `qcow2` | `qcow2` or `raw` | +| `machineTypes.` | ≥ 1 | | `{vcpus, memoryMiB}` size profile | +| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique | +| `hosts[].address` | yes | | SSH endpoint, `host:port` | +| `hosts[].user` | yes | | SSH user, member of the `libvirt` group | +| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host | +| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds | +| `hosts[].bridge` | yes | | host bridge for the VM's NIC | + +## Placement + +| Proxy field | Meaning for kvm | +|---|---| +| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required. Unknown → `Failed` | +| `placement.machineType` | key in `machineTypes`. Unknown → `Failed` | +| `placement.image` | key in `images`, or an `http(s)://` URL used directly. Anything else → `Failed` | +| `placement.region` | ignored | + +`Failed` here means the reconciler stops retrying and the message names +the offending value; fix the spec (which replaces the VM) or the config. + +## What lands on the host + +For a Proxy whose deterministic name is `proxy-abc…`, on the chosen +host's pool: + +| Object | Name | Lifetime | +|---|---|---| +| base image | `base-.qcow2` (or `base-.qcow2` for a URL) | cached; survives Proxy deletion | +| image marker | `base-.qcow2.sha256` | with the base image | +| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base | with the Proxy | +| seed ISO | `proxy-abc…-seed.iso` | with the Proxy | +| domain | `proxy-abc…` | with the Proxy | + +The domain is a q35 machine with host-passthrough CPU, the overlay on +virtio, the seed on a SATA cdrom, one virtio NIC on the bridge, a serial +console, the guest-agent channel, a virtio RNG and no graphics. Its +`` carries the operator's element (managed flag, owning UID, +creation time) — that is how orphan GC recognises its own VMs, since +libvirt has no labels. + +## The image cache + +The base image is downloaded **by the operator, through libvirt**, into +the host's pool the first time a Proxy asks for it on that host — nothing +is pre-placed on the hypervisor, and the operator needs no filesystem +access there. The bytes stream from the URL through the operator into +the pool with a running sha256; nothing is buffered. The marker volume is +written last and holds the digest, so a crash mid-download leaves a base +volume without a marker, which the next attempt deletes and redoes. + +A re-download happens when: the marker is missing, or `sha256` is +configured and differs from the marker's content. Otherwise the cached +copy is used for as long as it exists, even if the URL's content changed +upstream (a `latest` URL is pinned at the moment of first fetch, per +host). + +To evict manually: + +```sh +ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256 +ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2 +``` + +⚠️ **Do not change an image entry's `url` or `sha256`, and do not evict +its base volume, while Proxies using that image exist on the host.** The +overlays back onto the base file; deleting or replacing it corrupts every +VM that uses it. Delete those Proxies first, or add the new image under a +new key and move Proxies over by changing their `placement.image` (which +replaces the VM). + +## Logging into a VM + +- Put your key in the Proxy's cloud-init (`ssh_authorized_keys`) and + `ssh debian@` (Debian) / `ssh ubuntu@` (Ubuntu). +- Or on the host: `virsh console proxy-abc…` attaches to the serial + console; cloud-init's output scrolls by there on first boot. +- `virsh domifaddr proxy-abc… --source arp` / `--source agent` shows what + the operator sees when it looks for the IP. + +## Known gaps + +- **No quota or capacity detection.** libvirt reports an out-of-space + pool or an over-committed host as an ordinary error; the reconciler + treats it as transient and retries with backoff rather than the slower + quota backoff. +- **Orphan volumes without a domain are invisible to GC.** A crash + between creating the volumes and defining the domain leaves an overlay + and seed that `ListByTag` cannot see (it lists domains). They are + cleaned when the same Proxy reconciles again (Delete removes volumes by + name even with no domain), but a Proxy CR that is gone for good leaves + them for `virsh vol-list` to find. +- **The first image fetch is synchronous.** It holds a reconcile worker + for the download (LAN speed, a few hundred MB, once per image per + host). Other Proxies keep reconciling on the remaining workers. +- **Replacing an image in use breaks running VMs** — see the warning + under *The image cache*. +- **IP discovery needs either guest traffic or the agent.** A VM on a + bridge with no DHCP never gets an address and stays `Provisioning` + forever; check `virsh console`. + +## Testing + +Unit tests run over an in-memory fake of the libvirt API. The optional +integration test against a real host is described in +[testing.md](testing.md#the-kvm-integration-test-real-hypervisor). diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index 215080a..a256fa4 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -8,7 +8,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md - [x] Step 3 — Transport + errors (`conn.go`, `errors.go`) - [x] Step 3b — Image cache (`image.go`) - [x] Step 4 — Provider methods (`kvm.go`) over a fake API -- [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG +- [x] Step 5 — Wiring, manifests, samples, docs, CHANGELOG - [x] Step 6 — Integration test against a real hypervisor (written and compiled; first real run is part of Step 7) - [ ] Step 7 — Manual validation + MR @@ -338,3 +338,54 @@ Step 3b "confirm on a real host" notes get answered. The test's config is built directly rather than through the provider package's validation (unexported), so a typo in an env var surfaces from `New` or from libvirt, not from a validation message. + +## Step 5 — Wiring, manifests, samples, docs, CHANGELOG + +Done after Step 6 (see that entry for why the order flipped). Everything +the plan listed, plus a few things it did not: + +- `cmd/main.go`: `"kvm": kvm.New` in the builtins map. No RBAC change — + the provider talks to hypervisors, not the API server. +- `config/manager/manager.yaml`: `kvm-ssh` Secret mounted read-only at + `/etc/proxy-operator/kvm` with `optional: true`, so deployments without + a kvm provider are unaffected. It nests under the providers-config + mount at `/etc/proxy-operator`; kustomize renders it (`kubectl + kustomize config/default` checked). The container runs as uid 65532 + and Secret files default to 0644, so the key is readable without an + `fsGroup`; the `kubectl create secret` recipe is in a comment next to + the mount, in `docs/kvm-provider.md` and in the README. +- `config/samples/providers-config.yaml`: `kvm-home` block with Debian 13 + and Ubuntu 24.04 official image URLs, two machine types, one host, and + every optional field shown commented out. Parsed through + `provider.LoadConfigFile` in a throwaway test to make sure the sample + validates. `config/samples/proxy_kvm.yaml` uses the GCP sample's squid + cloud-init and shows `ssh_authorized_keys` commented out rather than + set — a sample with a placeholder key would either fail cloud-init or + invite pasting a real key into a committed file. Added to the samples + kustomization; `config/manager/providers_config.yaml`'s comment now + mentions kvm. +- `docs/kvm-provider.md`: prerequisites with a three-line preflight + (`virsh list`, `pool-info`, `ip link` over ssh), cluster setup, the full + config reference table, placement table, a table of what lands on the + host and its lifetime, how the image cache works and how to evict, the + **warning not to change or evict an image while Proxies use it** (the + reviewer's finding from Step 4, carried as a documented rule since the + content-addressed-naming alternative was not chosen), how to log in, + and the known gaps. +- README: providers bullet and a "KVM setup" section mirroring the GCP + one. `docs/architecture.md`: components row, a §5 block for the kvm + call flow in the same style as the other two, and five decision-log + entries (SSH transport, operator-fetched images, seed always rebuilt, + `nostate` → Provisioning, skip-vs-fail in the GC sweep). +- `CHANGELOG.md` entry at the top, stating explicitly that the provider + is not yet validated on a real host. + +Verification: `go build ./...`, `go vet ./...`, `go test -short` on +`cmd` and `internal/provider/...`, and both kustomizations rendered. +`hack/providers-dev.yaml` untouched, as planned. + +Worth noting: the CRD has no rule requiring all three placement fields +(that is GCP's own check), so the single-host "zone may be omitted" +behaviour works without an API change. And with this step the branch is +feature-complete but **unproven**: Step 7 runs the integration test and +the end-to-end checklist against a real hypervisor before the MR.