From fcbf8ddc91b7ef6ffb1db418100d13c7d4aaccac Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 16 Sep 2026 19:54:25 +0000 Subject: [PATCH] kvm2: user guide, appendix column and section, architecture and README entries Co-Authored-By: Claude Fable 5.1 --- README.md | 5 + docs/architecture.md | 91 ++- docs/kvm2-provider.md | 597 ++++++++++++++++++ .../2026-09-16-1333-kvm2-provider.md | 22 +- docs/providers-existing.md | 335 ++++++++-- docs/providers.md | 8 +- 6 files changed, 985 insertions(+), 73 deletions(-) create mode 100644 docs/kvm2-provider.md diff --git a/README.md b/README.md index 832b2f1..40876df 100644 --- a/README.md +++ b/README.md @@ -160,6 +160,10 @@ Managed proxies as VMs on libvirt/KVM hosts you own. Full guide: size (`placement.machineType`) and an image (`placement.image`) — see [config/samples/proxy_kvm.yaml](config/samples/proxy_kvm.yaml). +The `kvm2` provider uses the same Secret and the same block shape (`type: kvm2`, +block `kvm2:`), so the steps above apply to it unchanged; its guide is +[docs/kvm2-provider.md](docs/kvm2-provider.md). + Cloud-init from a Secret: the Secret **must** carry the label `crawl.example.com/cloud-init: "true"` — the operator's cache only holds labelled Secrets, so an unlabelled one is invisible (the Proxy reports @@ -187,6 +191,7 @@ can be two differently-configured instances of the same `gcp` type (see | `kubernetes` | A real Squid pod (`ubuntu/squid:6.6-24.04_edge` by default) in the same cluster the operator runs in | `image` (optional) | Needs no cloud account — local dev, CI, and the kind quickstart. Pods share the cluster's egress IP, so it exercises the full lifecycle but not distinct egress paths. | | `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. | | `kvm` | A libvirt VM on one of your own hypervisors, reached over SSH | `sshKeyFile`, `knownHostsFile` (unless `insecureIgnoreHostKey`), `machineTypes`, `hosts` (required); `images`, `diskSizeGb`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress. The base image is downloaded into each host's pool on first use; `Create` waits for the VM to start. See [docs/kvm-provider.md](docs/kvm-provider.md). | +| `kvm2` | The same: a libvirt VM on one of your own hypervisors, reached over SSH | the same block shape as `kvm`, under `kvm2:` | The second-generation libvirt provider, built from the provider contract. Adds a child span per libvirt RPC, a pool-space precheck reported as quota (the reconciler waits instead of failing), SSH credentials re-read on every dial, and image downloads and health checks on connections of their own. It does not adopt `kvm`'s VMs. See [docs/kvm2-provider.md](docs/kvm2-provider.md). | ### Adding a provider diff --git a/docs/architecture.md b/docs/architecture.md index f1a9703..2e9e3d7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -10,6 +10,7 @@ | kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) | | gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops | | kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds | +| kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial | | Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events | | Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex | | Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` | @@ -168,6 +169,39 @@ providerID = hosts//domains/; GC identity lives in the domain's element (libvirt has no labels). All RPCs run under a context- bounded call: go-libvirt never times out on its own, so a cancelled reconcile closes the SSH connection to unblock it. + +kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract +prov.Create ──► placement checked against the catalogue (Permanent before any dial) + DomainLookupByName: exists → uid guard (foreign uid = Permanent), ─┐ every step + start if needed, wait for running, done │ idempotent, + else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a + + 16 MiB must fit, else ErrQuotaExceeded (reconciler │ per-name lock + waits 5m) │ shared with + ensure base image: marker present = cached; otherwise GET │ Delete + streamed into StorageVolUpload on a DEDICATED connection, │ + StoragePoolRefresh, marker written last │ + overlay volume (exists = reuse) │ + seed ISO (always rebuilt) │ + DomainDefineXML ( in kvm2's own namespace: │ + managed, uid, createdAt) + DomainCreate, poll until │ + running (2m cap) ─┘ +prov.Get ──► DomainLookupByName + DomainGetMetadata + DomainGetState + + DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4) + running without IP = Provisioning; nostate = Provisioning; + only ErrNoDomain is NotFound +prov.Delete ──► destroy (not running = fine) → undefine (nvram, managed save, snapshots) + → delete overlay + seed by name; unconfigured host = logged no-op +prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose kvm2 + says managed=true; one host down fails the sweep, + one unreadable domain is skipped +Check ──► per host, on a FRESH connection: version, hostname, pool lookup, + pool info, close; never the pooled one +providerID = hosts//domains/. Every RPC has its own budget (dial 10s, +RPC 30s, DomainCreate 90s, seed upload 60s, image transfer 20m): a call that +outlives its ctx or budget kills the pooled connection (the only way to unblock +go-libvirt), so other calls in flight on that host fail once with Transient and +retry; the next call redials. Each RPC is a child span kvm2. under +provider.*, plus kvm2.dial and kvm2.download. ``` The reconciler never watches provider-side resources (Pods or GCP VMs). @@ -343,6 +377,15 @@ incoming trace context): - `provider.create|get|delete|list` (client spans, `provider.WithTracing` decorator wired outermost around `WithMetrics`); the GCP SDK's own otelhttp transport contributes HTTP child spans automatically. +- kvm2 child spans: under `provider.*`, one `kvm2.` client span per + libvirt RPC (for example `kvm2.DomainDefineXML`, attributes `kvm2.host`, + `kvm2.domain`, `kvm2.pool`, `kvm2.volume`, `kvm2.libvirt_code` on + failure), `kvm2.dial` around the SSH handshake and libvirt connect, and + `kvm2.download` around an image fetch; under `providerhealth.check`, + `kvm2.dial` and the four `kvm2.` spans of a check. They come + from a span decorator over the seam, applied once in the dialer, so no + RPC is unspanned. The `libvirt call` log line of an RPC is written + inside its span. The `kvm` provider has no child spans. - Kubernetes API calls become client child spans via a wrapped `rest.Config` transport — gated on an existing parent span, so informer list/watch long-polls and leader-election renewals never create root @@ -368,11 +411,12 @@ the slog handler's ctx. Answers "can this cluster reach each provider's backend right now?" without waiting for a Proxy to fail. A provider opts in by implementing `provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per -target and never fails as a whole. All three providers do: +target and never fails as a whole. All four providers do: | Provider | Target | What one check does | |---|---|---| | kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) | +| kvm2 | each `kvm2.hosts` entry | on a fresh SSH+libvirt connection dialled for the check and closed after it: libvirt version and hostname, then the configured pool must exist and be running; `details` carry `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB` and `poolAvailableGiB` ([details](kvm2-provider.md#health-check)) | | kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on | | gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` | @@ -613,3 +657,48 @@ they read as choices, not accidents. Chronological by build step. - **A provider name that is not a valid object name** gets metrics but no `ProviderHealth` object, with a warning at start. Tightening config validation instead would crash-loop existing deployments on upgrade. +- **kvm2 is a second provider type built clean-room from the contract,** + not a rewrite of `kvm` in place. Its spec + (`docs/kvm2-provider-spec.md`) and code were written from + [provider-contract.md](provider-contract.md), the kvm user guide and + go-libvirt alone, never from `internal/provider/kvm/`, as a test of + whether the contract is enough to build a provider. Both types ship, + share the `KVMConfig` block shape and the `kvm-ssh` Secret, and may + share a host. +- **kvm2 image downloads run on a dedicated connection.** A cancelled or + overlong call kills the connection it ran on (the only way to unblock + go-libvirt), so a 435 MB upload on the pooled connection could die + because a poll for another Proxy was cancelled. The transfer dials its + own connection and closes it afterwards; the cheap checks and the marker + stay on the pooled one. +- **kvm2 health checks dial fresh connections**, the opposite of the kvm + decision above. With credentials re-read on every dial a fresh + connection does pick up a rotated Secret, and a check that runs out of + time then closes only its own connection, so it cannot disrupt a + reconcile. The cost is one sshd session per host per round. +- **A cancelled kvm2 call kills the pooled connection.** Every RPC has a + budget; when a call outlives its ctx or budget the adapter closes the + socket and the SSH client, returns Transient, and marks the connection + dead so the next call redials. Other calls in flight on that host fail + once with Transient and their callers retry. Documented in the guide + rather than avoided: the alternative, leaving a hung RPC blocked + forever, pins a worker. +- **kvm2 marks domains in its own metadata namespace**, so it never lists + or adopts domains the `kvm` provider made. Switching an entry from `kvm` + to `kvm2` means deleting its Proxies first; the two types can share a + host without one collecting the other's VMs. +- **kvm2 turns pool free space into the quota signal.** libvirt reports a + full pool as an ordinary error, which would retry on the fast backoff. + Before creating any volume, `Create` compares `StoragePoolGetInfo`'s + available bytes with the image (if uncached) plus `diskSizeGb` plus 16 + MiB for the seed, and returns `ErrQuotaExceeded` below that, so the + reconciler waits 5 minutes instead. +- **kvm2 classifies SSH authentication and host-key failures as + Permanent.** A key missing from `authorized_keys` or a host key missing + from or different in known_hosts needs an admin; kvm retried these + forever. A missing key or known_hosts file stays Transient, because the + optional Secret may land later. +- **kvm2 re-reads the SSH key and known_hosts on every dial**, so rotating + the Secret needs no restart; nothing is cached. The constructor only + checks that the files exist, to warn early without failing the provider + set. diff --git a/docs/kvm2-provider.md b/docs/kvm2-provider.md new file mode 100644 index 0000000..2c46c4e --- /dev/null +++ b/docs/kvm2-provider.md @@ -0,0 +1,597 @@ +# The kvm2 provider + +Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. It is the +second-generation libvirt provider, built clean-room from +[provider-contract.md](provider-contract.md) and the design in +[kvm2-provider-spec.md](kvm2-provider-spec.md), and it ships side by side +with the first-generation `kvm` provider: both types can be configured at +once, both take the same config block shape, and both read the same +`kvm-ssh` Secret. What kvm2 adds over kvm: a child span per libvirt RPC, +a pool-space precheck that maps to the reconciler's quota wait, SSH +credentials re-read on every dial, image downloads and health checks on +connections of their own, an adoption guard on the owning UID, and SSH +authentication failures that stop retrying. + +Like kvm, the operator reaches each host over SSH, forwards the libvirt +unix socket through that connection and speaks libvirt's own RPC over it, +so the hosts need nothing beyond `sshd` and `libvirtd` and the operator +image carries no libvirt libraries. VMs boot a stock distro cloud image +that the operator downloads into the host's storage pool on first use, +with the Proxy's cloud-init delivered on a NoCloud seed ISO built +in-process. + +## Hypervisor prerequisites + +Per host, once: + +1. **libvirt and KVM** (`libvirtd` running, `/dev/kvm` present). +2. **An SSH user in the `libvirt` group**, so it may open + `/var/run/libvirt/libvirt-sock`. The operator authenticates with a + private key; nothing else on the host is touched. The socket is used + as `qemu:///system`, so no further host permissions are needed. +3. **A bridge to a network with DHCP** (for example `br0` bridged onto + the LAN). Each VM gets one virtio NIC on it and takes its IP from that + network's DHCP. The operator's pod network and the crawler clients + must route to those IPs: the health engine probes `http://:` + through the proxy and clients get the same address from the discovery + API. Nothing on the host filters the port; the provider does not use + `spec.port` at all. +4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`) + with room for one base image per distro (a few hundred MB) plus the + per-VM overlays (`diskSizeGb` each, thin-provisioned). The pool must be + running: a stopped pool fails the health check with `PoolNotActive` and + fails `Create` permanently. +5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu, + `edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default with secure boot + off (libvirt picks the OVMF binary itself); a domain start fails with + "no firmware found" without it. Only `firmware: bios` images skip this. +6. **Outbound internet from the VMs**, in practice: a vendor-data script + installs `qemu-guest-agent` from the distro mirrors on first boot, and + the agent is how the operator learns the VM's address. Without it the + operator falls back to the host's ARP table, which only sees traffic + addressed to the host itself, so on a plain bridge a VM without the + agent stays `Provisioning`. + +Check from the operator's point of view before wiring it in: + +```sh +ssh novakj@beelink virsh -c qemu:///system list --all # user can drive libvirt +ssh novakj@beelink virsh -c qemu:///system pool-info default +ssh novakj@beelink ip link show br0 +``` + +## Cluster-side setup + +1. Create the SSH Secret from a key pair dedicated to the operator and a + `known_hosts` file holding every host's key. It is the same `kvm-ssh` + Secret the `kvm` provider uses; one Secret serves both types, and if + you already have it there is nothing to do here. + + ```sh + # once: a key pair for the operator; the .pub goes on every host + ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator + ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub novakj@beelink + + # once per host: its host key, under the address the *operator* will + # dial (hosts[].address), which may differ from what your laptop uses + ssh-keyscan -H beelink.tail8c69c.ts.net >> ~/.ssh/kvm_known_hosts + + # the Secret: the PRIVATE key (not the .pub) plus known_hosts + kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \ + --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \ + --from-file=known_hosts=$HOME/.ssh/kvm_known_hosts + ``` + + `make kvm-ssh-secret` runs that last command as an apply, so it also + updates an existing Secret; override the paths with + `KVM_SSH_KEY=... KVM_KNOWN_HOSTS=...`. Putting the `.pub` in the + Secret is the classic mistake: the target's `kvm-ssh-secret` check + refuses a file without `PRIVATE KEY` in it, and the provider itself + would fail every call permanently with a "parsing ssh key file" error. + Without `known_hosts`, set `insecureIgnoreHostKey: true` and accept + that a MITM could hand you a fake hypervisor. + + `config/manager/manager.yaml` mounts the Secret (optional) at + `/etc/proxy-operator/kvm` as two files, `id_ed25519` and + `known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the + provider config point at. The deployment starts fine without the + Secret, even with a kvm2 provider configured: `New` only checks that + the files exist and logs a `WARNING:` line when one does not; the + provider's proxies fail transiently and recover once the Secret lands. + + **Rotating the key or a host key needs no restart.** Both files are + read and parsed on every dial (`loadCredentials` in + `internal/provider/kvm2/ssh.go`); nothing is cached. Re-run the + command or the make target, and the next connection, health check + included, uses the new material. Connections that are already open + stay open until they drop. + +2. Add a provider entry to the providers config + (`config/manager/providers_config.yaml` in-cluster; full sample in + `config/samples/providers-config.yaml`). The type is `kvm2` and the + block is `kvm2:`; its fields are those of the `kvm` block: + + ```yaml + - name: kvm2-home + type: kvm2 + kvm2: + sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 + knownHostsFile: /etc/proxy-operator/kvm/known_hosts + images: + debian-13: + url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 + machineTypes: + small: {vcpus: 1, memoryMiB: 1024} + hosts: + - name: beelink + address: beelink.tail8c69c.ts.net:22 + user: novakj + bridge: br0 + ``` + + Validation messages name the block, so a mistake reads + `kvm2.hosts[0]: bridge is required`, not `kvm.hosts`. The providers + config is read once per process: adding a host or an image entry needs + a manager restart; credentials do not. + +3. Apply a Proxy (`config/samples/proxy_kvm2.yaml`) and watch it: + + ```sh + kubectl get px -w + # NAME MODE PROVIDER PHASE IP HEALTHY + # proxy-kvm2-sample Managed kvm2-home Provisioning + # proxy-kvm2-sample Managed kvm2-home Running 192.168.1.73 + # proxy-kvm2-sample Managed kvm2-home Ready 192.168.1.73 True + ``` + + The first Proxy for a given image on a given host sits in + `Provisioning` for the length of the image download; the operator log + shows `fetching base image into host pool` with the URL and size and + `base image cached` with the size, duration and digest. + +## Config reference + +The block is `kvm2:` and its Go type is the shared `KVMConfig` in +`internal/provider/config_kvm.go`, so the fields, defaults and validation +are the same as for `kvm`. Defaults are applied by `withDefaults` in +`internal/provider/kvm2/kvm2.go`. + +| Field | Required | Default | Meaning | +|---|---|---|---| +| `sshKeyFile` | yes | | private key used for every host; OpenSSH or PEM, parsed on every dial | +| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against, read on every dial | +| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely; `New` logs a `WARNING:` line when it is on | +| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it. Also what the pool-space precheck reserves per VM | +| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on | +| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table | +| `images.` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only: it becomes the cached volume name `base-.` | +| `images..url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw). The server must send `Content-Length` | +| `images..sha256` | | | expected digest, 64 hex characters; verified after download, a change triggers a re-download | +| `images..format` | | `qcow2` | `qcow2` or `raw` | +| `images..firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition | +| `machineTypes.` | at least one | | `{vcpus, memoryMiB}` size profile, both greater than zero | +| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique. Letters, digits, dots, dashes and underscores only: it becomes part of the providerID `hosts//domains/` | +| `hosts[].address` | yes | | SSH endpoint, `host:port` | +| `hosts[].user` | yes | | SSH user, member of the `libvirt` group | +| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host | +| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds | +| `hosts[].bridge` | yes | | host bridge for the VM's NIC | + +## Placement + +All three fields are checked before any host is dialled. An unknown value +is `ErrPermanent`, which the reconciler latches as `Failed` with the +message below until the spec changes. + +| Proxy field | Meaning for kvm2 | Error when unknown | +|---|---|---| +| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required | `placement.zone "x" is not in kvm2.hosts`; with several hosts and no zone, `placement.zone is required when kvm2.hosts lists more than one host` | +| `placement.machineType` | key in `machineTypes` | `placement.machineType "x" is not in kvm2.machineTypes` | +| `placement.image` | key in `images`, or an `http(s)://` URL used directly with the defaults (`qcow2`, `uefi`) | `placement.image "x" is neither a key in kvm2.images nor an http(s) URL` | +| `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | | + +The messages come from `Create` in `internal/provider/kvm2/create.go`, +`host` in `kvm2.go` and `resolveImage` in `image.go`. Fix the spec (which +replaces the VM) or the config. Catalogue edits affect only new VMs: the +spec hash covers the placement values, not what the config maps them to. + +## What lands on the host + +For a Proxy whose deterministic name is `proxy-abc…`, on the chosen +host's pool: + +| Object | Name | Lifetime | +|---|---|---| +| base image | `base-.` (for a URL image, `` is the first 16 hex characters of the URL's sha256) | cached; survives Proxy deletion | +| image marker | `base-..sha256`, a 64-byte raw volume holding the digest | with the base image | +| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base file, `diskSizeGb` capacity | with the Proxy | +| seed ISO | `proxy-abc…-seed.iso`, raw | with the Proxy | +| domain | `proxy-abc…` | with the Proxy | + +The names are built in `internal/provider/kvm2/volume.go`. + +The domain (`buildDomainXML` in `xml.go`) is a `q35` machine with a +host-passthrough CPU, ACPI and APIC, memory in MiB and vcpus from the +machine type, the overlay on `vda` (virtio) and the seed as a read-only +SATA cdrom on `sda`, one virtio NIC on the bridge, a serial console on a +pty, the guest-agent channel `org.qemu.guest_agent.0`, a virtio RNG fed +from `/dev/urandom`, and no graphics. Both disks are `type="file"` with +the paths `StorageVolGetPath` returned: AppArmor derives QEMU's profile +from those paths, and volume-type disks fail with permission denied on +stock Debian and Ubuntu hosts. UEFI images get `` with +the `secure-boot` feature disabled; BIOS images get neither. + +Its `` carries one element in kvm2's own namespace, +`https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2` +(`metadataURI` in `metadata.go`): + +```xml + + true + 0f3a… + 2026-09-16T17:42:11Z + +``` + +That is how orphan GC recognises its own VMs, since libvirt has no +labels, and how a repeat `Create` knows whether an existing domain of the +same name belongs to this Proxy: a different non-empty `uid` is a +permanent error naming both UIDs; an empty one is adopted. The namespace +is kvm2's own, so the `kvm` provider's domains are invisible to it (see +*Migrating from kvm*). + +The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled +`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte +for byte**: any format cloud-init accepts works, including a Jinja +template, a MIME archive, a shell script or a gzipped payload; the +operator never parses or re-renders it. `meta-data` holds +`instance-id: ` and `local-hostname: `, so a reboot is never +taken for a new instance. `vendor-data` carries the guest-agent +bootstrap. The ISO is staged in a temp directory under `/tmp` (the +manager's 64Mi emptyDir), assembled in memory (tens of KiB) and uploaded +as a raw volume; the staging directory is removed before the call +returns. The seed is rebuilt on every `Create` attempt, because a +previous attempt may have been cut off mid-upload and a truncated seed +would boot a VM without cloud-init. + +The guest agent rides along in `vendor-data` as a **shell script** +(`guestAgentScript` in `seed.go`), not a cloud-config, on purpose: +cloud-init merges a vendor-data cloud-config under the user-data with +mergers that keep the user-data's value for every top-level key both +define and never append lists, so a Proxy carrying its own `packages:` +would silently lose the agent. Scripts bypass the merge. The script is +idempotent, picks whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk` +the image has, enables the service with `systemctl` or OpenRC, and logs +into `/var/log/cloud-init-output.log`. With `vendorData` also configured, +the two travel as separate parts of a MIME multipart (`text/plain` for +your document, `text/x-shellscript` for the script), with the +`MIME-Version` header first so cloud-init detects it. With +`installGuestAgent: false` the file is your `vendorData` alone, or empty. + +## The image cache + +The base image is downloaded **by the operator, through libvirt**, into +the host's pool the first time a Proxy asks for it on that host. Nothing +is pre-placed on the hypervisor and the operator needs no filesystem +access there. The bytes stream from the URL through the operator into +`StorageVolUpload` with a running sha256; nothing is buffered +(`fetchImage` and `uploadImage` in `internal/provider/kvm2/image.go`). + +The upload runs on a **dedicated connection** to the host, dialled for +the transfer and closed after it, not on the pooled connection the other +operations share. A cancelled or overlong call kills the connection it +ran on (see *Timeouts and connections*), and a poll for some other Proxy +being cancelled must not cut off a 20-minute upload. + +The download needs a `Content-Length`: the volume's capacity is declared +before the upload starts. The request sends `Accept-Encoding: identity` +so a transparently gzipped body cannot hide the length. A response +without it, or with length 0, is a permanent error naming the URL, and so +is any 4xx status; 429 and 5xx are transient and retried. + +The marker volume is written **last**, on the pooled connection, after +`StoragePoolRefresh` has made libvirt read the file's real format. Its +presence is what marks a base complete: a crash mid-download leaves a +base volume without a marker, which the next attempt deletes and redoes +(`cachedImage`). A re-download also happens when the marker's content is +not a 64-character hex digest, or when `sha256` is configured and differs +from it. Otherwise the cached copy is used for as long as it exists, even +if the URL's content changed upstream: a `latest` URL is pinned at the +moment of first fetch, per host. Concurrent Creates for the same image +on the same host queue on a per-(host, volume) lock; the first fetches, +the rest find the marker. + +To evict manually: + +```sh +ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256 +ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2 +``` + +**Warning: do not change an image entry's `url` or `sha256`, and do not +evict its base volume, while Proxies using that image exist on the +host.** The overlays back onto the base file; deleting or replacing it +corrupts every VM that uses it. Delete those Proxies first, or add the +new image under a new key and move Proxies over by changing their +`placement.image` (which replaces the VM). + +## Quota: the pool-space precheck + +Before creating any volume, `Create` reads `StoragePoolGetInfo` and +compares the pool's free space with what it is about to commit +(`checkPoolSpace` and `neededBytes` in `image.go`): the image's +`Content-Length` when the base is not cached yet (found with a `HEAD` +request, or a `GET` whose body is closed at once when the server refuses +`HEAD`), plus `diskSizeGb` in bytes for the overlay, plus 16 MiB for the +seed. Below that it returns `ErrQuotaExceeded` with a message naming the +pool, the bytes available and the bytes needed. The reconciler answers +quota with a wait, not a failure: the Proxy shows `Provisioned=False` +with reason `QuotaExceeded` and is retried after `QuotaRetry` (5 +minutes), off the error backoff curve and never `Failed`. Free up the +pool or lower `diskSizeGb` and the next attempt goes through. + +The precheck is the only quota signal; libvirt itself reports a full pool +as an ordinary error, which would be transient. The health check exposes +the same number as `poolAvailableGiB`, so a pool that is filling up shows +there before Creates start waiting. + +A pool that is not running fails the precheck permanently (`storage pool + is not running`): start it with `virsh pool-start` and +`virsh pool-autostart`. + +## Migrating from `kvm` + +The `kvm2` block is the `kvm` block under another name, so an entry ports +by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The +`kvm-ssh` Secret is shared; nothing changes on the hosts. + +What does not port is the VMs. kvm2 marks its domains in its own metadata +namespace and neither lists nor adopts domains the `kvm` provider made: +`ListByTag` skips them, so orphan GC never sees them, and a Proxy's +`spec.provider` is immutable, so an existing Proxy cannot be moved to the +new entry anyway. Do it in this order: + +1. Delete the Proxies on the `kvm` entry and wait for them to go; the + `kvm` provider removes their domains and volumes. +2. Change the entry (or add a second one under a new name and keep the + old one for a while; both types may share a host). +3. Restart the manager, then recreate the Proxies with the new + `spec.provider`. + +A `kvm` domain left behind on a host is invisible to kvm2 forever; clean +it up on the host with `virsh undefine --nvram ` and +`virsh vol-delete` for its `.qcow2` and `-seed.iso`. + +## Logging + +The provider logs through controller-runtime's logger, named `kvm2`, with +`provider` (the configured name) on every line, and `host` and `name` +(the domain) on every line about an instance. Every line inside a traced +operation also carries `traceID` and `spanID`. + +- `--zap-log-level=debug` (logr V(1)): + - one line per operation outcome: `kvm2 domain ensured running` with the + `id`, `kvm2 domain fetched` with `state`, `ip` and `uid`, + `kvm2 domain not found`, `kvm2 domain deleted`, `kvm2 domains listed` + with `hosts` and `managed` counts, `kvm2 host checked` with `ready`, + `reason` and `latency`; plus the steps inside a Create + (`kvm2 domain already defined`, `overlay volume created`, + `seed volume uploaded` with `bytes`, `domain defined` with `uid` and + `firmware`, `domain started`, `fetching base image into host pool`, + `base image cached`); + - **one `libvirt call` line per libvirt RPC**, written by the decorator + in `internal/provider/kvm2/observe.go` inside that RPC's span: `rpc` + (the go-libvirt method name, for example `DomainLookupByName`), the + identifiers it was given (`domain`, `pool`, `volume`), `duration`, and + on failure `error` plus `libvirtCode` and `libvirtError` (the code's + name, for example `ErrNoDomain`). A Get is four or five RPCs; a + Create with a cached image is around twenty; + - connection events from `internal/provider/kvm2/hosts.go`: + `libvirt host connected` with `address`, `user` and `duration`, and + `libvirt host connect failed` with the same plus `error` and `stage` + (`credentials`, `tcp`, `ssh` or `socket`); + - `kvm2 backend call failed` with `op` and the raw `error` once per + failed operation, before the error is returned; + - a line per domain that `ListByTag` skips, and a line when the image + marker forces a refetch. +- `--zap-log-level=2`: the `libvirt call` lines also carry the domain and + volume `xml`, the `metadata` element, the interface `addresses`, the + listed `domains`, the pool `state` and sizes and the volume `path`; the + `domain defined` line carries the `xml`; `kvm2 create resolved` logs the + resolved parameters (pool, bridge, machine type, sizes, image volume, + firmware) and `cloudInitBytes`; `kvm2 listed instance` logs each listed + domain. +- Never logged at any level: the cloud-init content, the SSH key, the + known_hosts content, and upload or download bodies. The seed and image + uploads report `offset` and `length` only. There is no raw wire tier + like the GCP provider's V(5); go-libvirt exposes no hook for it. + +## Tracing + +With an OTLP exporter configured ([architecture.md +§10](architecture.md#10-tracing-internaltracing)), every backend call is +a child span of the operation span the decorators open +(`provider.create`, `provider.get`, `provider.delete`, `provider.list`) +or of the monitor's `providerhealth.check`: + +| Span | Around | Attributes | +|---|---|---| +| `kvm2.`, for example `kvm2.DomainDefineXML` | one libvirt RPC (`observe.go`) | `kvm2.host`, and the identifiers passed: `kvm2.domain`, `kvm2.pool`, `kvm2.volume`; on failure the status is Error and `kvm2.libvirt_code` carries the libvirt error number | +| `kvm2.dial` | the SSH handshake and the libvirt connect (`hosts.go`) | `kvm2.host`, `kvm2.address`, `kvm2.user` | +| `kvm2.download` | one image fetch, from the HTTP request to the marker (`image.go`) | `kvm2.host`, `kvm2.volume`, `kvm2.url_host`, `kvm2.sha256_configured`, and `kvm2.bytes` once the response arrived | + +All spans are client spans. No attribute carries cloud-init, key material +or a URL beyond its host. The `libvirt call` log line for an RPC is +written inside that RPC's span, so its `spanID` points at the span. + +## Timeouts and connections + +No caller puts a deadline on the context, so every backend call carries +its own budget (`defaultBudgets` in `internal/provider/kvm2/backend.go`, +applied by `call`): + +| Call | Budget | +|---|---| +| TCP connect plus SSH handshake plus libvirt connect | 10 s | +| ordinary RPC | 30 s | +| `DomainCreate` | 90 s | +| `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s | +| `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier | +| wait for running after `DomainCreate` | 2 m, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`) | +| the `HEAD` request of the pool-space precheck | 30 s | +| the courtesy `ConnectClose` when a connection is closed | 5 s | + +go-libvirt's calls take no context and block, so the only way to abandon +one is to close the connection under it. That is what a cancelled or +overlong call does: the adapter kills the connection (socket and SSH +client), returns `ErrTransient` wrapping the context or deadline error, +and marks the connection dead so the next call redials. The consequence: +**other RPCs in flight on that host's pooled connection fail once with +`ErrTransient`** and their callers retry on a fresh connection. In the +reconciler that is one requeue with backoff; in GC it is one skipped +sweep. + +Two callers are kept off the pooled connection for that reason +(`dialFresh` in `hosts.go`): the image upload, because a 435 MB transfer +must not die because a poll for another Proxy was cancelled, and the +health check, because a check that runs out of the monitor's timeout must +close only its own connection. + +Otherwise there is one pooled SSH plus libvirt connection per host, +dialled lazily on the first call that needs it and shared by every +operation for that host (go-libvirt multiplexes RPCs on one connection). +Concurrent callers wait for one dial and share the result; waiting ends +with the caller's context. A `keepalive@openssh.com` request every 15 s +marks the connection dead when it fails or gets no reply within another +15 s, as does go-libvirt noticing the socket is gone; the next call +redials. Create and Delete for the same domain name are serialised on a +per-name lock, so two Deletes (GC and the reconciler) never race on +destroy and undefine, and a Delete cannot interleave with a Create's +steps; Get and ListByTag take no lock. + +## Health check + +The operator checks every configured host every `--provider-check-interval` +(default 1m) and publishes the result in the `ProviderHealth` object named +after the provider (`kubectl get ph kvm2-home -o yaml`) and in the +`proxy_operator_provider_target_up{provider,target}` gauge. One target per +`hosts[]` entry, in config order, all checked concurrently. + +Each check dials a **fresh connection** to the host and closes it when +done (`probeHost` in `internal/provider/kvm2/check.go`); the pooled +connection is never touched, so a check that runs out of time cannot fail +a reconcile's call in flight, and a fresh dial picks up a rotated Secret +at once. The sequence is the SSH and libvirt handshakes, then four RPCs: +`ConnectGetLibVersion`, `ConnectGetHostname`, `StoragePoolLookupByName` +and `StoragePoolGetInfo` on the host's `pool`. The cost per host per +round is one sshd session and those four calls. + +Every target's `details` carry `user` and `pool`. As facts arrive they +gain `libvirtVersion` (decoded `major.minor.micro`), `hostname`, +`poolState` (virsh's word: `running`, `inactive`, ...), +`poolCapacityGiB` and `poolAvailableGiB`, so a target that fails late in +the sequence still says what the host answered before. The free space is +the number the quota precheck compares against. + +A failing target says where the check broke (`checkReason` in `check.go`): + +| Reason | Meaning | Look at | +|---|---|---| +| `Reachable` | ready | | +| `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host | +| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path | +| `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) | +| `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group | +| `Timeout` | the connection was up, but the monitor's `--provider-check-timeout` (default 15s) or an RPC's budget ran out before the check finished | a hung `libvirtd` or a saturated link | +| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` | +| `PoolNotActive` | the pool exists but is not running | `virsh pool-start ` and `virsh pool-autostart ` | +| `APIError` | any other libvirt error | the message carries libvirt's own text | + +## Error classes + +Every error is classified once, in `classify` in +`internal/provider/kvm2/errors.go`, with errors that carry a class from +their source (validation, the image fetch, the dial) keeping it. The +reconciler latches `ErrPermanent` as `Failed` until the spec changes, +waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from +`Get`, and retries everything else with backoff. + +| Class | What lands there | +|---|---| +| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one | +| `ErrQuotaExceeded` | the pool-space precheck | +| `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create | +| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring | + +Error strings name paths, hosts, volumes and libvirt's own text. They +never contain key bytes, the known_hosts content or cloud-init. + +## Known gaps + +- **Create blocks**, including the first image download on each host, + holding one of the three reconcile workers for its duration (LAN + speed, a few hundred MB, once per image per host). Other Proxies keep + reconciling on the remaining workers. Warm the cache with one Proxy per + image per host before creating many. +- **Volumes without a domain are invisible to GC.** A crash between + creating the overlay or seed and defining the domain leaves volumes + that `ListByTag` cannot see (it lists domains). The same Proxy's next + Create reuses the overlay and rebuilds the seed, and its Delete removes + both by name even with no domain, but a Proxy CR that is gone for good + leaves them for `virsh vol-list` to find. +- **A host removed from the config orphans its VMs.** `Delete` for a + providerID naming an unconfigured host is a logged no-op success, so + the Proxy finishes deleting instead of pinning its finalizer, but the + domain and its volumes stay on the hypervisor and orphan GC cannot see + them either (it only sweeps configured hosts). Delete the Proxies before + retiring a host, or clean up afterwards with `virsh undefine --nvram` + plus `virsh vol-delete`. `Get` fails permanently for such a providerID + on purpose: a typo in `hosts` must not look like a missing VM and + trigger a rebuild elsewhere. +- **No adoption of `kvm` domains.** See *Migrating from kvm*. +- **A cancelled call drops the host's pooled connection once**, so other + RPCs in flight on that host fail once with `ErrTransient` and retry. + The image upload and the health check are isolated on their own + connections. +- **`Content-Length` is required** for image downloads; a mirror that + only chunks is rejected permanently. Point the catalogue at one that + sends the length. +- **A paused or pmsuspended domain cannot be adopted.** A repeat `Create` + for a domain in that state calls `DomainCreate`, which libvirt refuses + as "already running"; the provider then waits for the domain to report + running, the 2-minute wait expires, and the call fails transiently on + every retry. Resume the domain on the host (`virsh resume `) or + delete the Proxy. +- **Parser text in credential errors.** A key or known_hosts file that + does not parse fails with the parser's own message, which can quote a + PEM header type or a malformed known_hosts token. Key bytes are never + quoted. +- **The sample cloud-init is an open proxy.** + `config/samples/proxy_kvm2.yaml` installs Squid with + `http_access allow all`; restrict it before exposing the bridge network + to anything but the cluster. +- **IP discovery needs the agent or guest traffic to the host.** A VM on + a bridge with no DHCP never gets an address and stays `Provisioning` + forever; check `virsh console ` on the host. The same symptom + with a booted, addressed VM means the guest agent never came up: check + `systemctl status qemu-guest-agent`, `cloud-init status --long` and the + vendor script's output in `/var/log/cloud-init-output.log` (a VM without + outbound internet cannot install it). A user-data that disables + vendor-data (`vendor_data: {enabled: false}`) also removes the agent, by + design. `virsh domifaddr --source agent` and `--source arp` show + what the operator sees, in that order. +- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM** + ([Debian bug #1111240](https://bugs.debian.org/1111240)), which is why + `firmware` defaults to `uefi`; the same image boots under UEFI with the + serial console intact. Images that need `firmware: bios` are the + exception. + +## Testing + +Unit tests run over an in-memory fake of the libvirt seam and an +in-process SSH server. The optional integration test against a real host +is described in +[testing.md](testing.md#the-kvm2-integration-test-real-hypervisor); run it +with `make test-integration-kvm2`. `TestKVM2Check_realHost` runs the +health check against the same host and creates nothing: +`go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/`. diff --git a/docs/plans-executions/2026-09-16-1333-kvm2-provider.md b/docs/plans-executions/2026-09-16-1333-kvm2-provider.md index 98db9da..e1f7698 100644 --- a/docs/plans-executions/2026-09-16-1333-kvm2-provider.md +++ b/docs/plans-executions/2026-09-16-1333-kvm2-provider.md @@ -11,7 +11,7 @@ Plan: [docs/plans/2026-09-16-1333-kvm2-provider.md](../plans/2026-09-16-1333-kvm - [x] Step 6 — The four contract methods - [x] Step 7 — Checker and the CRD comment - [x] Step 8 — Deployment files, Makefile, integration test -- [ ] Step 9 — Documentation +- [x] Step 9 — Documentation - [ ] Step 10 — Milestone review - [ ] Step 11 — Real-host validation, changelog, MR @@ -318,3 +318,23 @@ make -n test-integration-kvm2 ``` All clean. Attestation: "files read outside the allowed set: none". + +## Step 9 — Documentation + +2026-09-16 19:54. Eighth subagent step: `docs/kvm2-provider.md` (sixteen +sections modelled on the kvm guide), a `kvm2` column in the three tables of +`docs/providers-existing.md` plus a kvm2 section and renumbering, the +components row, §5 block, §10 span bullet, §11 row and eight Decisions +bullets in `docs/architecture.md`, a README row and paragraph, and the +guide in the `docs/providers.md` index. The agent verified each claim +against the package rather than the spec and dropped a draft sentence +claiming kvm and kvm2 share the base-image cache, because checking it would +have meant reading v1's volume names. + +Checks re-run by the orchestrator: no em-dash in the guide, no line +references, each of the 19 axis codes appears once in the appendix table +with seven cells, every link and anchor across the seven touched documents +resolves. The one added em-dash is the `Leader-elected` cell of the new +components-table row, where the table's own convention wins. + +Attestation: "files read outside the allowed set: none". diff --git a/docs/providers-existing.md b/docs/providers-existing.md index 9019293..59751a8 100644 --- a/docs/providers-existing.md +++ b/docs/providers-existing.md @@ -1,13 +1,17 @@ # The existing providers This is the appendix to [provider-contract.md](provider-contract.md): how the -three providers that exist today (kubernetes, gcp, kvm) meet the contract, -which profile each one chose on every axis, and what is still missing. It is -reference material for whoever writes the next provider, not a user guide; -the user guides are [kvm-provider.md](kvm-provider.md), +four providers that exist today (kubernetes, gcp, kvm, kvm2) meet the +contract, which profile each one chose on every axis, and what is still +missing. It is reference material for whoever writes the next provider, not +a user guide; the user guides are [kvm-provider.md](kvm-provider.md), +[kvm2-provider.md](kvm2-provider.md), [gcp-vm-validation.md](gcp-vm-validation.md) and [gcp-in-specific-project.md](gcp-in-specific-project.md). The README's -"Implemented providers" table stays the short public summary. +"Implemented providers" table stays the short public summary. kvm2 is the +one provider built from the contract itself (clean-room, from +[kvm2-provider-spec.md](kvm2-provider-spec.md), without reading kvm's code), +so its column is also a record of what the contract produced. The per-provider call flows are drawn in [architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world), @@ -23,59 +27,59 @@ Code references are a file path plus a function or symbol name. One row per axis of contract §4, in the same order, with the profile name and the symbol that shows it. -| Axis | kubernetes | gcp | kvm | -|---|---|---|---| -| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | -| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | -| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | -| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | -| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `` element written in `DomainDefineXML` (`metadata.go`) | -| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | -| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | -| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | -| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | -| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones//instances/` | name-is-key: `hosts//domains/` | -| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | -| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | -| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | -| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | -| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | -| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | -| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | -| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | -| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | +| Axis | kubernetes | gcp | kvm | kvm2 | +|---|---|---|---|---| +| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) | +| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 2-minute poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) | +| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | +| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) | +| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) | +| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) | +| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended (`ListByTag`) | +| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) | +| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) | +| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones//instances/` | name-is-key: `hosts//domains/` | name-is-key: `hosts//domains/`, with a UID guard on adoption (`create`) | +| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) | +| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none | +| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | +| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) | +| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) | +| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection | +| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier | +| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) | +| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` | ## 2. Other differences Rows that are not profile axes but that a new provider author compares against. -| Feature | kubernetes | gcp | kvm | -|---|---|---|---| -| Create verifies the adopted instance's UID | no | no | no | -| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | -| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | -| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | -| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | -| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | -| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | -| Checker target | `apiserver` | the project | each host, concurrently | -| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | -| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | -| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | -| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | -| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | -| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | -| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | +| Feature | kubernetes | gcp | kvm | kvm2 | +|---|---|---|---|---| +| Create verifies the adopted instance's UID | no | no | no | **yes**: a different non-empty UID in the metadata is Permanent, an empty one is adopted (`create`) | +| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image | +| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | +| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no | +| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped | +| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined | +| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) | +| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection | +| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got | +| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) | +| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) | +| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter | +| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | +| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN | +| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself | ## 3. Observability status -| Layer | kubernetes | gcp | kvm | -|---|---|---|---| -| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | -| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | -| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | -| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | +| Layer | kubernetes | gcp | kvm | kvm2 | +|---|---|---|---|---| +| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators | +| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx | +| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span | +| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) | ## 4. kubernetes (`internal/provider/kubernetes`) @@ -329,18 +333,205 @@ the author's view. tries ARP first; it says `host` and `name` are on every log line, but the check and list paths lack them. -## 7. Gaps shared by the existing providers +## 7. kvm2 (`internal/provider/kvm2`) -- **Adoption is not verified.** None of the three checks `LabelUID` when - `Create` finds an existing instance with the right name. -- **The Create comment does not match kvm.** The interface comment in - `internal/provider/provider.go` says `Create` returns as soon as the - request is submitted; kvm blocks until the domain has started. +The full user guide is [kvm2-provider.md](kvm2-provider.md) and the design +it was built from is [kvm2-provider-spec.md](kvm2-provider-spec.md); this +section is the author's view. Nothing in the package was taken from +`internal/provider/kvm`; where the two agree it is because both follow the +same libvirt. + +- **Package and seam.** `kvm2.go` (`Provider`, `New`, `withDefaults`, + `host`, `wrapErr`, the per-name lock), `create.go`, `get.go`, + `delete.go`, `list.go` (the four methods), `backend.go` (the seam, the + production adapter and the budgets), `ssh.go` (credentials, the forwarded + socket, keepalive), `hosts.go` (the pooled table and the production + dialer), `observe.go` (the span and log decorator), `image.go` (cache, + download, quota precheck), `seed.go`, `xml.go`, `volume.go`, + `metadata.go`, `id.go`, `state.go`, `errors.go`, `check.go`. The seam is + `backend`: the twenty go-libvirt calls the provider uses, each with `ctx` + first (go-libvirt's own methods take none and block), plus `Close`. + `libvirtBackend` adapts `*libvirt.Libvirt` and runs every call through + `call`, which applies the budget and the abandon policy. The `dialer` + seam separates the SSH transport: `newProvider` injects one for tests, + `newLibvirtDialer` is production. The `observe` decorator is applied + once, in `libvirtDialer.dial`, so provider code never holds an + undecorated backend. +- **Config.** `ProviderConfig.KVM2`, the same `KVMConfig` type as kvm, + validated by `validateKVMBlock` with block name `kvm2` so messages read + `kvm2.hosts`. Defaults (`diskSizeGb` 10, socket, pool, image format and + firmware) are filled by `withDefaults`. `New` never dials; it checks that + the credential files exist and warns when one does not, or when + `insecureIgnoreHostKey` is on. +- **ID.** `formatProviderID(host, name)` gives `hosts//domains/`; + `parseProviderID` accepts exactly four segments with the literal + `hosts` and `domains` and non-empty host and name, else `ErrPermanent`. +- **Create.** `Create` validates `Zone` (`host`; optional with one host), + `MachineType` (a catalogue key) and `Image` (`resolveImage`: a catalogue + key or an http(s) URL with qcow2 and uefi defaults) before any dial, and + logs the resolved parameters with `cloudInitBytes` at V(2). Then, under + the per-name lock (`lockName`) and on the pooled connection + (`hostTable.get`), `create`: `DomainLookupByName`; found means the + adoption guard (a non-empty metadata UID that differs from `req.UID` is + Permanent) and `ensureRunning`. Otherwise `StoragePoolLookupByName`, + `imageSizeIfUncached` (marker lookup, then `HEAD` or a closed `GET`) and + `checkPoolSpace` with `neededBytes` (image if uncached, plus + `diskSizeGb`, plus 16 MiB), then `ensureImage` (`cachedImage` on the + pooled connection, `fetchImage` streaming into `uploadImage` on a + `dialFresh` connection, `StoragePoolRefresh`, `writeMarker` last), + `ensureOverlay` (`ErrStorageVolExist` means reuse), `writeSeed` (delete, + build, upload; always rebuilt), two `StorageVolGetPath`, `buildDomainXML` + with file-type disks and the metadata element, `DomainDefineXML`, + `ensureRunning` (`DomainGetState`; `DomainCreate` with + `ErrOperationInvalid` as already running; `waitRunning` polls every + 500 ms for up to 2 minutes, Transient on expiry). `Create` returns only + when the domain is running. +- **Get and state.** `mapState(state, ip)`: + + | libvirt state | IP | Provider state | + |---|---|---| + | nostate | any | Provisioning | + | running, blocked | empty | Provisioning | + | running, blocked | set | Running | + | paused, shutdown, shutoff, crashed, pmsuspended, anything else | any | Stopped | + + `get`: `DomainLookupByName` (only `ErrNoDomain` reaches NotFound), + `readMetadata` (`DomainGetMetadata` with `metadataURI`; + `ErrNoDomainMetadata` means unmanaged with an empty UID; an element that + does not parse is Permanent), `inspect` (`DomainGetState`, then for an + active domain `address`). `addressSources` asks the guest agent first, + then ARP; a source that fails is skipped at V(2), only ctx ending is an + error, so a cancelled Get is Transient and not a false Provisioning. + `firstIPv4` drops loopback, link-local and unspecified addresses and an + address whose type tag and literal disagree. `UID` and `CreatedAt` are + read back on every call; an unparsable `createdAt` gives the zero time. +- **Delete.** Under the per-name lock: `removeDomain` (`DomainDestroy`, + tolerating not running and gone; `DomainUndefineFlags` with NVRAM, + managed save and snapshot metadata) and `removeVolumes` (overlay and seed + by name through `deleteVolumeIfPresent`; a missing pool is a no-op). A + host no longer in the config is a logged no-op success. Nothing is left + when nil is returned. +- **ListByTag.** `ConnectListAllDomains` (active and inactive) on every + host in config order; `readMetadata` and `managed` keep kvm2's own + domains; `inspect` fills state and address. A domain whose metadata or + state cannot be read is skipped at V(1), unless ctx ended, which fails + the call. A host that cannot be dialled or listed fails the whole call. +- **Errors.** `classify(op, err)`: an error already carrying a class keeps + it; libvirt codes through `classifyLibvirt` (`ErrNoDomain` and + `ErrNoStorageVol` NotFound, Transient when `op == "create"`; + `ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, + `ErrAuthFailed`, `ErrAccessDenied` Permanent; the rest Transient); + `knownhosts.KeyError` and the `unable to authenticate` text from + `x/crypto/ssh` Permanent; everything else Transient, including a missing + credential file. `permanentf` and `transientf` classify at the source: + placement, XML builders, a parse failure of the key or known_hosts, an + image 4xx, a missing or zero `Content-Length`, a sha256 mismatch, a pool + that is not running (Permanent); an image 429 or 5xx, a short body, the + running wait expiring (Transient). `checkPoolSpace` is the only source of + `ErrQuotaExceeded`. `logAPIError` writes `kvm2 backend call failed` with + `libvirtErrorFields`. +- **Check.** `Check` runs `checkHost` per host concurrently, each through + `probeHost` on a `dialFresh` connection that is closed when the probe + ends; the pooled table is never consulted. `ConnectGetLibVersion`, + `ConnectGetHostname`, `StoragePoolLookupByName`, `StoragePoolGetInfo`; + a pool that is not running is `poolNotActiveError`. Details are filled + as facts arrive (`user` and `pool` always; `libvirtVersion`, `hostname`, + `poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkReason`: a + `dialError` is judged by its stage (`credentials` gives + `CredentialsUnavailable`; `ssh` with a `knownhosts.KeyError` or the auth + text gives `Unauthorized`; any other stage, including a dial that ran + out of time, gives `Unreachable`); then `context.DeadlineExceeded` or + `Canceled` `Timeout`, `ErrAccessDenied` or `ErrAuthFailed` `Forbidden`, + `ErrNoStoragePool` `PoolNotFound`, any other libvirt code `APIError`, + anything else `Unreachable`; `PoolNotActive` from the pool state. +- **Connections.** `hostTable` keeps one pooled connection per host, + dialled lazily under a per-host one-slot semaphore so waiting ends with + the caller's ctx; a dead backend (`liveness`) is closed and redialled on + the next `get`. `dialSSHSocket` runs credentials, TCP, the SSH handshake + and the `direct-streamlocal` channel to the socket, each failure a + `dialError` with its stage; `libvirtDialer.connect` bounds the whole + sequence plus the libvirt `Connect` with the dial budget and starts the + `keepalive` (15 s, `keepalive@openssh.com`) and a watcher on go-libvirt's + `Disconnected`. `call` runs each blocking go-libvirt call in a goroutine + and waits on the result, ctx and a budget timer; abandoning a call + (`link.markDead`) closes the socket and the SSH client, returns Transient + wrapping the ctx or deadline error, and every other call waiting on that + link fails once with Transient. `loadCredentials` runs on every dial. +- **Logging.** Logger from the context, named `kvm2`, with `provider`, + and `host` and `name` on lines about an instance. V(1): one line per + operation outcome (`kvm2 domain ensured running`, `kvm2 domain fetched`, + `kvm2 domain deleted`, `kvm2 domains listed`, `kvm2 host checked`), one + `libvirt call` line per RPC from `observe` (rpc name, identifiers, + duration, error fields), the connection events `libvirt host connected` + and `libvirt host connect failed` with `stage`, and `kvm2 backend call + failed`. V(2): XML, metadata, addresses, listed names, pool sizes, the + resolved Create parameters. Upload and download bodies, cloud-init and + key material are never logged. +- **Tracing.** `observe.call` starts `kvm2.` per RPC with + `kvm2.host` and the identifiers (`kvm2.domain`, `kvm2.pool`, + `kvm2.volume`), sets `kvm2.libvirt_code` on failure and writes the + `libvirt call` line inside the span; `libvirtDialer.dial` starts + `kvm2.dial` (`kvm2.address`, `kvm2.user`); `fetchImage` starts + `kvm2.download` (`kvm2.url_host`, `kvm2.sha256_configured`, + `kvm2.bytes`). Both the decorator and the dialer take `tracing.Option` + so tests inject a recorder; `New` leaves them on the global provider. +- **Tests.** Pure builders (`xml_test.go`, `volume_test.go`, + `seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go`); + `errors_test.go` (the classify table with real SSH error strings and a + real `knownhosts.KeyError`); `backend_test.go` (the adapter over + go-libvirt's own mock through a `net.Pipe`: deadline kills the + connection, cancel unblocks, killed by another caller); `hosts_test.go`; + `ssh_test.go` over `sshserver_test.go` (every dial stage, host-key + policies, key material never in errors or logs); `image_test.go` + (marker rules, streaming asserted by counting bytes, dedicated upload + connection); `create_test.go`, `get_test.go`, `delete_test.go`, + `list_test.go`, `kvm2_test.go`, `provider_test.go` (the contract §11 + list, named test by test in the spec's entry 21); `observe_test.go` + (every seam method spanned and logged exactly once, by reflection); + `check_test.go` (the reason table, never touches the pool). The double is + `fakeBackend` in `fake_test.go`, whose `twin` shares state under its own + call log so a test can tell the pooled connection from a dedicated one. + Real backend: `kvm2_integration_test.go` behind `//go:build integration`, + driven by `KVM2_TEST_*` variables and `make test-integration-kvm2`, + described in + [testing.md](testing.md#the-kvm2-integration-test-real-hypervisor). +- **Deployment.** The `kvm-ssh` Secret and its mount are shared with kvm + (`config/manager/manager.yaml` says so), `make kvm-ssh-secret` is reused, + a commented `kvm2-home` entry in both providers configs, + `config/samples/proxy_kvm2.yaml` (zone `beelink`, matching the shipped + config), the `kvm2` type in the `status.type` comment of + `api/v1alpha1/providerhealth_types.go`. No RBAC. +- **Known gaps** (see also [kvm2-provider.md § Known gaps](kvm2-provider.md#known-gaps)). + - `Create` blocks, including the first image download on each host. + - GC blind spots: volumes without a domain are invisible to GC, and so are + VMs on a host removed from the config, which `Delete` treats as done. + - No adoption of kvm's domains: a switched entry needs its Proxies + deleted first. + - Cancelling one operation cuts off the others on that host once: a + cancelled `call` kills the pooled connection. + - `Content-Length` is required for image downloads. + - A paused or pmsuspended domain cannot be adopted by a repeat `Create`: + `DomainCreate` refuses, `waitRunning` expires, Transient on every retry. + - Credential parse errors carry the parser's own text, which can quote a + PEM header type or a malformed known_hosts token (never key bytes). + - No reused-overlay check: an existing overlay is reused without checking + its backing image or size. + - The sample cloud-init installs Squid with `http_access allow all`. + +## 8. Gaps shared by the existing providers + +- **Adoption is not verified in three of the four.** kubernetes, gcp and + kvm adopt any existing instance with the right name without checking + `LabelUID`; kvm2 returns `ErrPermanent` when the UID differs. +- **The Create comment does not match kvm and kvm2.** The interface comment + in `internal/provider/provider.go` says `Create` returns as soon as the + request is submitted; both libvirt providers block until the domain has + started. - **`Stopped` and `Terminated` are produced inconsistently.** kubernetes - never produces Stopped, kvm never produces Terminated, and the reconciler - treats them the same. + never produces Stopped, kvm and kvm2 never produce Terminated, and the + reconciler treats them the same. - **Different `ListByTag` partial-failure policies.** gcp leaves failed - parts out, kvm fails the whole list. Both are safe. + parts out, kvm and kvm2 fail the whole list. Both are safe. - **Stale `mock` comments** in `internal/provider/provider.go`, `internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go` (and so the generated CRD's `spec.provider` description). @@ -351,33 +542,41 @@ the author's view. is still open. - **Tracing.** kvm has no child spans, so a kvm trace shows `provider.create` and nothing under it; correlation works only through `traceID` on its V(1) - log lines. Only kubernetes is tested against a real collector, and that - test checks only the operation spans; nothing checks that gcp's SDK spans - or the kubernetes API spans nest under `provider.*`. + log lines. kvm2 has them, but only an in-memory recorder checks their + topology. Only kubernetes is tested against a real collector, and that + test checks only the operation spans; nothing checks that gcp's SDK spans, + the kubernetes API spans or kvm2's spans nest under `provider.*` in a + real collector. -## 8. Proposed features +## 9. Proposed features -Listed only. Nothing below is designed or implemented. +Listed only. Nothing below is designed or implemented, except where a +bullet says kvm2 already does it. - **Child spans in kvm:** extend the `tracingAPI` decorator in `kvm/trace.go`, which already wraps every libvirt call, to also start a span per RPC, and add spans for the SSH dial and the image download. The seam's methods take no `ctx` today (`hostPool.call` hands `fn` only the - API), so the span context has to be plumbed in. -- **Tempo e2e checks for child spans**, and for gcp and kvm through their - real backend runbooks. + API), so the span context has to be plumbed in. kvm2's `observe.go` over + its ctx-first seam is the shape this would take. +- **Tempo e2e checks for child spans**, and for gcp, kvm and kvm2 through + their real backend runbooks. - **A shared conformance suite:** for example `internal/provider/providertest`, which runs the contract §11 tests against any provider plus a fake backend, so a new provider gets the contract tests for free. - **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent` on a mismatch instead of adopting an instance that belongs to someone else. + kvm2 does this; the other three do not. - **Report asynchronous create failures:** gcp keeps the operation name, and `Get` reports the failure instead of NotFound. - **Capacity or quota signals** for kvm, such as pool free space and host - memory, mapped to `ErrQuotaExceeded`. + memory, mapped to `ErrQuotaExceeded`. kvm2 has the pool-space precheck; + host memory is still unchecked in both. - **GC for secondary resources:** mark volumes and similar resources, and let `ListByTag` or a second listing method cover them. -- **Reload credentials and config without a restart.** +- **Reload credentials and config without a restart.** kvm2 re-reads its + SSH material on every dial; the providers config is still read once per + process for every type. - **Checker `Details`** for kubernetes (server version) and gcp (quota headroom). - **Internal-IP mode** for gcp, and a firewall rule managed per provider. diff --git a/docs/providers.md b/docs/providers.md index 5fbb976..c8431d9 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -13,11 +13,13 @@ the existing ones happen to work. a new provider's design answers, one entry per decision the contract leaves open. Copy it to `docs/-provider-spec.md`, fill it in, and hand it to the implementer with the other two files. -- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp and - kvm meet the contract, a table mapping each of them to every profile, their - per-provider reference notes, shared gaps and proposed features. +- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp, + kvm and kvm2 meet the contract, a table mapping each of them to every + profile, their per-provider reference notes, shared gaps and proposed + features. Per-provider user guides: [kvm-provider.md](kvm-provider.md), +[kvm2-provider.md](kvm2-provider.md), [gcp-vm-validation.md](gcp-vm-validation.md) and [gcp-in-specific-project.md](gcp-in-specific-project.md).