Add KVM provider transport: SSH-forwarded libvirt connections and error classification

Per-host connection pool over SSH with the libvirt unix socket forwarded
through the session, lazy dial and redial after disconnect, context
cancellation for go-libvirt's context-less RPCs, handshake deadline and
SSH keepalives. Error classification onto the provider taxonomy. Tested
with a fake dialer and an in-process SSH server that forwards unix sockets.

Step 3 of docs/plans/2026-09-04-1404-kvm-provider.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
2026-09-05 03:27:57 +00:00
co-authored by Claude Fable 5.1
parent 7aea14fbb3
commit d171c0426e
7 changed files with 1059 additions and 2 deletions
@@ -5,7 +5,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md
- [x] Step 0 — Save plan + branch `feat/kvm-provider`
- [x] Step 1 — Config (`KVMConfig` & friends, `validate()` case, tests)
- [x] Step 2 — Pure builders (`domainxml.go`, `volume.go`, `seed.go`, `id.go`, `metadata.go`)
- [ ] Step 3 — Transport + errors (`conn.go`, `errors.go`)
- [x] Step 3 — Transport + errors (`conn.go`, `errors.go`)
- [ ] Step 3b — Image cache (`image.go`)
- [ ] Step 4 — Provider methods (`kvm.go`) over a fake API
- [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG
@@ -99,3 +99,55 @@ instead of mutating the config in place, so the registry's shared
`qemu-guest-agent` via `packages:` + `runcmd`, which means the first boot
needs network access to the distro mirrors; a Proxy on an isolated bridge
would still work (ARP discovery), just without the agent.
## Step 3 — Transport + errors
`conn.go` holds the `libvirtAPI` seam (17 methods, signatures as in the
plan), a per-host connection pool, and the SSH dialer; `errors.go` holds
`classify`, `isLibvirtError`, `hasCode` and `logAPIError`. Both are
unit-tested; the whole package still runs in about a second under `-race`.
The one thing the plan did not anticipate: **go-libvirt RPCs take no
context and never time out**. A hung hypervisor would pin a reconcile worker
forever. So the pool exposes `call(ctx, host, fn)` rather than handing out
raw clients: `fn` runs in a goroutine, and if `ctx` ends first the pool
closes the SSH connection out from under it (which makes the blocked RPC
return) and reports `ctx.Err()`. The same helper (`awaitOrCancel`) bounds
`lv.Connect()` during dial. Two more liveness measures for the same reason:
the SSH handshake gets a deadline (ctx's, or 30 s), and each connection runs
an OpenSSH `keepalive@openssh.com` loop every 15 s that closes the client
when a request errors or goes unanswered for one interval.
Connection-drop policy in `call`: any error that is *not* a `libvirt.Error`
(so io.EOF, "connection closed", SSH failures) drops the cached connection
and the next call redials; a daemon-reported `libvirt.Error` keeps it. Dials
are serialised per host (concurrent Creates on one host share a dial) but
hosts don't block each other.
As Step 1 noted, the dialer uses `libvirt.NewWithDialer` with a
`socket.Dialer` that wraps `ssh.Client.Dial("unix", socket)` rather than the
deprecated `New(net.Conn)`.
Tests: the pool is exercised through a fake dialer (reuse, redial after
`Disconnected`, drop on transport error, keep on libvirt error, cancel
releases a blocked call, concurrent callers share one dial, hosts
independent). The SSH layer is tested against an in-process
`x/crypto/ssh` server that serves `direct-streamlocal@openssh.com` channels
by dialing the requested unix path (an echo socket standing in for
`libvirt-sock`), with keys generated per test and a real `known_hosts` file
written via `knownhosts.Line`. Covered: forward round-trip and the socket
path the server saw, unknown host key rejected with `knownhosts.KeyError`,
`insecureIgnoreHostKey`, wrong client key, cancelled ctx, missing key or
known_hosts file, and keepalive both closing a stalled client and leaving a
healthy one alone. The libvirt handshake (`Connect()`) itself is *not*
covered here — faking libvirt's XDR RPC just for that is not worth it; the
Step 6 integration test covers it against a real host.
`go mod tidy` promoted `go-libvirt` and `golang.org/x/crypto` to direct
dependencies.
Worth noting: `classify` maps `ErrOperationInvalid` (e.g. destroying a
domain that is already off) to Transient; Delete in Step 4 must check that
code itself via `hasCode` and treat it as success rather than relying on
the classifier. Nothing maps to `ErrQuotaExceeded` — libvirt has no quota
concept; an out-of-space pool surfaces as a Transient `ErrInternalError`.