Add the KVM health check: reachability and storage pool per host

Each configured host is checked concurrently over the pooled SSH+libvirt
connection: libvirt version and hostname prove the daemon answers, then
the configured pool must exist and be running, with its free space
reported. Dial failures are now tagged with their stage (SSH material,
TCP, SSH handshake, libvirt connect) by a dialError that keeps the
message and unwraps to the cause, so classification is unchanged and the
check can say why a host is unreachable. A pool verdict is decided after
the call returns so it never drops a working connection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-11 20:17:35 +00:00
co-authored by Claude Opus 5
parent 1962365f44
commit 04dd2c0284
9 changed files with 694 additions and 5 deletions
@@ -4,7 +4,7 @@ Plan: [docs/plans/2026-09-11-1526-provider-health-checks.md](../plans/2026-09-11
- [x] Step 0 — Save the plan
- [x] Step 1 — ProviderHealth API and CRD
- [x] Step 2 — Contract plus the kubernetes and GCP checks
- [ ] Step 3 — KVM check
- [x] Step 3 — KVM check
- [ ] Step 4 — Monitor
- [ ] Step 5 — Metrics and wiring
- [ ] Step 6 — Docs
@@ -95,3 +95,46 @@ Worth noting:
a disabled Compute API is reported as `Forbidden` rather than `Unreachable`.
- Once a context has ended, both checks report `Timeout` whatever the error says. The
monitor's timeout is the usual cause, and the underlying error only repeats it.
## Step 3 — KVM check
`conn.go` now wraps each of the four dial stages in a `dialError`: ssh-material, tcp,
ssh-handshake and libvirt-connect. `Error()` returns the inner message unchanged and
`Unwrap()` returns the cause, so `classify`/`isClassified` and every existing
message-matching test behave as before. `TestDialError_keepsClassification` pins this
down.
The three new RPCs went into `libvirtAPI`, `tracingAPI` and `fakeAPI`. The method-count
guard in `TestTracingAPI_logsEveryMethod` fails until a traced row exists for each new
method, which is what forced the three new rows.
`Check` runs one goroutine per host, over the shared connection, through
`hostPool.call`. The pool verdict is decided only after `call` returns, because
returning it as an error from inside the callback would make `call` drop a perfectly
good connection.
Build, vet (also with the integration tag, since that file changed) and race tests:
```bash
go build ./internal/provider/... && go vet ./internal/provider/kvm/ \
&& go vet -tags=integration ./internal/provider/kvm/ \
&& go test -race -count=1 ./internal/provider/kvm/
```
Worth noting:
- **Pool state wording:** `poolState` uses virsh's words (`running`, `inactive`, …)
rather than go-libvirt's constant names, so it reads like `virsh pool-list` output.
- **Details:** each host also reports its SSH `user`. That is the first thing to check
when the reason is `SSHHandshakeFailed`.
- **Tests renamed or merged:**
- The planned `TestSSHAuth_dial_stages` is `TestLazySSH_dial_tagsTheFailingStage`. It
dials through `lazySSH`, so the ssh-material stage is covered as well as the three
stages the in-process SSH server can trigger.
- The pool-inactive and pool-missing cases are merged into `TestCheck_libvirtVerdicts`,
which also covers a daemon error. Each case runs two checks and asserts a single
dial.
- **Tests added beyond the plan:**
- `TestCheck_timeout`: a daemon that stops answering gives `Timeout`.
- `TestDialError_keepsClassification`.
- **Not run yet:** `TestKVMCheck_realHost` in the integration file only compiles here.
It needs beelink, so it goes on the Step 7 checklist for the user to run.