Add the KVM health check: reachability and storage pool per host
Each configured host is checked concurrently over the pooled SSH+libvirt connection: libvirt version and hostname prove the daemon answers, then the configured pool must exist and be running, with its free space reported. Dial failures are now tagged with their stage (SSH material, TCP, SSH handshake, libvirt connect) by a dialError that keeps the message and unwraps to the cause, so classification is unchanged and the check can say why a host is unreachable. A pool verdict is decided after the call returns so it never drops a working connection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -4,7 +4,7 @@ Plan: [docs/plans/2026-09-11-1526-provider-health-checks.md](../plans/2026-09-11
|
||||
- [x] Step 0 — Save the plan
|
||||
- [x] Step 1 — ProviderHealth API and CRD
|
||||
- [x] Step 2 — Contract plus the kubernetes and GCP checks
|
||||
- [ ] Step 3 — KVM check
|
||||
- [x] Step 3 — KVM check
|
||||
- [ ] Step 4 — Monitor
|
||||
- [ ] Step 5 — Metrics and wiring
|
||||
- [ ] Step 6 — Docs
|
||||
@@ -95,3 +95,46 @@ Worth noting:
|
||||
a disabled Compute API is reported as `Forbidden` rather than `Unreachable`.
|
||||
- Once a context has ended, both checks report `Timeout` whatever the error says. The
|
||||
monitor's timeout is the usual cause, and the underlying error only repeats it.
|
||||
|
||||
## Step 3 — KVM check
|
||||
|
||||
`conn.go` now wraps each of the four dial stages in a `dialError`: ssh-material, tcp,
|
||||
ssh-handshake and libvirt-connect. `Error()` returns the inner message unchanged and
|
||||
`Unwrap()` returns the cause, so `classify`/`isClassified` and every existing
|
||||
message-matching test behave as before. `TestDialError_keepsClassification` pins this
|
||||
down.
|
||||
|
||||
The three new RPCs went into `libvirtAPI`, `tracingAPI` and `fakeAPI`. The method-count
|
||||
guard in `TestTracingAPI_logsEveryMethod` fails until a traced row exists for each new
|
||||
method, which is what forced the three new rows.
|
||||
|
||||
`Check` runs one goroutine per host, over the shared connection, through
|
||||
`hostPool.call`. The pool verdict is decided only after `call` returns, because
|
||||
returning it as an error from inside the callback would make `call` drop a perfectly
|
||||
good connection.
|
||||
|
||||
Build, vet (also with the integration tag, since that file changed) and race tests:
|
||||
|
||||
```bash
|
||||
go build ./internal/provider/... && go vet ./internal/provider/kvm/ \
|
||||
&& go vet -tags=integration ./internal/provider/kvm/ \
|
||||
&& go test -race -count=1 ./internal/provider/kvm/
|
||||
```
|
||||
|
||||
Worth noting:
|
||||
- **Pool state wording:** `poolState` uses virsh's words (`running`, `inactive`, …)
|
||||
rather than go-libvirt's constant names, so it reads like `virsh pool-list` output.
|
||||
- **Details:** each host also reports its SSH `user`. That is the first thing to check
|
||||
when the reason is `SSHHandshakeFailed`.
|
||||
- **Tests renamed or merged:**
|
||||
- The planned `TestSSHAuth_dial_stages` is `TestLazySSH_dial_tagsTheFailingStage`. It
|
||||
dials through `lazySSH`, so the ssh-material stage is covered as well as the three
|
||||
stages the in-process SSH server can trigger.
|
||||
- The pool-inactive and pool-missing cases are merged into `TestCheck_libvirtVerdicts`,
|
||||
which also covers a daemon error. Each case runs two checks and asserts a single
|
||||
dial.
|
||||
- **Tests added beyond the plan:**
|
||||
- `TestCheck_timeout`: a daemon that stops answering gives `Timeout`.
|
||||
- `TestDialError_keepsClassification`.
|
||||
- **Not run yet:** `TestKVMCheck_realHost` in the integration file only compiles here.
|
||||
It needs beelink, so it goes on the Step 7 checklist for the user to run.
|
||||
|
||||
Reference in New Issue
Block a user