kvm2: apply the milestone review

Shorter start wait with immediate verdicts for dead or paused domains,
ListByTag without state or address RPCs, Timeout for an expired check
round, no adoption of unmarked domains, reference-counted keyed locks,
dead options removed, HEAD fallback on any non-2xx.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
2026-09-16 21:19:50 +00:00
co-authored by Claude Fable 5.1
parent fcbf8ddc91
commit 2510842396
21 changed files with 586 additions and 254 deletions
+3 -2
View File
@@ -172,7 +172,8 @@ closes the SSH connection to unblock it.
kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract
prov.Create ──► placement checked against the catalogue (Permanent before any dial)
DomainLookupByName: exists → uid guard (foreign uid = Permanent), ─┐ every step
DomainLookupByName: exists → mark+uid guard (no mark or foreign ─┐ every step
uid = Permanent; paused = Permanent), │
start if needed, wait for running, done │ idempotent,
else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a
+ 16 MiB must fit, else ErrQuotaExceeded (reconciler │ per-name lock
@@ -184,7 +185,7 @@ prov.Create ──► placement checked against the catalogue (Permanent before
seed ISO (always rebuilt) │
DomainDefineXML (<metadata> in kvm2's own namespace: │
managed, uid, createdAt) + DomainCreate, poll until │
running (2m cap) ─┘
running (15s cap; a domain that dies at once = Transient) ─┘
prov.Get ──► DomainLookupByName + DomainGetMetadata + DomainGetState
+ DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4)
running without IP = Provisioning; nostate = Provisioning;
+52 -24
View File
@@ -41,9 +41,12 @@ ID.
**Answer:** `name-is-key`. libvirt addresses domains by name
(`DomainLookupByName`); `req.Name` is used verbatim as the domain name and as
the base of the per-VM volume names. Create looks the name up first and adopts
an existing domain only if its metadata `uid` is empty or equals `req.UID`; a
different non-empty `uid` returns `ErrPermanent` naming both UIDs. This closes
the contract's "adoption is not verified" limit for this provider.
an existing domain only if it carries the kvm2 mark (`managed` true, entry 8)
and its metadata `uid` is empty or equals `req.UID`. A domain without the
mark returns `ErrPermanent` saying it was not created by this provider: kvm2
never starts or takes over a VM someone else defined under that name. A
different non-empty `uid` returns `ErrPermanent` naming both UIDs. This
closes the contract's "adoption is not verified" limit for this provider.
**Evidence:** `DomainLookupByName(Name string) (Domain, error)` in the
go-libvirt module; contract §2.1 idempotency and §13.
@@ -201,15 +204,24 @@ libvirt has no locks or tasks; every RPC is synchronous.
**Answer:** `create-return`: `blocking`. `async-ops`: `none`. Create returns
when the domain is running, holding one reconcile worker for the duration,
including the first image download on a host (minutes on a cold pool; the
guide says so). The wait after `DomainCreate` is capped at 2 minutes; every
RPC has its own deadline (entry 18).
guide says so). The wait after `DomainCreate` is capped at 15 seconds:
`DomainCreate` is synchronous in libvirt, so the first `DomainGetState`
normally answers running and the cap only covers state settling. Two rules
keep the wait from pinning a worker: a state after the start that is neither
active nor `nostate` (shutoff, crashed, paused, pmsuspended, shutdown) is
`ErrTransient` at once, since polling will not bring that guest up; and a
domain found paused or pmsuspended before the start is `ErrPermanent`
(`cannot be started by the operator; resume or destroy it on the host`), an
admin's doing that `DomainCreate` would refuse and the reconciler must not
retry. Every RPC has its own deadline (entry 18).
Steps, each idempotent, in order, all under the per-name lock (entry 19):
1. Validate placement (entry 3). No backend call yet.
2. `DomainLookupByName`. Found: adoption guard (entry 2), then start it if it
is not running (`DomainCreate`; `ErrOperationInvalid` means already
running), return the ID.
is not running (paused or pmsuspended is `ErrPermanent`; `DomainCreate`;
`ErrOperationInvalid` means already running), poll as in step 9, return
the ID.
3. Pool space precheck (entry 13, quota).
4. Ensure the base image (entry 4): marker present and matching means done.
5. Overlay volume `<name>.qcow2`, qcow2 backed by the base file with
@@ -220,7 +232,9 @@ Steps, each idempotent, in order, all under the per-name lock (entry 19):
7. `StorageVolGetPath` for the overlay and the seed.
8. `DomainDefineXML` with file-type disks (entry 15's AppArmor note), the
metadata element, firmware per image.
9. `DomainCreate`, then poll `DomainGetState` until running.
9. `DomainCreate`, then poll `DomainGetState` until running: `nostate` is
polled again, any other inactive state is `ErrTransient` at once, and the
15-second cap is `ErrTransient` too.
10. Return `hosts/<host>/domains/<name>`.
A retry after a crash finds whatever the previous attempt got to: an existing
@@ -267,13 +281,16 @@ contract §2.3.
**Answer:** `fail-whole`, per host, with single unreadable domains skipped.
Scope: every host in `hosts`, in config order, `ConnectListAllDomains` with
active and inactive domains. For each domain, read the metadata element; keep
those with `managed` true, then read state and addresses for the `Instance`.
A domain whose metadata or state cannot be read (mid teardown, corrupt
element) is skipped with a V(1) line; a host that cannot be dialled or whose
listing fails fails the whole call, so GC skips this provider for the sweep
rather than act on a partial view. Filtering is client-side; libvirt has no
label index. Cost: one `DomainGetMetadata` per domain per sweep, about half a
millisecond each over the tunnel.
those with `managed` true and build the `Instance` from the domain name and
the element alone: `ID`, `UID` and `CreatedAt`, which is all GC reads
(contract §2.4). `State` and `IP` are left empty; no state or address RPC is
made. A domain whose metadata cannot be read (mid teardown, corrupt element)
is skipped with a V(1) line, unless ctx ended, which fails the call; a host
that cannot be dialled or whose listing fails fails the whole call, so GC
skips this provider for the sweep rather than act on a partial view.
Filtering is client-side; libvirt has no label index. Cost: exactly one
`DomainGetMetadata` per domain per sweep, about half a millisecond each over
the tunnel.
**Evidence:** execution log run 6 (per-domain metadata cost); contract §2.4
and §4.7.
@@ -390,9 +407,11 @@ pooled connection, so a check that runs out of time closes only its own
connection and cannot disrupt a reconcile. Cost: one SSH handshake and four
cheap RPCs per host per round.
Reasons: `Reachable`; `Timeout` (ctx ended before the check finished);
`Unreachable` (TCP connect or the SSH transport failed, or the link died
mid-check); `Unauthorized` (SSH authentication refused, host key unknown or
Reasons: `Reachable`; `Timeout` (the round's ctx ended before the check
finished, whichever stage was running, the dial included, or an RPC's own
budget ran out once the connection was up); `Unreachable` (TCP connect or the
SSH transport failed, including on the dial's own 10 s budget while the round
still had time, or the link died mid-check); `Unauthorized` (SSH authentication refused, host key unknown or
changed); `Forbidden` (`ErrAccessDenied` or `ErrAuthFailed` from libvirt: the
socket refused the user); `APIError` (any other libvirt error);
`CredentialsUnavailable` (the key or known_hosts file is missing or
@@ -458,7 +477,7 @@ goroutine and waits on the result, `ctx.Done()` and a timer:
| `DomainCreate` | 90 s |
| `StorageVolUpload` (seed) | 60 s |
| `StorageVolUpload` (image), HTTP download | ctx, capped at 20 m |
| wait for running after `DomainCreate` | 2 m |
| wait for running after `DomainCreate` | 15 s (entry 10) |
| HTTP client for images | no client timeout; the per-request ctx carries the cap |
When a call is abandoned (ctx cancelled or budget exceeded), the adapter
@@ -479,10 +498,14 @@ every 15 s) marks the connection dead on failure.
**Answer:** shared state is the host connection table (one pooled connection
per host, dialled lazily, under a mutex with a per-host dial mutex so
concurrent callers dial once and share the result), the per-`(host, volume)`
download locks, and a keyed mutex per instance name. Create and Delete for the
same name run under that name's lock, so two Deletes (GC and the reconciler)
never race on destroy and undefine, and a Delete cannot interleave with a
Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes
download locks, and a lock per instance name. Both lock sets are one instance
each of `keyedLocks`: a one-slot semaphore per key, so a waiter gives up when
its ctx ends, reference-counted so an entry is removed from the map once no
caller holds or waits for it and the map does not grow with every name ever
seen, while two waiters on one key always share one semaphore. Create and
Delete for the same name run under that name's lock, so two Deletes (GC and
the reconciler) never race on destroy and undefine, and a Delete cannot
interleave with a Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes
RPCs on one connection, so calls for different names proceed concurrently
on the shared connection. Three reconciles, a GC sweep and a check round at
once are safe; the check uses its own connections.
@@ -598,7 +621,7 @@ entry after the user confirms.
## 24. Known limits accepted at design time
**Answer:** eight limits, each with its workaround:
**Answer:** nine limits, each with its workaround:
- **Create blocks**, including the first image download per host, holding one
of three reconcile workers. Workaround: warm the cache with one Proxy per
@@ -615,6 +638,11 @@ entry after the user confirms.
isolated on its own connection.
- **`Content-Length` is required** for image downloads; a chunked mirror is
rejected as Permanent.
- **A paused or pmsuspended domain is not started.** A repeat `Create` for
a domain in that state is `ErrPermanent` (entry 10), so the Proxy latches
as Failed until its spec changes, rather than retrying a `DomainCreate`
that libvirt refuses. Resume the domain on the host (`virsh resume`) and
touch the spec, or delete the Proxy.
- **No v1 log-format compatibility**: the `libvirt call` lines have the same
keys where the same facts exist, but dashboards keyed on `kvm` logger names
need `kvm2`.
+27 -20
View File
@@ -235,10 +235,12 @@ Its `<metadata>` carries one element in kvm2's own namespace,
That is how orphan GC recognises its own VMs, since libvirt has no
labels, and how a repeat `Create` knows whether an existing domain of the
same name belongs to this Proxy: a different non-empty `uid` is a
permanent error naming both UIDs; an empty one is adopted. The namespace
is kvm2's own, so the `kvm` provider's domains are invisible to it (see
*Migrating from kvm*).
same name belongs to this Proxy: a domain without the element is a
permanent error (`carries no kvm2 mark; it was not created by this
provider`), a different non-empty `uid` is a permanent error naming both
UIDs, and a marked domain with a matching or empty `uid` is adopted. The
namespace is kvm2's own, so the `kvm` provider's domains are invisible to
it (see *Migrating from kvm*).
The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled
`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
@@ -321,8 +323,10 @@ Before creating any volume, `Create` reads `StoragePoolGetInfo` and
compares the pool's free space with what it is about to commit
(`checkPoolSpace` and `neededBytes` in `image.go`): the image's
`Content-Length` when the base is not cached yet (found with a `HEAD`
request, or a `GET` whose body is closed at once when the server refuses
`HEAD`), plus `diskSizeGb` in bytes for the overlay, plus 16 MiB for the
request, or a `GET` whose body is closed at once when `HEAD` answers
anything but 2xx, since mirrors refuse it with 405, 403 or 501 alike; the
`GET`'s status is then what counts), plus `diskSizeGb` in bytes for the
overlay, plus 16 MiB for the
seed. Below that it returns `ErrQuotaExceeded` with a message naming the
pool, the bytes available and the bytes needed. The reconciler answers
quota with a wait, not a failure: the Proxy shows `Provisioned=False`
@@ -347,9 +351,12 @@ by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The
What does not port is the VMs. kvm2 marks its domains in its own metadata
namespace and neither lists nor adopts domains the `kvm` provider made:
`ListByTag` skips them, so orphan GC never sees them, and a Proxy's
`spec.provider` is immutable, so an existing Proxy cannot be moved to the
new entry anyway. Do it in this order:
`ListByTag` skips them, so orphan GC never sees them, and `Create` refuses
a domain of the same name that lacks the kvm2 mark with a permanent error
rather than start it (a Proxy recreated under the new entry gets the same
deterministic name, so this is what happens when step 1 below is skipped).
A Proxy's `spec.provider` is immutable, so an existing Proxy cannot be
moved to the new entry anyway. Do it in this order:
1. Delete the Proxies on the `kvm` entry and wait for them to go; the
`kvm` provider removes their domains and volumes.
@@ -437,7 +444,7 @@ applied by `call`):
| `DomainCreate` | 90 s |
| `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s |
| `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier |
| wait for running after `DomainCreate` | 2 m, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`) |
| wait for running after `DomainCreate` | 15 s, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`); `DomainCreate` is synchronous, so the first poll normally answers running, and a domain that is shut off, crashed or paused right after the start fails at once rather than waiting out the cap |
| the `HEAD` request of the pool-space precheck | 30 s |
| the courtesy `ConnectClose` when a connection is closed | 5 s |
@@ -499,10 +506,10 @@ A failing target says where the check broke (`checkReason` in `check.go`):
|---|---|---|
| `Reachable` | ready | |
| `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host |
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget while the round still had time, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
| `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) |
| `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group |
| `Timeout` | the connection was up, but the monitor's `--provider-check-timeout` (default 15s) or an RPC's budget ran out before the check finished | a hung `libvirtd` or a saturated link |
| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out before the check finished, whichever stage was running, the dial included; or the connection was up and an RPC's own budget ran out | a hung `libvirtd`, a saturated link, or a round timeout shorter than the dial takes |
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
| `APIError` | any other libvirt error | the message carries libvirt's own text |
@@ -518,10 +525,10 @@ waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
| Class | What lands there |
|---|---|
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
| `ErrQuotaExceeded` | the pool-space precheck |
| `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring |
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring or the domain going shut off or crashed right after its start |
Error strings name paths, hosts, volumes and libvirt's own text. They
never contain key bytes, the known_hosts content or cloud-init.
@@ -556,12 +563,12 @@ never contain key bytes, the known_hosts content or cloud-init.
- **`Content-Length` is required** for image downloads; a mirror that
only chunks is rejected permanently. Point the catalogue at one that
sends the length.
- **A paused or pmsuspended domain cannot be adopted.** A repeat `Create`
for a domain in that state calls `DomainCreate`, which libvirt refuses
as "already running"; the provider then waits for the domain to report
running, the 2-minute wait expires, and the call fails transiently on
every retry. Resume the domain on the host (`virsh resume <name>`) or
delete the Proxy.
- **A paused or pmsuspended domain is not started.** A repeat `Create`
for a domain in that state fails permanently (`is DomainPaused and
cannot be started by the operator; resume or destroy it on the host`),
so the Proxy shows `Failed` until its spec changes; the operator never
resumes a VM an admin paused. Resume it on the host
(`virsh resume <name>`) and touch the spec, or delete the Proxy.
- **Parser text in credential errors.** A key or known_hosts file that
does not parse fails with the parser's own message, which can quote a
PEM header type or a malformed known_hosts token. Key bytes are never
@@ -12,7 +12,7 @@ Plan: [docs/plans/2026-09-16-1333-kvm2-provider.md](../plans/2026-09-16-1333-kvm
- [x] Step 7 — Checker and the CRD comment
- [x] Step 8 — Deployment files, Makefile, integration test
- [x] Step 9 — Documentation
- [ ] Step 10 — Milestone review
- [x] Step 10 — Milestone review
- [ ] Step 11 — Real-host validation, changelog, MR
## Step 0 — Plan file and branch
@@ -338,3 +338,45 @@ resolves. The one added em-dash is the `Leader-elected` cell of the new
components-table row, where the table's own convention wins.
Attestation: "files read outside the allowed set: none".
## Step 10 — Milestone review
2026-09-16 21:19. The `go-operator-reviewer` agent reviewed the package
against the contract and the spec (v1 code off limits to it as well) and
returned seven findings, all accepted and applied by a fresh fix agent in
one commit with regression tests:
1. `waitRunning`'s two-minute cap only ever elapsed in the bad cases and
pinned a reconcile worker; now 15 s, a domain that dies right after
start is Transient at once, and a paused or pmsuspended domain is
Permanent before `DomainCreate` (an admin's doing).
2. `ListByTag` read state and addresses that GC never uses, one agent RPC
per active domain per sweep; it now builds instances from the name and
the metadata only (`TestListByTag_doesNotReadStateOrAddresses`).
3. A health round that ran out of time was blamed on the host
(`Unreachable`); `checkHost` now reports `Timeout` whenever the round's
ctx ended, whichever dial stage was running, and the promptness test
uses the production error shape.
4. `Create` adopted an unmarked domain and never wrote the mark, so GC could
not see it; an unmarked domain under the name is now Permanent
(`TestCreate_adoptsOnlyMarkedDomains`).
5. `hostTable.closeAll` had no production caller and `withBudgets` could not
work (the dialer captured the defaults first); both removed.
6. The per-name and per-image lock maps grew without bound; replaced by a
reference-counted `keyedLocks` whose entries vanish when uncontended.
7. `imageSizeIfUncached` falls back to GET on any non-2xx HEAD, and two
Delete no-op lines moved from V(0) to V(1).
The spec (entries 2, 10, 12, 16, 18, 19, 24), the guide, the appendix and
the architecture flow were updated in the same commit. The reviewer also
listed what it checked and found correct: the error boundary, the state
rules, Delete's idempotency and the replacement window, ListByTag's
partial-failure policy, the abandon policy and double-close guards,
resource handling on every path, and secret hygiene in the format strings.
Lint: `bin/golangci-lint` still panics on this toolchain (Go 1.27 against
a staticcheck built for 1.26), reproducible on untouched packages, so
`make lint` could not gate this step; `go vet` with every build tag is
clean. Tests: `go test -race -count=3 ./internal/provider/kvm2/` (168
top-level tests) and `go test -short ./...` pass; `make test` with envtest
runs next.
+33 -25
View File
@@ -30,12 +30,12 @@ and the symbol that shows it.
| Axis | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) |
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 2-minute poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) |
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) |
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) |
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended (`ListByTag`) |
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) |
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) |
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) |
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) |
@@ -56,7 +56,7 @@ against.
| Feature | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| Create verifies the adopted instance's UID | no | no | no | **yes**: a different non-empty UID in the metadata is Permanent, an empty one is adopted (`create`) |
| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) |
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image |
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no |
@@ -372,9 +372,10 @@ same libvirt.
logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
the per-name lock (`lockName`) and on the pooled connection
(`hostTable.get`), `create`: `DomainLookupByName`; found means the
adoption guard (a non-empty metadata UID that differs from `req.UID` is
Permanent) and `ensureRunning`. Otherwise `StoragePoolLookupByName`,
`imageSizeIfUncached` (marker lookup, then `HEAD` or a closed `GET`) and
adoption guard (no kvm2 mark, or a non-empty metadata UID that differs
from `req.UID`, is Permanent) and `ensureRunning`. Otherwise
`StoragePoolLookupByName`, `imageSizeIfUncached` (marker lookup, then
`HEAD`, or a closed `GET` when `HEAD` is not 2xx) and
`checkPoolSpace` with `neededBytes` (image if uncached, plus
`diskSizeGb`, plus 16 MiB), then `ensureImage` (`cachedImage` on the
pooled connection, `fetchImage` streaming into `uploadImage` on a
@@ -382,10 +383,11 @@ same libvirt.
`ensureOverlay` (`ErrStorageVolExist` means reuse), `writeSeed` (delete,
build, upload; always rebuilt), two `StorageVolGetPath`, `buildDomainXML`
with file-type disks and the metadata element, `DomainDefineXML`,
`ensureRunning` (`DomainGetState`; `DomainCreate` with
`ErrOperationInvalid` as already running; `waitRunning` polls every
500 ms for up to 2 minutes, Transient on expiry). `Create` returns only
when the domain is running.
`ensureRunning` (`DomainGetState`, where paused or pmsuspended is
Permanent; `DomainCreate` with `ErrOperationInvalid` as already running;
`waitRunning` polls every 500 ms for up to 15 seconds, Transient on
expiry and at once for any inactive state other than `nostate`).
`Create` returns only when the domain is running.
- **Get and state.** `mapState(state, ip)`:
| libvirt state | IP | Provider state |
@@ -398,8 +400,8 @@ same libvirt.
`get`: `DomainLookupByName` (only `ErrNoDomain` reaches NotFound),
`readMetadata` (`DomainGetMetadata` with `metadataURI`;
`ErrNoDomainMetadata` means unmanaged with an empty UID; an element that
does not parse is Permanent), `inspect` (`DomainGetState`, then for an
active domain `address`). `addressSources` asks the guest agent first,
does not parse is Permanent), then `DomainGetState` and, for an active
domain, `address`. `addressSources` asks the guest agent first,
then ARP; a source that fails is skipped at V(2), only ctx ending is an
error, so a cancelled Get is Transient and not a false Provisioning.
`firstIPv4` drops loopback, link-local and unspecified addresses and an
@@ -413,9 +415,12 @@ same libvirt.
when nil is returned.
- **ListByTag.** `ConnectListAllDomains` (active and inactive) on every
host in config order; `readMetadata` and `managed` keep kvm2's own
domains; `inspect` fills state and address. A domain whose metadata or
state cannot be read is skipped at V(1), unless ctx ended, which fails
the call. A host that cannot be dialled or listed fails the whole call.
domains, and the `Instance` is `ID`, `UID` and `CreatedAt` from the name
and the element, with `State` and `IP` empty: GC reads nothing else
(contract §2.4), so no state or address RPC is made. A domain whose
metadata cannot be read is skipped at V(1), unless ctx ended, which
fails the call. A host that cannot be dialled or listed fails the whole
call.
- **Errors.** `classify(op, err)`: an error already carrying a class keeps
it; libvirt codes through `classifyLibvirt` (`ErrNoDomain` and
`ErrNoStorageVol` NotFound, Transient when `op == "create"`;
@@ -436,14 +441,16 @@ same libvirt.
`ConnectGetHostname`, `StoragePoolLookupByName`, `StoragePoolGetInfo`;
a pool that is not running is `poolNotActiveError`. Details are filled
as facts arrive (`user` and `pool` always; `libvirtVersion`, `hostname`,
`poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkReason`: a
`dialError` is judged by its stage (`credentials` gives
`CredentialsUnavailable`; `ssh` with a `knownhosts.KeyError` or the auth
text gives `Unauthorized`; any other stage, including a dial that ran
out of time, gives `Unreachable`); then `context.DeadlineExceeded` or
`Canceled` `Timeout`, `ErrAccessDenied` or `ErrAuthFailed` `Forbidden`,
`ErrNoStoragePool` `PoolNotFound`, any other libvirt code `APIError`,
anything else `Unreachable`; `PoolNotActive` from the pool state.
`poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkHost` answers
`Timeout` whenever the round's ctx ended, whatever stage was running and
whatever shape the error has; otherwise `checkReason`: a `dialError` is
judged by its stage (`credentials` gives `CredentialsUnavailable`; `ssh`
with a `knownhosts.KeyError` or the auth text gives `Unauthorized`; any
other stage, including a dial that ran out of its own budget, gives
`Unreachable`); then `context.DeadlineExceeded` or `Canceled` `Timeout`,
`ErrAccessDenied` or `ErrAuthFailed` `Forbidden`, `ErrNoStoragePool`
`PoolNotFound`, any other libvirt code `APIError`, anything else
`Unreachable`; `PoolNotActive` from the pool state.
- **Connections.** `hostTable` keeps one pooled connection per host,
dialled lazily under a per-host one-slot semaphore so waiting ends with
the caller's ctx; a dead backend (`liveness`) is closed and redialled on
@@ -510,8 +517,9 @@ same libvirt.
- Cancelling one operation cuts off the others on that host once: a
cancelled `call` kills the pooled connection.
- `Content-Length` is required for image downloads.
- A paused or pmsuspended domain cannot be adopted by a repeat `Create`:
`DomainCreate` refuses, `waitRunning` expires, Transient on every retry.
- A paused or pmsuspended domain is not started by a repeat `Create`:
`ensureRunning` returns Permanent, so the Proxy latches until its spec
changes; the admin resumes the VM on the host.
- Credential parse errors carry the parser's own text, which can quote a
PEM header type or a malformed known_hosts token (never key bytes).
- No reused-overlay check: an existing overlay is reused without checking
+13 -6
View File
@@ -62,9 +62,15 @@ func (p *Provider) checkHost(ctx context.Context, host provider.KVMHost) provide
ts.Reason = provider.ReasonReachable
default:
var inactive *poolNotActiveError
if errors.As(err, &inactive) {
switch {
case ctx.Err() != nil:
// The round ran out of time, whichever stage was running: the
// error's shape (a dial that saw the ctx end, an abandoned
// call) says nothing the admin should act on.
ts.Reason = provider.ReasonTimeout
case errors.As(err, &inactive):
ts.Reason = reasonPoolNotActive
} else {
default:
ts.Reason = checkReason(err)
}
ts.Message = err.Error()
@@ -121,10 +127,11 @@ func (e *poolNotActiveError) Error() string {
return fmt.Sprintf("storage pool %q is %s", e.pool, e.state)
}
// checkReason maps a failed probe to a reason. A dial failure is judged by
// its stage, so a TCP connect or an SSH handshake that ran out of time is
// Unreachable, not Timeout: the host did not answer. Timeout is for ctx
// ending or a budget expiring once the connection was up.
// checkReason maps a failed probe to a reason once checkHost has ruled out
// the round's ctx ending. A dial failure is judged by its stage, so a TCP
// connect or an SSH handshake that ran out of the dial budget on its own is
// Unreachable, not Timeout: the host did not answer. Timeout is for a
// budget expiring once the connection was up.
func checkReason(err error) string {
var de *dialError
if errors.As(err, &de) {
+15 -8
View File
@@ -33,7 +33,9 @@ func checkContext() (context.Context, *logCapture) {
}
// checkDialer hands out a distinct fake per host name, fails the hosts in
// errs, counts every dial, and, with block set, waits for ctx to end.
// errs, counts every dial, and, with block set, waits for ctx to end and
// then fails the way dialSSHSocket does when the TCP connect sees the ctx
// end: a dialError at the tcp stage wrapping the ctx's error.
type checkDialer struct {
backends map[string]*fakeBackend
errs map[string]error
@@ -46,7 +48,7 @@ func (d *checkDialer) dial(ctx context.Context, host provider.KVMHost) (backend,
d.dials.Add(1)
if d.block {
<-ctx.Done()
return nil, fmt.Errorf("%w: dialing %s: %w", provider.ErrTransient, host.Name, ctx.Err())
return nil, &dialError{stage: stageTCP, host: host.Name, err: fmt.Errorf("dial tcp %s: %w", host.Address, ctx.Err())}
}
if err := d.errs[host.Name]; err != nil {
return nil, err
@@ -140,13 +142,18 @@ func TestCheck_hostsCheckedConcurrently(t *testing.T) {
}
}
// TestCheck_expiredContextReturnsPromptly drives the production shape of a
// round that ran out of time during the dial: the dialer fails at the tcp
// stage wrapping context.DeadlineExceeded, which checkReason alone would
// call Unreachable; the verdict must still be Timeout because the round's
// ctx ended.
func TestCheck_expiredContextReturnsPromptly(t *testing.T) {
t.Parallel()
cfg := twoHosts()
p := newProvider(testProvider, cfg, &checkDialer{block: true})
ctx, _ := checkContext()
ctx, cancel := context.WithCancel(ctx)
cancel()
ctx, cancel := context.WithTimeout(ctx, 0)
defer cancel()
start := time.Now()
got := p.Check(ctx)
if elapsed := time.Since(start); elapsed > time.Second {
@@ -159,8 +166,8 @@ func TestCheck_expiredContextReturnsPromptly(t *testing.T) {
if ts.Ready || ts.Reason != provider.ReasonTimeout {
t.Errorf("target %s = ready %v reason %s, want not ready, Timeout", ts.Name, ts.Ready, ts.Reason)
}
if ts.Message == "" {
t.Errorf("target %s has no message", ts.Name)
if !strings.Contains(ts.Message, context.DeadlineExceeded.Error()) || !strings.Contains(ts.Message, string(stageTCP)) {
t.Errorf("target %s message = %q, want the tcp stage and the deadline named", ts.Name, ts.Message)
}
}
}
@@ -217,7 +224,7 @@ func TestCheck_reasonsTable(t *testing.T) {
wantReason: provider.ReasonUnreachable,
},
{
name: "tcp timed out",
name: "tcp timed out on the dial budget but ctx is fine",
dialErr: func(*testing.T) error {
return &dialError{stage: stageTCP, host: testHost, err: fmt.Errorf("dial tcp 192.0.2.1:22: %w", context.DeadlineExceeded)}
},
@@ -248,7 +255,7 @@ func TestCheck_reasonsTable(t *testing.T) {
wantReason: provider.ReasonUnreachable,
},
{
name: "ssh handshake timed out",
name: "ssh handshake timed out on the dial budget but ctx is fine",
dialErr: func(*testing.T) error { return sshHandshake(context.DeadlineExceeded) },
wantReason: provider.ReasonUnreachable,
},
+21 -6
View File
@@ -15,9 +15,11 @@ import (
// The wait after DomainCreate (spec entry 18): Create returns only once the
// domain is running, so Get never sees a domain kvm2 defined but did not
// start (spec entry 9).
// start (spec entry 9). DomainCreate is synchronous, so the first read
// normally succeeds; the cap only covers state settling and must not pin a
// reconcile worker for long.
const (
defaultRunningWait = 2 * time.Minute
defaultRunningWait = 15 * time.Second
defaultRunningPoll = 500 * time.Millisecond
)
@@ -75,6 +77,9 @@ func (p *Provider) create(ctx context.Context, log logr.Logger, be backend, host
if err != nil {
return err
}
if !meta.managed() {
return permanentf("domain %s exists but carries no kvm2 mark; it was not created by this provider", req.Name)
}
if meta.UID != "" && meta.UID != req.UID {
return permanentf("domain %s belongs to proxy uid %s, not %s", req.Name, meta.UID, req.UID)
}
@@ -211,14 +216,19 @@ func (p *Provider) writeSeed(ctx context.Context, log logr.Logger, be backend, p
// ensureRunning starts the domain unless it is active already, then waits
// until libvirt reports it running. ErrOperationInvalid from DomainCreate
// means another caller started it between the state read and the call.
// means another caller started it between the state read and the call. A
// paused or pmsuspended domain is an admin's doing and DomainCreate would
// refuse it; the verdict is Permanent so the reconciler does not retry it.
func (p *Provider) ensureRunning(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) error {
state, _, err := be.DomainGetState(ctx, dom, 0)
if err != nil {
return fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
}
if isActive(libvirt.DomainState(state)) {
switch s := libvirt.DomainState(state); {
case isActive(s):
return nil
case s == libvirt.DomainPaused || s == libvirt.DomainPmsuspended:
return permanentf("domain %s is %s and cannot be started by the operator; resume or destroy it on the host", dom.Name, s)
}
switch err := be.DomainCreate(ctx, dom); {
case err == nil:
@@ -233,7 +243,9 @@ func (p *Provider) ensureRunning(ctx context.Context, log logr.Logger, be backen
// waitRunning polls DomainGetState until the domain is active, the cap
// passes (Transient: the next Create attempt finds the domain defined and
// tries again) or ctx ends.
// tries again) or ctx ends. Only nostate is worth another poll; a domain
// that is shut off, crashed or paused right after its start is not going
// to come up on its own, so that is Transient at once.
func (p *Provider) waitRunning(ctx context.Context, be backend, dom libvirt.Domain) error {
deadline := time.NewTimer(p.runningWait)
defer deadline.Stop()
@@ -244,8 +256,11 @@ func (p *Provider) waitRunning(ctx context.Context, be backend, dom libvirt.Doma
if err != nil {
return fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
}
if isActive(libvirt.DomainState(state)) {
switch s := libvirt.DomainState(state); {
case isActive(s):
return nil
case s != libvirt.DomainNostate:
return transientf("domain %s went %s right after start", dom.Name, s)
}
select {
case <-ticker.C:
+84 -8
View File
@@ -182,6 +182,64 @@ func TestCreate_returnsOnlyWhenRunning(t *testing.T) {
})
}
func TestCreate_domainThatDiesAfterStartIsTransientAtOnce(t *testing.T) {
t.Parallel()
for _, state := range []libvirt.DomainState{libvirt.DomainCrashed, libvirt.DomainShutoff, libvirt.DomainPaused} {
t.Run(state.String(), func(t *testing.T) {
t.Parallel()
// The cap is well over the bound asserted below, so a poll that
// ran to the cap would fail the test.
fx := newFixture(t, 1, withRunningWait(time.Millisecond, 10*time.Second))
fx.fake.stateAfterStart = state
start := time.Now()
id, err := fx.p.Create(fx.ctx, fx.req(testName))
elapsed := time.Since(start)
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrTransient)
if !strings.Contains(err.Error(), "right after start") {
t.Errorf("error = %v, want the state after start named", err)
}
if elapsed > time.Second {
t.Errorf("Create took %v, want it to give up at once, not poll to the cap", elapsed)
}
if got := fx.fake.callCount("DomainCreate"); got != 1 {
t.Errorf("DomainCreate calls = %d, want 1", got)
}
})
}
}
func TestCreate_pausedDomainIsPermanent(t *testing.T) {
t.Parallel()
for _, state := range []libvirt.DomainState{libvirt.DomainPaused, libvirt.DomainPmsuspended} {
t.Run(state.String(), func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, state, managedMetadata(testUID, time.Now()))
id, err := fx.p.Create(fx.ctx, fx.req(testName))
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrPermanent)
if !strings.Contains(err.Error(), "cannot be started by the operator") {
t.Errorf("error = %v, want the admin action named", err)
}
for _, rpc := range []string{"DomainCreate", "DomainDefineXML", "DomainDestroy"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
fx.withDomain(testName, func(d *fakeDomain) {
if d.state != state {
t.Errorf("domain state = %s, want it left %s", d.state, state)
}
})
})
}
}
func TestCreate_rejectsUnknownPlacement(t *testing.T) {
t.Parallel()
tests := []struct {
@@ -248,24 +306,42 @@ func TestCreate_adoptionWithForeignUIDIsPermanent(t *testing.T) {
}
}
func TestCreate_adoptsMatchingOrEmptyUID(t *testing.T) {
func TestCreate_adoptsOnlyMarkedDomains(t *testing.T) {
t.Parallel()
tests := []struct {
name string
metadata string
name string
metadata string
wantAdopt bool
}{
{name: "matching uid", metadata: managedMetadata(testUID, time.Now())},
{name: "empty uid", metadata: managedMetadata("", time.Now())},
{name: "matching uid", metadata: managedMetadata(testUID, time.Now()), wantAdopt: true},
{name: "managed, empty uid", metadata: managedMetadata("", time.Now()), wantAdopt: true},
{name: "no metadata element", metadata: ""},
{name: "element without the managed mark", metadata: `<instance><uid>` + testUID + `</uid></instance>`},
{name: "managed false", metadata: `<instance><managed>false</managed><uid>` + testUID + `</uid></instance>`},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, libvirt.DomainRunning, tc.metadata)
fx.create(testName)
if got := fx.fake.callCount("DomainDefineXML"); got != 0 {
t.Errorf("DomainDefineXML calls = %d, want 0 (adopted)", got)
id, err := fx.p.Create(fx.ctx, fx.req(testName))
if tc.wantAdopt {
if err != nil || id != formatProviderID(testHost, testName) {
t.Fatalf("Create = %q, %v; want the existing domain adopted", id, err)
}
} else {
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrPermanent)
if !strings.Contains(err.Error(), "carries no kvm2 mark") {
t.Errorf("error = %v, want the missing mark named", err)
}
}
for _, rpc := range []string{"DomainDefineXML", "DomainCreate", "StorageVolCreateXML", "DomainDestroy"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
})
}
+2 -2
View File
@@ -26,7 +26,7 @@ func (p *Provider) Delete(ctx context.Context, providerID string) error {
log := p.logger(ctx).WithValues("host", hostName, "name", name)
host, err := p.host(hostName)
if err != nil {
log.Info("host is not configured; delete has nothing to clean up", "id", providerID)
log.V(1).Info("host is not configured; delete has nothing to clean up", "id", providerID)
return nil
}
unlock, err := p.lockName(ctx, name)
@@ -82,7 +82,7 @@ func removeDomain(ctx context.Context, log logr.Logger, be backend, name string)
func removeVolumes(ctx context.Context, log logr.Logger, be backend, poolName, name string) error {
pool, err := be.StoragePoolLookupByName(ctx, poolName)
if hasCode(err, libvirt.ErrNoStoragePool) {
log.Info("pool not found, no volumes to delete", "pool", poolName)
log.V(1).Info("pool not found, no volumes to delete", "pool", poolName)
return nil
}
if err != nil {
+18 -4
View File
@@ -128,7 +128,7 @@ func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
})
t.Run("pool gone", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
fx := newFixture(t, 1)
fx.p.cfg.Hosts[0].Pool = "retired"
fx.fake.addDomain(testName, libvirt.DomainShutoff, managedMetadata(testUID, time.Now()))
if err := fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)); err != nil {
@@ -138,7 +138,7 @@ func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
t.Error("domain still defined after Delete")
}
if got := fx.logs.matching("pool not found, no volumes to delete"); len(got) != 1 || !strings.Contains(got[0], `"pool"="retired"`) {
t.Errorf("'pool not found' Info line = %v", got)
t.Errorf("'pool not found' V(1) line = %v", got)
}
})
t.Run("nothing at all", func(t *testing.T) {
@@ -152,7 +152,7 @@ func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
func TestDelete_unconfiguredHostIsNoOp(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
fx := newFixture(t, 1)
id := formatProviderID("nas9", testName)
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v, want nil", err)
@@ -164,7 +164,21 @@ func TestDelete_unconfiguredHostIsNoOp(t *testing.T) {
t.Errorf("backend calls = %v, want none", got)
}
if got := fx.logs.matching("host is not configured; delete has nothing to clean up"); len(got) != 1 || !strings.Contains(got[0], `"host"="nas9"`) {
t.Errorf("Info line = %v, want one naming the host", got)
t.Errorf("V(1) line = %v, want one naming the host", got)
}
}
func TestDelete_noOpLinesAreSilentAtV0(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
fx.p.cfg.Hosts[0].Pool = "retired"
for _, id := range []string{formatProviderID("nas9", testName), formatProviderID(testHost, testName)} {
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete(%s) error = %v, want nil", id, err)
}
}
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("Delete logged %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
}
}
+12 -4
View File
@@ -36,10 +36,13 @@ type fakeBackend struct {
// sha256 in fixed chunks instead of holding the bytes, for tests that
// push more through the fake than they want in memory.
discardUploads bool
// startDelayPolls, when set, makes DomainCreate leave the domain shut
// off; it turns running on the startDelayPolls-th DomainGetState after
// that, as a real host that is still booting QEMU would.
// startDelayPolls, when set, makes DomainCreate leave the domain in
// nostate; it turns running on the startDelayPolls-th DomainGetState
// after that, as a host whose QEMU is still settling would.
startDelayPolls int
// stateAfterStart, when not nostate, is what DomainCreate leaves the
// domain in instead of running, as a guest that dies at once would.
stateAfterStart libvirt.DomainState
// undefineFlags records the flags of the last DomainUndefineFlags per
// domain name.
undefineFlags map[string]libvirt.DomainUndefineFlagsValues
@@ -291,9 +294,14 @@ func (f *fakeBackend) DomainCreate(ctx context.Context, dom libvirt.Domain) erro
if isActive(d.state) {
return fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is already running")
}
if f.startDelayPolls > 0 {
switch {
case f.startDelayPolls > 0:
d.state = libvirt.DomainNostate
d.pendingStart = f.startDelayPolls
return nil
case f.stateAfterStart != libvirt.DomainNostate:
d.state = f.stateAfterStart
return nil
}
d.state = libvirt.DomainRunning
d.id = int32(len(f.domains))
+18 -26
View File
@@ -42,6 +42,8 @@ func (p *Provider) Get(ctx context.Context, providerID string) (*provider.Instan
return inst, nil
}
// get builds the Instance from the domain's metadata, its state and, for an
// active domain, its discovered address.
func get(ctx context.Context, log logr.Logger, be backend, hostName, name string) (*provider.Instance, error) {
dom, err := be.DomainLookupByName(ctx, name)
if err != nil {
@@ -51,11 +53,24 @@ func get(ctx context.Context, log logr.Logger, be backend, hostName, name string
if err != nil {
return nil, err
}
inst, err := inspect(ctx, log, be, hostName, dom, meta)
state, _, err := be.DomainGetState(ctx, dom, 0)
if err != nil {
return nil, err
return nil, fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
}
return &inst, nil
s := libvirt.DomainState(state)
var ip string
if isActive(s) {
if ip, err = address(ctx, log, be, dom); err != nil {
return nil, err
}
}
return &provider.Instance{
ID: formatProviderID(hostName, dom.Name),
IP: ip,
State: mapState(s, ip),
UID: meta.UID,
CreatedAt: meta.createdAt(),
}, nil
}
// readMetadata returns the kvm2 mark of a domain, or the zero value (not
@@ -72,29 +87,6 @@ func readMetadata(ctx context.Context, be backend, dom libvirt.Domain) (instance
return parseInstanceMetadata(raw)
}
// inspect builds the Instance from the backend's state and, for an active
// domain, its discovered address.
func inspect(ctx context.Context, log logr.Logger, be backend, hostName string, dom libvirt.Domain, meta instanceMetadata) (provider.Instance, error) {
state, _, err := be.DomainGetState(ctx, dom, 0)
if err != nil {
return provider.Instance{}, fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
}
s := libvirt.DomainState(state)
var ip string
if isActive(s) {
if ip, err = address(ctx, log, be, dom); err != nil {
return provider.Instance{}, err
}
}
return provider.Instance{
ID: formatProviderID(hostName, dom.Name),
IP: ip,
State: mapState(s, ip),
UID: meta.UID,
CreatedAt: meta.createdAt(),
}, nil
}
// addressSources is the discovery order of spec entry 7: the guest agent
// answers where the host's ARP table never sees guest traffic.
var addressSources = []struct {
-21
View File
@@ -2,7 +2,6 @@ package kvm2
import (
"context"
"errors"
"fmt"
"sync"
"time"
@@ -93,26 +92,6 @@ func (t *hostTable) dialFresh(ctx context.Context, host provider.KVMHost) (backe
return t.dialer.dial(ctx, host)
}
// closeAll closes every pooled connection, waiting for in-flight dials.
func (t *hostTable) closeAll() error {
t.mu.Lock()
entries := make([]*hostEntry, 0, len(t.hosts))
for _, e := range t.hosts {
entries = append(entries, e)
}
t.mu.Unlock()
var errs []error
for _, e := range entries {
e.sem <- struct{}{}
if e.backend != nil {
errs = append(errs, e.backend.Close())
e.backend = nil
}
<-e.sem
}
return errors.Join(errs...)
}
// libvirtDialer is the production dialer: SSH transport, go-libvirt over
// the forwarded socket, the connect handshake, then the observe decorator,
// so provider code never holds an undecorated backend.
-17
View File
@@ -176,23 +176,6 @@ func TestHosts_dialErrorIsNotCached(t *testing.T) {
}
}
func TestHosts_closeAll(t *testing.T) {
t.Parallel()
table := newHostTable(&stubDialer{fresh: func() backend { return newFakeBackend() }})
a, _ := table.get(context.Background(), hostA)
b, _ := table.get(context.Background(), hostB)
if err := table.closeAll(); err != nil {
t.Fatalf("closeAll error = %v", err)
}
if a.(*fakeBackend).closed != 1 || b.(*fakeBackend).closed != 1 {
t.Errorf("closed counts = %d, %d, want 1, 1", a.(*fakeBackend).closed, b.(*fakeBackend).closed)
}
again, _ := table.get(context.Background(), hostA)
if again == a {
t.Error("get after closeAll returned the closed backend")
}
}
// perHostDialer routes each host to its own stub.
type perHostDialer map[string]*stubDialer
+15 -22
View File
@@ -66,24 +66,15 @@ func resolveImage(cfg provider.KVMConfig, image string) (imageSource, error) {
}
// lockImage takes the per-(host, volume) lock of spec entry 19, or gives up
// with ctx. The lock is a one-slot channel rather than a mutex so a caller
// whose ctx ends while another caller's download runs returns promptly
// (contract §1.5) instead of waiting out a 20-minute transfer.
// with ctx: a caller whose ctx ends while another caller's download runs
// returns promptly (contract §1.5) instead of waiting out a 20-minute
// transfer.
func (p *Provider) lockImage(ctx context.Context, hostName, volume string) (func(), error) {
key := hostName + "/" + volume
p.imageMu.Lock()
sem, ok := p.imageLocks[key]
if !ok {
sem = make(chan struct{}, 1)
p.imageLocks[key] = sem
}
p.imageMu.Unlock()
select {
case sem <- struct{}{}:
return func() { <-sem }, nil
case <-ctx.Done():
return nil, fmt.Errorf("%w: waiting for image %s on host %s: %w", provider.ErrTransient, volume, hostName, ctx.Err())
release, err := p.imageLocks.acquire(ctx, hostName+"/"+volume)
if err != nil {
return nil, fmt.Errorf("%w: waiting for image %s on host %s: %w", provider.ErrTransient, volume, hostName, err)
}
return release, nil
}
// ensureImage makes the base image present and complete in the host's pool
@@ -208,7 +199,7 @@ func deleteVolumeIfPresent(ctx context.Context, be backend, pool libvirt.Storage
// finding no marker, and removing it now frees the space at once.
func (p *Provider) fetchImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, be backend, src imageSource) (libvirt.StorageVol, error) {
name := src.volumeName()
ctx, cancel := context.WithTimeout(ctx, p.budgets.transfer)
ctx, cancel := context.WithTimeout(ctx, defaultBudgets.transfer)
defer cancel()
ctx, span := tracing.StartSpan(ctx, tracing.Tracer(p.tracing...), "kvm2.download",
trace.WithSpanKind(trace.SpanKindClient),
@@ -337,7 +328,7 @@ func (c *countingReader) state() (int64, error) {
// budget bounds it on its own. Failure is logged, not returned: the next
// attempt finds the volume without a marker and deletes it then.
func (p *Provider) discardVolume(ctx context.Context, log logr.Logger, be backend, pool libvirt.StoragePool, name string) {
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), p.budgets.rpc)
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), defaultBudgets.rpc)
defer cancel()
if err := deleteVolumeIfPresent(ctx, be, pool, name); err != nil {
log.V(1).Info("incomplete base volume left behind; the next attempt removes it", "error", err.Error())
@@ -421,8 +412,10 @@ func urlHost(rawURL string) string {
// imageSizeIfUncached returns the image's Content-Length when the base is
// not cached on the host, and 0 when the marker is present, for the quota
// precheck of spec entry 13. It never downloads: HEAD first, and a GET whose
// body is closed at once when the server does not allow HEAD. A marker is
// taken at face value here; ensureImage decides whether it still matches.
// body is closed at once when HEAD answers anything but 2xx, since servers
// refuse HEAD with 405, 403 or 501 alike; the GET's status is what gets
// classified. A marker is taken at face value here; ensureImage decides
// whether it still matches.
func (p *Provider) imageSizeIfUncached(ctx context.Context, be backend, pool libvirt.StoragePool, src imageSource) (int64, error) {
_, err := be.StorageVolLookupByName(ctx, pool, src.markerName())
switch {
@@ -431,14 +424,14 @@ func (p *Provider) imageSizeIfUncached(ctx context.Context, be backend, pool lib
case !hasCode(err, libvirt.ErrNoStorageVol):
return 0, fmt.Errorf("looking up marker %s: %w", src.markerName(), err)
}
ctx, cancel := context.WithTimeout(ctx, p.budgets.rpc)
ctx, cancel := context.WithTimeout(ctx, defaultBudgets.rpc)
defer cancel()
resp, err := p.do(ctx, http.MethodHead, src.URL)
if err != nil {
return 0, err
}
_ = resp.Body.Close()
if resp.StatusCode == http.StatusMethodNotAllowed {
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
if resp, err = p.do(ctx, http.MethodGet, src.URL); err != nil {
return 0, err
}
+36 -12
View File
@@ -152,6 +152,17 @@ func serveStatus(code int) http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) { http.Error(w, http.StatusText(code), code) }
}
// headFails answers HEAD with code and hands every other method to next.
func headFails(code int, next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodHead {
http.Error(w, http.StatusText(code), code)
return
}
next(w, r)
}
}
func TestResolveImage_table(t *testing.T) {
t.Parallel()
cfg := withDefaults(provider.KVMConfig{Images: map[string]provider.KVMImage{
@@ -883,17 +894,30 @@ func TestImageSizeIfUncached_headThenGetFallback(t *testing.T) {
wantMethods: []string{http.MethodHead},
},
{
name: "HEAD not allowed falls back to GET with the body closed",
handler: func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodHead {
http.Error(w, "nope", http.StatusMethodNotAllowed)
return
}
serveBytes(body)(w, r)
},
name: "HEAD not allowed falls back to GET with the body closed",
handler: headFails(http.StatusMethodNotAllowed, serveBytes(body)),
wantSize: int64(len(body)),
wantMethods: []string{http.MethodHead, http.MethodGet},
},
{
name: "HEAD forbidden falls back to GET",
handler: headFails(http.StatusForbidden, serveBytes(body)),
wantSize: int64(len(body)),
wantMethods: []string{http.MethodHead, http.MethodGet},
},
{
name: "HEAD not implemented falls back to GET",
handler: headFails(http.StatusNotImplemented, serveBytes(body)),
wantSize: int64(len(body)),
wantMethods: []string{http.MethodHead, http.MethodGet},
},
{
name: "HEAD refused and GET 404 is permanent from the GET",
handler: headFails(http.StatusMethodNotAllowed, serveStatus(http.StatusNotFound)),
wantMethods: []string{http.MethodHead, http.MethodGet},
wantClass: provider.ErrPermanent,
wantErr: "404",
},
{
name: "marker present means nothing to download",
handler: serveBytes(body),
@@ -902,16 +926,16 @@ func TestImageSizeIfUncached_headThenGetFallback(t *testing.T) {
wantMethods: nil,
},
{
name: "404 is permanent",
name: "404 on both is permanent",
handler: serveStatus(http.StatusNotFound),
wantMethods: []string{http.MethodHead},
wantMethods: []string{http.MethodHead, http.MethodGet},
wantClass: provider.ErrPermanent,
wantErr: "404",
},
{
name: "503 is transient",
name: "503 on both is transient",
handler: serveStatus(http.StatusServiceUnavailable),
wantMethods: []string{http.MethodHead},
wantMethods: []string{http.MethodHead, http.MethodGet},
wantClass: provider.ErrTransient,
},
{
+62 -26
View File
@@ -33,7 +33,6 @@ type Provider struct {
now func() time.Time
hosts *hostTable
http *http.Client
budgets budgets
tracing []tracing.Option
// runningWait caps the wait for a started domain to report running,
@@ -42,28 +41,23 @@ type Provider struct {
runningPoll time.Duration
// imageLocks serialises image fetches per (host, volume), spec entry 19.
imageMu sync.Mutex
imageLocks map[string]chan struct{}
imageLocks *keyedLocks
// nameLocks serialises Create and Delete per instance name, spec entry
// 19, so two Deletes never race on destroy and undefine and a Delete
// cannot interleave with a Create's steps.
nameMu sync.Mutex
nameLocks map[string]chan struct{}
nameLocks *keyedLocks
}
// providerOption adjusts what newProvider builds; tests inject an HTTP
// client bound to a test server, a span recorder and shorter budgets.
// client bound to a test server, a span recorder and a shorter running
// wait.
type providerOption func(*Provider)
func withHTTPClient(c *http.Client) providerOption {
return func(p *Provider) { p.http = c }
}
func withBudgets(b budgets) providerOption {
return func(p *Provider) { p.budgets = b }
}
func withTracing(opts ...tracing.Option) providerOption {
return func(p *Provider) { p.tracing = opts }
}
@@ -104,7 +98,7 @@ func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, er
// newProvider is the constructor tests use: d hands out the backends, the
// production one being newLibvirtDialer. Without options the image
// downloads use http.DefaultClient (system CA roots, no client timeout: the
// per-request ctx carries the transfer budget) and the default budgets.
// per-request ctx carries the transfer budget).
func newProvider(name string, cfg provider.KVMConfig, d dialer, opts ...providerOption) *Provider {
p := &Provider{
name: name,
@@ -112,11 +106,10 @@ func newProvider(name string, cfg provider.KVMConfig, d dialer, opts ...provider
now: time.Now,
hosts: newHostTable(d),
http: http.DefaultClient,
budgets: defaultBudgets,
runningWait: defaultRunningWait,
runningPoll: defaultRunningPoll,
imageLocks: make(map[string]chan struct{}),
nameLocks: make(map[string]chan struct{}),
imageLocks: newKeyedLocks(),
nameLocks: newKeyedLocks(),
}
for _, opt := range opts {
opt(p)
@@ -187,21 +180,64 @@ func (p *Provider) wrapErr(op, id string, err error) error {
return provider.Wrap(classify(op, err), op, p.name, id, err)
}
// lockName takes the per-name lock of spec entry 19, or gives up with ctx.
// A one-slot channel rather than a mutex, so a caller whose ctx ends while
// another caller's Create is downloading an image returns promptly.
// lockName takes the per-name lock of spec entry 19, or gives up with ctx,
// so a caller whose ctx ends while another caller's Create is downloading
// an image returns promptly.
func (p *Provider) lockName(ctx context.Context, name string) (func(), error) {
p.nameMu.Lock()
sem, ok := p.nameLocks[name]
if !ok {
sem = make(chan struct{}, 1)
p.nameLocks[name] = sem
release, err := p.nameLocks.acquire(ctx, name)
if err != nil {
return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, err)
}
p.nameMu.Unlock()
return release, nil
}
// keyedLocks is a set of one-slot semaphores by key (spec entry 19): a
// semaphore rather than a mutex so a waiter can give up with its ctx. Each
// entry counts the callers holding or waiting for it and is dropped from
// the map at zero, so the map does not grow with every name ever seen while
// two waiters on one key still always share one semaphore.
type keyedLocks struct {
mu sync.Mutex
locks map[string]*keyedLock
}
type keyedLock struct {
sem chan struct{}
refs int
}
func newKeyedLocks() *keyedLocks {
return &keyedLocks{locks: make(map[string]*keyedLock)}
}
// acquire takes the lock for key, or gives up with ctx's error. The
// returned func releases it and must be called exactly once.
func (k *keyedLocks) acquire(ctx context.Context, key string) (func(), error) {
k.mu.Lock()
l, ok := k.locks[key]
if !ok {
l = &keyedLock{sem: make(chan struct{}, 1)}
k.locks[key] = l
}
l.refs++
k.mu.Unlock()
select {
case sem <- struct{}{}:
return func() { <-sem }, nil
case l.sem <- struct{}{}:
return func() {
<-l.sem
k.unref(key, l)
}, nil
case <-ctx.Done():
return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, ctx.Err())
k.unref(key, l)
return nil, ctx.Err()
}
}
func (k *keyedLocks) unref(key string, l *keyedLock) {
k.mu.Lock()
defer k.mu.Unlock()
l.refs--
if l.refs == 0 {
delete(k.locks, key)
}
}
+81
View File
@@ -5,6 +5,7 @@ import (
"errors"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@@ -254,6 +255,86 @@ func TestWithDefaults_doesNotAliasInput(t *testing.T) {
}
}
func lockCount(k *keyedLocks) int {
k.mu.Lock()
defer k.mu.Unlock()
return len(k.locks)
}
func TestKeyedLocks_entryRemovedWhenUncontended(t *testing.T) {
t.Parallel()
k := newKeyedLocks()
release, err := k.acquire(context.Background(), "a")
if err != nil {
t.Fatalf("acquire error = %v", err)
}
if lockCount(k) != 1 {
t.Errorf("entries while held = %d, want 1", lockCount(k))
}
release()
if lockCount(k) != 0 {
t.Errorf("entries after release = %d, want 0", lockCount(k))
}
if _, err := k.acquire(context.Background(), "a"); err != nil {
t.Fatalf("acquire after release error = %v", err)
}
}
func TestKeyedLocks_waitersShareOneLock(t *testing.T) {
t.Parallel()
k := newKeyedLocks()
const n = 20
var inside, overlaps atomic.Int32
var wg sync.WaitGroup
for range n {
wg.Go(func() {
release, err := k.acquire(context.Background(), "shared")
if err != nil {
t.Errorf("acquire error = %v", err)
return
}
if inside.Add(1) > 1 {
overlaps.Add(1)
}
time.Sleep(time.Millisecond)
inside.Add(-1)
release()
})
}
wg.Wait()
if overlaps.Load() != 0 {
t.Errorf("%d holders overlapped on one key", overlaps.Load())
}
if lockCount(k) != 0 {
t.Errorf("entries after every holder released = %d, want 0", lockCount(k))
}
}
func TestKeyedLocks_waiterGivesUpWithContext(t *testing.T) {
t.Parallel()
k := newKeyedLocks()
release, err := k.acquire(context.Background(), "a")
if err != nil {
t.Fatalf("acquire error = %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
start := time.Now()
if _, err := k.acquire(ctx, "a"); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("waiter error = %v, want DeadlineExceeded", err)
}
if time.Since(start) > time.Second {
t.Errorf("waiter took %v, want to give up with its ctx", time.Since(start))
}
if lockCount(k) != 1 {
t.Errorf("entries with the holder still in = %d, want 1: the waiter's exit must not drop the holder's entry", lockCount(k))
}
release()
if lockCount(k) != 0 {
t.Errorf("entries after the holder released = %d, want 0", lockCount(k))
}
}
func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) {
t.Parallel()
p := newProvider("kvm2-home", twoHosts(), &stubDialer{})
+9 -13
View File
@@ -10,10 +10,13 @@ import (
)
// ListByTag is spec entry 12: every domain on every configured host, in
// config order, filtered client-side on the kvm2 mark. A host that cannot
// be reached or listed fails the whole call (fail-whole); a single domain
// that cannot be read is skipped, unless ctx ended, which fails the call
// rather than pass off an empty view as the truth.
// config order, filtered client-side on the kvm2 mark. GC uses only ID, UID
// and CreatedAt (contract §2.4), all of which the name and the metadata
// element give, so no state or address RPC is made and State and IP stay
// empty. A host that cannot be reached or listed fails the whole call
// (fail-whole); a single domain whose metadata cannot be read is skipped,
// unless ctx ended, which fails the call rather than pass off an empty view
// as the truth.
func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
log := p.logger(ctx)
var out []provider.Instance
@@ -43,15 +46,8 @@ func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
if !meta.managed() {
continue
}
inst, err := inspect(ctx, dlog, be, host.Name, dom, meta)
if err != nil {
if ctx.Err() != nil {
return nil, p.wrapErr("list", "", err)
}
dlog.V(1).Info("skipping domain that could not be inspected", "error", err.Error())
continue
}
dlog.V(2).Info("kvm2 listed instance", "id", inst.ID, "state", inst.State, "uid", inst.UID, "createdAt", inst.CreatedAt)
inst := provider.Instance{ID: formatProviderID(host.Name, dom.Name), UID: meta.UID, CreatedAt: meta.createdAt()}
dlog.V(2).Info("kvm2 listed instance", "id", inst.ID, "uid", inst.UID, "createdAt", inst.CreatedAt)
out = append(out, inst)
}
}
+42 -7
View File
@@ -72,8 +72,43 @@ func TestListByTag_acrossHostsInConfigOrder(t *testing.T) {
if got := listIDs(list); strings.Join(got, ",") != strings.Join(want, ",") {
t.Errorf("ListByTag = %v, want %v (config order, then name order)", got, want)
}
if list[0].State != provider.StateStopped || list[1].State != provider.StateProvisioning {
t.Errorf("states = %s, %s, want Stopped, Provisioning", list[0].State, list[1].State)
if list[0].UID != "uid-a" || list[1].UID != "uid-z" || list[2].UID != "uid-b" {
t.Errorf("uids = %s, %s, %s, want uid-a, uid-z, uid-b", list[0].UID, list[1].UID, list[2].UID)
}
}
func TestListByTag_doesNotReadStateOrAddresses(t *testing.T) {
t.Parallel()
fx := newFixture(t, 2)
id := fx.create(testName)
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcAgent, "10.0.0.5")
fx.fake.addDomain("proxy-off", libvirt.DomainShutoff, managedMetadata("uid-off", time.Now()))
before := len(fx.fake.callLog())
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if len(list) != 2 || list[0].ID != id || list[1].ID != formatProviderID(testHost, "proxy-off") {
t.Fatalf("ListByTag = %v, want the two managed domains", listIDs(list))
}
for _, inst := range list {
if inst.State != "" || inst.IP != "" {
t.Errorf("listed %s = state %q ip %q, want both empty: GC does not use them", inst.ID, inst.State, inst.IP)
}
if inst.UID == "" || inst.CreatedAt.IsZero() {
t.Errorf("listed %s lacks uid or createdAt: %+v", inst.ID, inst)
}
}
for _, rpc := range fx.fake.callLog()[before:] {
switch rpc {
case "ConnectListAllDomains", "DomainGetMetadata":
default:
t.Errorf("ListByTag made %s; want only the listing and one metadata read per domain", rpc)
}
}
if got := fx.logs.matching("kvm2 listed instance"); len(got) != 2 || strings.Contains(got[0], `"state"=`) {
t.Errorf("listed instance lines = %v, want two without a state", got)
}
}
@@ -126,12 +161,12 @@ func TestListByTag_unreadableDomainSkipped(t *testing.T) {
line: "skipping domain with unreadable metadata",
},
{
name: "state rpc fails for the first domain only",
name: "domain gone between listing and metadata read",
setup: func(fx *fixture) {
fx.fake.addDomain("proxy-bad", libvirt.DomainRunning, managedMetadata("uid-bad", time.Now()))
var calls int
fx.fake.mu.Lock()
fx.fake.hooks["DomainGetState"] = func() error {
fx.fake.hooks["DomainGetMetadata"] = func() error {
calls++
if calls == 1 {
return fakeErr(libvirt.ErrNoDomain, "Domain not found")
@@ -140,7 +175,7 @@ func TestListByTag_unreadableDomainSkipped(t *testing.T) {
}
fx.fake.mu.Unlock()
},
line: "skipping domain that could not be inspected",
line: "skipping domain with unreadable metadata",
},
}
for _, tc := range tests {
@@ -172,7 +207,7 @@ func TestListByTag_includesInactiveDomains(t *testing.T) {
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if len(list) != 1 || list[0].ID != id || list[0].State != provider.StateStopped {
t.Errorf("ListByTag = %+v, want the shut-off domain as Stopped", list)
if len(list) != 1 || list[0].ID != id || list[0].UID != testUID {
t.Errorf("ListByTag = %+v, want the shut-off domain listed with its uid", list)
}
}