kvm2: apply the milestone review
Shorter start wait with immediate verdicts for dead or paused domains, ListByTag without state or address RPCs, Timeout for an expired check round, no adoption of unmarked domains, reference-counted keyed locks, dead options removed, HEAD fallback on any non-2xx. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
@@ -172,7 +172,8 @@ closes the SSH connection to unblock it.
|
||||
|
||||
kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract
|
||||
prov.Create ──► placement checked against the catalogue (Permanent before any dial)
|
||||
DomainLookupByName: exists → uid guard (foreign uid = Permanent), ─┐ every step
|
||||
DomainLookupByName: exists → mark+uid guard (no mark or foreign ─┐ every step
|
||||
uid = Permanent; paused = Permanent), │
|
||||
start if needed, wait for running, done │ idempotent,
|
||||
else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a
|
||||
+ 16 MiB must fit, else ErrQuotaExceeded (reconciler │ per-name lock
|
||||
@@ -184,7 +185,7 @@ prov.Create ──► placement checked against the catalogue (Permanent before
|
||||
seed ISO (always rebuilt) │
|
||||
DomainDefineXML (<metadata> in kvm2's own namespace: │
|
||||
managed, uid, createdAt) + DomainCreate, poll until │
|
||||
running (2m cap) ─┘
|
||||
running (15s cap; a domain that dies at once = Transient) ─┘
|
||||
prov.Get ──► DomainLookupByName + DomainGetMetadata + DomainGetState
|
||||
+ DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4)
|
||||
running without IP = Provisioning; nostate = Provisioning;
|
||||
|
||||
+52
-24
@@ -41,9 +41,12 @@ ID.
|
||||
**Answer:** `name-is-key`. libvirt addresses domains by name
|
||||
(`DomainLookupByName`); `req.Name` is used verbatim as the domain name and as
|
||||
the base of the per-VM volume names. Create looks the name up first and adopts
|
||||
an existing domain only if its metadata `uid` is empty or equals `req.UID`; a
|
||||
different non-empty `uid` returns `ErrPermanent` naming both UIDs. This closes
|
||||
the contract's "adoption is not verified" limit for this provider.
|
||||
an existing domain only if it carries the kvm2 mark (`managed` true, entry 8)
|
||||
and its metadata `uid` is empty or equals `req.UID`. A domain without the
|
||||
mark returns `ErrPermanent` saying it was not created by this provider: kvm2
|
||||
never starts or takes over a VM someone else defined under that name. A
|
||||
different non-empty `uid` returns `ErrPermanent` naming both UIDs. This
|
||||
closes the contract's "adoption is not verified" limit for this provider.
|
||||
|
||||
**Evidence:** `DomainLookupByName(Name string) (Domain, error)` in the
|
||||
go-libvirt module; contract §2.1 idempotency and §13.
|
||||
@@ -201,15 +204,24 @@ libvirt has no locks or tasks; every RPC is synchronous.
|
||||
**Answer:** `create-return`: `blocking`. `async-ops`: `none`. Create returns
|
||||
when the domain is running, holding one reconcile worker for the duration,
|
||||
including the first image download on a host (minutes on a cold pool; the
|
||||
guide says so). The wait after `DomainCreate` is capped at 2 minutes; every
|
||||
RPC has its own deadline (entry 18).
|
||||
guide says so). The wait after `DomainCreate` is capped at 15 seconds:
|
||||
`DomainCreate` is synchronous in libvirt, so the first `DomainGetState`
|
||||
normally answers running and the cap only covers state settling. Two rules
|
||||
keep the wait from pinning a worker: a state after the start that is neither
|
||||
active nor `nostate` (shutoff, crashed, paused, pmsuspended, shutdown) is
|
||||
`ErrTransient` at once, since polling will not bring that guest up; and a
|
||||
domain found paused or pmsuspended before the start is `ErrPermanent`
|
||||
(`cannot be started by the operator; resume or destroy it on the host`), an
|
||||
admin's doing that `DomainCreate` would refuse and the reconciler must not
|
||||
retry. Every RPC has its own deadline (entry 18).
|
||||
|
||||
Steps, each idempotent, in order, all under the per-name lock (entry 19):
|
||||
|
||||
1. Validate placement (entry 3). No backend call yet.
|
||||
2. `DomainLookupByName`. Found: adoption guard (entry 2), then start it if it
|
||||
is not running (`DomainCreate`; `ErrOperationInvalid` means already
|
||||
running), return the ID.
|
||||
is not running (paused or pmsuspended is `ErrPermanent`; `DomainCreate`;
|
||||
`ErrOperationInvalid` means already running), poll as in step 9, return
|
||||
the ID.
|
||||
3. Pool space precheck (entry 13, quota).
|
||||
4. Ensure the base image (entry 4): marker present and matching means done.
|
||||
5. Overlay volume `<name>.qcow2`, qcow2 backed by the base file with
|
||||
@@ -220,7 +232,9 @@ Steps, each idempotent, in order, all under the per-name lock (entry 19):
|
||||
7. `StorageVolGetPath` for the overlay and the seed.
|
||||
8. `DomainDefineXML` with file-type disks (entry 15's AppArmor note), the
|
||||
metadata element, firmware per image.
|
||||
9. `DomainCreate`, then poll `DomainGetState` until running.
|
||||
9. `DomainCreate`, then poll `DomainGetState` until running: `nostate` is
|
||||
polled again, any other inactive state is `ErrTransient` at once, and the
|
||||
15-second cap is `ErrTransient` too.
|
||||
10. Return `hosts/<host>/domains/<name>`.
|
||||
|
||||
A retry after a crash finds whatever the previous attempt got to: an existing
|
||||
@@ -267,13 +281,16 @@ contract §2.3.
|
||||
**Answer:** `fail-whole`, per host, with single unreadable domains skipped.
|
||||
Scope: every host in `hosts`, in config order, `ConnectListAllDomains` with
|
||||
active and inactive domains. For each domain, read the metadata element; keep
|
||||
those with `managed` true, then read state and addresses for the `Instance`.
|
||||
A domain whose metadata or state cannot be read (mid teardown, corrupt
|
||||
element) is skipped with a V(1) line; a host that cannot be dialled or whose
|
||||
listing fails fails the whole call, so GC skips this provider for the sweep
|
||||
rather than act on a partial view. Filtering is client-side; libvirt has no
|
||||
label index. Cost: one `DomainGetMetadata` per domain per sweep, about half a
|
||||
millisecond each over the tunnel.
|
||||
those with `managed` true and build the `Instance` from the domain name and
|
||||
the element alone: `ID`, `UID` and `CreatedAt`, which is all GC reads
|
||||
(contract §2.4). `State` and `IP` are left empty; no state or address RPC is
|
||||
made. A domain whose metadata cannot be read (mid teardown, corrupt element)
|
||||
is skipped with a V(1) line, unless ctx ended, which fails the call; a host
|
||||
that cannot be dialled or whose listing fails fails the whole call, so GC
|
||||
skips this provider for the sweep rather than act on a partial view.
|
||||
Filtering is client-side; libvirt has no label index. Cost: exactly one
|
||||
`DomainGetMetadata` per domain per sweep, about half a millisecond each over
|
||||
the tunnel.
|
||||
|
||||
**Evidence:** execution log run 6 (per-domain metadata cost); contract §2.4
|
||||
and §4.7.
|
||||
@@ -390,9 +407,11 @@ pooled connection, so a check that runs out of time closes only its own
|
||||
connection and cannot disrupt a reconcile. Cost: one SSH handshake and four
|
||||
cheap RPCs per host per round.
|
||||
|
||||
Reasons: `Reachable`; `Timeout` (ctx ended before the check finished);
|
||||
`Unreachable` (TCP connect or the SSH transport failed, or the link died
|
||||
mid-check); `Unauthorized` (SSH authentication refused, host key unknown or
|
||||
Reasons: `Reachable`; `Timeout` (the round's ctx ended before the check
|
||||
finished, whichever stage was running, the dial included, or an RPC's own
|
||||
budget ran out once the connection was up); `Unreachable` (TCP connect or the
|
||||
SSH transport failed, including on the dial's own 10 s budget while the round
|
||||
still had time, or the link died mid-check); `Unauthorized` (SSH authentication refused, host key unknown or
|
||||
changed); `Forbidden` (`ErrAccessDenied` or `ErrAuthFailed` from libvirt: the
|
||||
socket refused the user); `APIError` (any other libvirt error);
|
||||
`CredentialsUnavailable` (the key or known_hosts file is missing or
|
||||
@@ -458,7 +477,7 @@ goroutine and waits on the result, `ctx.Done()` and a timer:
|
||||
| `DomainCreate` | 90 s |
|
||||
| `StorageVolUpload` (seed) | 60 s |
|
||||
| `StorageVolUpload` (image), HTTP download | ctx, capped at 20 m |
|
||||
| wait for running after `DomainCreate` | 2 m |
|
||||
| wait for running after `DomainCreate` | 15 s (entry 10) |
|
||||
| HTTP client for images | no client timeout; the per-request ctx carries the cap |
|
||||
|
||||
When a call is abandoned (ctx cancelled or budget exceeded), the adapter
|
||||
@@ -479,10 +498,14 @@ every 15 s) marks the connection dead on failure.
|
||||
**Answer:** shared state is the host connection table (one pooled connection
|
||||
per host, dialled lazily, under a mutex with a per-host dial mutex so
|
||||
concurrent callers dial once and share the result), the per-`(host, volume)`
|
||||
download locks, and a keyed mutex per instance name. Create and Delete for the
|
||||
same name run under that name's lock, so two Deletes (GC and the reconciler)
|
||||
never race on destroy and undefine, and a Delete cannot interleave with a
|
||||
Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes
|
||||
download locks, and a lock per instance name. Both lock sets are one instance
|
||||
each of `keyedLocks`: a one-slot semaphore per key, so a waiter gives up when
|
||||
its ctx ends, reference-counted so an entry is removed from the map once no
|
||||
caller holds or waits for it and the map does not grow with every name ever
|
||||
seen, while two waiters on one key always share one semaphore. Create and
|
||||
Delete for the same name run under that name's lock, so two Deletes (GC and
|
||||
the reconciler) never race on destroy and undefine, and a Delete cannot
|
||||
interleave with a Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes
|
||||
RPCs on one connection, so calls for different names proceed concurrently
|
||||
on the shared connection. Three reconciles, a GC sweep and a check round at
|
||||
once are safe; the check uses its own connections.
|
||||
@@ -598,7 +621,7 @@ entry after the user confirms.
|
||||
|
||||
## 24. Known limits accepted at design time
|
||||
|
||||
**Answer:** eight limits, each with its workaround:
|
||||
**Answer:** nine limits, each with its workaround:
|
||||
|
||||
- **Create blocks**, including the first image download per host, holding one
|
||||
of three reconcile workers. Workaround: warm the cache with one Proxy per
|
||||
@@ -615,6 +638,11 @@ entry after the user confirms.
|
||||
isolated on its own connection.
|
||||
- **`Content-Length` is required** for image downloads; a chunked mirror is
|
||||
rejected as Permanent.
|
||||
- **A paused or pmsuspended domain is not started.** A repeat `Create` for
|
||||
a domain in that state is `ErrPermanent` (entry 10), so the Proxy latches
|
||||
as Failed until its spec changes, rather than retrying a `DomainCreate`
|
||||
that libvirt refuses. Resume the domain on the host (`virsh resume`) and
|
||||
touch the spec, or delete the Proxy.
|
||||
- **No v1 log-format compatibility**: the `libvirt call` lines have the same
|
||||
keys where the same facts exist, but dashboards keyed on `kvm` logger names
|
||||
need `kvm2`.
|
||||
|
||||
+27
-20
@@ -235,10 +235,12 @@ Its `<metadata>` carries one element in kvm2's own namespace,
|
||||
|
||||
That is how orphan GC recognises its own VMs, since libvirt has no
|
||||
labels, and how a repeat `Create` knows whether an existing domain of the
|
||||
same name belongs to this Proxy: a different non-empty `uid` is a
|
||||
permanent error naming both UIDs; an empty one is adopted. The namespace
|
||||
is kvm2's own, so the `kvm` provider's domains are invisible to it (see
|
||||
*Migrating from kvm*).
|
||||
same name belongs to this Proxy: a domain without the element is a
|
||||
permanent error (`carries no kvm2 mark; it was not created by this
|
||||
provider`), a different non-empty `uid` is a permanent error naming both
|
||||
UIDs, and a marked domain with a matching or empty `uid` is adopted. The
|
||||
namespace is kvm2's own, so the `kvm` provider's domains are invisible to
|
||||
it (see *Migrating from kvm*).
|
||||
|
||||
The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled
|
||||
`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
|
||||
@@ -321,8 +323,10 @@ Before creating any volume, `Create` reads `StoragePoolGetInfo` and
|
||||
compares the pool's free space with what it is about to commit
|
||||
(`checkPoolSpace` and `neededBytes` in `image.go`): the image's
|
||||
`Content-Length` when the base is not cached yet (found with a `HEAD`
|
||||
request, or a `GET` whose body is closed at once when the server refuses
|
||||
`HEAD`), plus `diskSizeGb` in bytes for the overlay, plus 16 MiB for the
|
||||
request, or a `GET` whose body is closed at once when `HEAD` answers
|
||||
anything but 2xx, since mirrors refuse it with 405, 403 or 501 alike; the
|
||||
`GET`'s status is then what counts), plus `diskSizeGb` in bytes for the
|
||||
overlay, plus 16 MiB for the
|
||||
seed. Below that it returns `ErrQuotaExceeded` with a message naming the
|
||||
pool, the bytes available and the bytes needed. The reconciler answers
|
||||
quota with a wait, not a failure: the Proxy shows `Provisioned=False`
|
||||
@@ -347,9 +351,12 @@ by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The
|
||||
|
||||
What does not port is the VMs. kvm2 marks its domains in its own metadata
|
||||
namespace and neither lists nor adopts domains the `kvm` provider made:
|
||||
`ListByTag` skips them, so orphan GC never sees them, and a Proxy's
|
||||
`spec.provider` is immutable, so an existing Proxy cannot be moved to the
|
||||
new entry anyway. Do it in this order:
|
||||
`ListByTag` skips them, so orphan GC never sees them, and `Create` refuses
|
||||
a domain of the same name that lacks the kvm2 mark with a permanent error
|
||||
rather than start it (a Proxy recreated under the new entry gets the same
|
||||
deterministic name, so this is what happens when step 1 below is skipped).
|
||||
A Proxy's `spec.provider` is immutable, so an existing Proxy cannot be
|
||||
moved to the new entry anyway. Do it in this order:
|
||||
|
||||
1. Delete the Proxies on the `kvm` entry and wait for them to go; the
|
||||
`kvm` provider removes their domains and volumes.
|
||||
@@ -437,7 +444,7 @@ applied by `call`):
|
||||
| `DomainCreate` | 90 s |
|
||||
| `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s |
|
||||
| `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier |
|
||||
| wait for running after `DomainCreate` | 2 m, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`) |
|
||||
| wait for running after `DomainCreate` | 15 s, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`); `DomainCreate` is synchronous, so the first poll normally answers running, and a domain that is shut off, crashed or paused right after the start fails at once rather than waiting out the cap |
|
||||
| the `HEAD` request of the pool-space precheck | 30 s |
|
||||
| the courtesy `ConnectClose` when a connection is closed | 5 s |
|
||||
|
||||
@@ -499,10 +506,10 @@ A failing target says where the check broke (`checkReason` in `check.go`):
|
||||
|---|---|---|
|
||||
| `Reachable` | ready | |
|
||||
| `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host |
|
||||
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
|
||||
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget while the round still had time, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
|
||||
| `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) |
|
||||
| `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group |
|
||||
| `Timeout` | the connection was up, but the monitor's `--provider-check-timeout` (default 15s) or an RPC's budget ran out before the check finished | a hung `libvirtd` or a saturated link |
|
||||
| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out before the check finished, whichever stage was running, the dial included; or the connection was up and an RPC's own budget ran out | a hung `libvirtd`, a saturated link, or a round timeout shorter than the dial takes |
|
||||
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
|
||||
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
|
||||
| `APIError` | any other libvirt error | the message carries libvirt's own text |
|
||||
@@ -518,10 +525,10 @@ waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
|
||||
|
||||
| Class | What lands there |
|
||||
|---|---|
|
||||
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
|
||||
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
|
||||
| `ErrQuotaExceeded` | the pool-space precheck |
|
||||
| `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
|
||||
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring |
|
||||
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring or the domain going shut off or crashed right after its start |
|
||||
|
||||
Error strings name paths, hosts, volumes and libvirt's own text. They
|
||||
never contain key bytes, the known_hosts content or cloud-init.
|
||||
@@ -556,12 +563,12 @@ never contain key bytes, the known_hosts content or cloud-init.
|
||||
- **`Content-Length` is required** for image downloads; a mirror that
|
||||
only chunks is rejected permanently. Point the catalogue at one that
|
||||
sends the length.
|
||||
- **A paused or pmsuspended domain cannot be adopted.** A repeat `Create`
|
||||
for a domain in that state calls `DomainCreate`, which libvirt refuses
|
||||
as "already running"; the provider then waits for the domain to report
|
||||
running, the 2-minute wait expires, and the call fails transiently on
|
||||
every retry. Resume the domain on the host (`virsh resume <name>`) or
|
||||
delete the Proxy.
|
||||
- **A paused or pmsuspended domain is not started.** A repeat `Create`
|
||||
for a domain in that state fails permanently (`is DomainPaused and
|
||||
cannot be started by the operator; resume or destroy it on the host`),
|
||||
so the Proxy shows `Failed` until its spec changes; the operator never
|
||||
resumes a VM an admin paused. Resume it on the host
|
||||
(`virsh resume <name>`) and touch the spec, or delete the Proxy.
|
||||
- **Parser text in credential errors.** A key or known_hosts file that
|
||||
does not parse fails with the parser's own message, which can quote a
|
||||
PEM header type or a malformed known_hosts token. Key bytes are never
|
||||
|
||||
@@ -12,7 +12,7 @@ Plan: [docs/plans/2026-09-16-1333-kvm2-provider.md](../plans/2026-09-16-1333-kvm
|
||||
- [x] Step 7 — Checker and the CRD comment
|
||||
- [x] Step 8 — Deployment files, Makefile, integration test
|
||||
- [x] Step 9 — Documentation
|
||||
- [ ] Step 10 — Milestone review
|
||||
- [x] Step 10 — Milestone review
|
||||
- [ ] Step 11 — Real-host validation, changelog, MR
|
||||
|
||||
## Step 0 — Plan file and branch
|
||||
@@ -338,3 +338,45 @@ resolves. The one added em-dash is the `Leader-elected` cell of the new
|
||||
components-table row, where the table's own convention wins.
|
||||
|
||||
Attestation: "files read outside the allowed set: none".
|
||||
|
||||
## Step 10 — Milestone review
|
||||
|
||||
2026-09-16 21:19. The `go-operator-reviewer` agent reviewed the package
|
||||
against the contract and the spec (v1 code off limits to it as well) and
|
||||
returned seven findings, all accepted and applied by a fresh fix agent in
|
||||
one commit with regression tests:
|
||||
|
||||
1. `waitRunning`'s two-minute cap only ever elapsed in the bad cases and
|
||||
pinned a reconcile worker; now 15 s, a domain that dies right after
|
||||
start is Transient at once, and a paused or pmsuspended domain is
|
||||
Permanent before `DomainCreate` (an admin's doing).
|
||||
2. `ListByTag` read state and addresses that GC never uses, one agent RPC
|
||||
per active domain per sweep; it now builds instances from the name and
|
||||
the metadata only (`TestListByTag_doesNotReadStateOrAddresses`).
|
||||
3. A health round that ran out of time was blamed on the host
|
||||
(`Unreachable`); `checkHost` now reports `Timeout` whenever the round's
|
||||
ctx ended, whichever dial stage was running, and the promptness test
|
||||
uses the production error shape.
|
||||
4. `Create` adopted an unmarked domain and never wrote the mark, so GC could
|
||||
not see it; an unmarked domain under the name is now Permanent
|
||||
(`TestCreate_adoptsOnlyMarkedDomains`).
|
||||
5. `hostTable.closeAll` had no production caller and `withBudgets` could not
|
||||
work (the dialer captured the defaults first); both removed.
|
||||
6. The per-name and per-image lock maps grew without bound; replaced by a
|
||||
reference-counted `keyedLocks` whose entries vanish when uncontended.
|
||||
7. `imageSizeIfUncached` falls back to GET on any non-2xx HEAD, and two
|
||||
Delete no-op lines moved from V(0) to V(1).
|
||||
|
||||
The spec (entries 2, 10, 12, 16, 18, 19, 24), the guide, the appendix and
|
||||
the architecture flow were updated in the same commit. The reviewer also
|
||||
listed what it checked and found correct: the error boundary, the state
|
||||
rules, Delete's idempotency and the replacement window, ListByTag's
|
||||
partial-failure policy, the abandon policy and double-close guards,
|
||||
resource handling on every path, and secret hygiene in the format strings.
|
||||
|
||||
Lint: `bin/golangci-lint` still panics on this toolchain (Go 1.27 against
|
||||
a staticcheck built for 1.26), reproducible on untouched packages, so
|
||||
`make lint` could not gate this step; `go vet` with every build tag is
|
||||
clean. Tests: `go test -race -count=3 ./internal/provider/kvm2/` (168
|
||||
top-level tests) and `go test -short ./...` pass; `make test` with envtest
|
||||
runs next.
|
||||
|
||||
+33
-25
@@ -30,12 +30,12 @@ and the symbol that shows it.
|
||||
| Axis | kubernetes | gcp | kvm | kvm2 |
|
||||
|---|---|---|---|---|
|
||||
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) |
|
||||
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 2-minute poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
|
||||
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
|
||||
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
|
||||
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) |
|
||||
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) |
|
||||
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) |
|
||||
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended (`ListByTag`) |
|
||||
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) |
|
||||
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) |
|
||||
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) |
|
||||
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) |
|
||||
@@ -56,7 +56,7 @@ against.
|
||||
|
||||
| Feature | kubernetes | gcp | kvm | kvm2 |
|
||||
|---|---|---|---|---|
|
||||
| Create verifies the adopted instance's UID | no | no | no | **yes**: a different non-empty UID in the metadata is Permanent, an empty one is adopted (`create`) |
|
||||
| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) |
|
||||
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image |
|
||||
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
|
||||
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no |
|
||||
@@ -372,9 +372,10 @@ same libvirt.
|
||||
logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
|
||||
the per-name lock (`lockName`) and on the pooled connection
|
||||
(`hostTable.get`), `create`: `DomainLookupByName`; found means the
|
||||
adoption guard (a non-empty metadata UID that differs from `req.UID` is
|
||||
Permanent) and `ensureRunning`. Otherwise `StoragePoolLookupByName`,
|
||||
`imageSizeIfUncached` (marker lookup, then `HEAD` or a closed `GET`) and
|
||||
adoption guard (no kvm2 mark, or a non-empty metadata UID that differs
|
||||
from `req.UID`, is Permanent) and `ensureRunning`. Otherwise
|
||||
`StoragePoolLookupByName`, `imageSizeIfUncached` (marker lookup, then
|
||||
`HEAD`, or a closed `GET` when `HEAD` is not 2xx) and
|
||||
`checkPoolSpace` with `neededBytes` (image if uncached, plus
|
||||
`diskSizeGb`, plus 16 MiB), then `ensureImage` (`cachedImage` on the
|
||||
pooled connection, `fetchImage` streaming into `uploadImage` on a
|
||||
@@ -382,10 +383,11 @@ same libvirt.
|
||||
`ensureOverlay` (`ErrStorageVolExist` means reuse), `writeSeed` (delete,
|
||||
build, upload; always rebuilt), two `StorageVolGetPath`, `buildDomainXML`
|
||||
with file-type disks and the metadata element, `DomainDefineXML`,
|
||||
`ensureRunning` (`DomainGetState`; `DomainCreate` with
|
||||
`ErrOperationInvalid` as already running; `waitRunning` polls every
|
||||
500 ms for up to 2 minutes, Transient on expiry). `Create` returns only
|
||||
when the domain is running.
|
||||
`ensureRunning` (`DomainGetState`, where paused or pmsuspended is
|
||||
Permanent; `DomainCreate` with `ErrOperationInvalid` as already running;
|
||||
`waitRunning` polls every 500 ms for up to 15 seconds, Transient on
|
||||
expiry and at once for any inactive state other than `nostate`).
|
||||
`Create` returns only when the domain is running.
|
||||
- **Get and state.** `mapState(state, ip)`:
|
||||
|
||||
| libvirt state | IP | Provider state |
|
||||
@@ -398,8 +400,8 @@ same libvirt.
|
||||
`get`: `DomainLookupByName` (only `ErrNoDomain` reaches NotFound),
|
||||
`readMetadata` (`DomainGetMetadata` with `metadataURI`;
|
||||
`ErrNoDomainMetadata` means unmanaged with an empty UID; an element that
|
||||
does not parse is Permanent), `inspect` (`DomainGetState`, then for an
|
||||
active domain `address`). `addressSources` asks the guest agent first,
|
||||
does not parse is Permanent), then `DomainGetState` and, for an active
|
||||
domain, `address`. `addressSources` asks the guest agent first,
|
||||
then ARP; a source that fails is skipped at V(2), only ctx ending is an
|
||||
error, so a cancelled Get is Transient and not a false Provisioning.
|
||||
`firstIPv4` drops loopback, link-local and unspecified addresses and an
|
||||
@@ -413,9 +415,12 @@ same libvirt.
|
||||
when nil is returned.
|
||||
- **ListByTag.** `ConnectListAllDomains` (active and inactive) on every
|
||||
host in config order; `readMetadata` and `managed` keep kvm2's own
|
||||
domains; `inspect` fills state and address. A domain whose metadata or
|
||||
state cannot be read is skipped at V(1), unless ctx ended, which fails
|
||||
the call. A host that cannot be dialled or listed fails the whole call.
|
||||
domains, and the `Instance` is `ID`, `UID` and `CreatedAt` from the name
|
||||
and the element, with `State` and `IP` empty: GC reads nothing else
|
||||
(contract §2.4), so no state or address RPC is made. A domain whose
|
||||
metadata cannot be read is skipped at V(1), unless ctx ended, which
|
||||
fails the call. A host that cannot be dialled or listed fails the whole
|
||||
call.
|
||||
- **Errors.** `classify(op, err)`: an error already carrying a class keeps
|
||||
it; libvirt codes through `classifyLibvirt` (`ErrNoDomain` and
|
||||
`ErrNoStorageVol` NotFound, Transient when `op == "create"`;
|
||||
@@ -436,14 +441,16 @@ same libvirt.
|
||||
`ConnectGetHostname`, `StoragePoolLookupByName`, `StoragePoolGetInfo`;
|
||||
a pool that is not running is `poolNotActiveError`. Details are filled
|
||||
as facts arrive (`user` and `pool` always; `libvirtVersion`, `hostname`,
|
||||
`poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkReason`: a
|
||||
`dialError` is judged by its stage (`credentials` gives
|
||||
`CredentialsUnavailable`; `ssh` with a `knownhosts.KeyError` or the auth
|
||||
text gives `Unauthorized`; any other stage, including a dial that ran
|
||||
out of time, gives `Unreachable`); then `context.DeadlineExceeded` or
|
||||
`Canceled` `Timeout`, `ErrAccessDenied` or `ErrAuthFailed` `Forbidden`,
|
||||
`ErrNoStoragePool` `PoolNotFound`, any other libvirt code `APIError`,
|
||||
anything else `Unreachable`; `PoolNotActive` from the pool state.
|
||||
`poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkHost` answers
|
||||
`Timeout` whenever the round's ctx ended, whatever stage was running and
|
||||
whatever shape the error has; otherwise `checkReason`: a `dialError` is
|
||||
judged by its stage (`credentials` gives `CredentialsUnavailable`; `ssh`
|
||||
with a `knownhosts.KeyError` or the auth text gives `Unauthorized`; any
|
||||
other stage, including a dial that ran out of its own budget, gives
|
||||
`Unreachable`); then `context.DeadlineExceeded` or `Canceled` `Timeout`,
|
||||
`ErrAccessDenied` or `ErrAuthFailed` `Forbidden`, `ErrNoStoragePool`
|
||||
`PoolNotFound`, any other libvirt code `APIError`, anything else
|
||||
`Unreachable`; `PoolNotActive` from the pool state.
|
||||
- **Connections.** `hostTable` keeps one pooled connection per host,
|
||||
dialled lazily under a per-host one-slot semaphore so waiting ends with
|
||||
the caller's ctx; a dead backend (`liveness`) is closed and redialled on
|
||||
@@ -510,8 +517,9 @@ same libvirt.
|
||||
- Cancelling one operation cuts off the others on that host once: a
|
||||
cancelled `call` kills the pooled connection.
|
||||
- `Content-Length` is required for image downloads.
|
||||
- A paused or pmsuspended domain cannot be adopted by a repeat `Create`:
|
||||
`DomainCreate` refuses, `waitRunning` expires, Transient on every retry.
|
||||
- A paused or pmsuspended domain is not started by a repeat `Create`:
|
||||
`ensureRunning` returns Permanent, so the Proxy latches until its spec
|
||||
changes; the admin resumes the VM on the host.
|
||||
- Credential parse errors carry the parser's own text, which can quote a
|
||||
PEM header type or a malformed known_hosts token (never key bytes).
|
||||
- No reused-overlay check: an existing overlay is reused without checking
|
||||
|
||||
@@ -62,9 +62,15 @@ func (p *Provider) checkHost(ctx context.Context, host provider.KVMHost) provide
|
||||
ts.Reason = provider.ReasonReachable
|
||||
default:
|
||||
var inactive *poolNotActiveError
|
||||
if errors.As(err, &inactive) {
|
||||
switch {
|
||||
case ctx.Err() != nil:
|
||||
// The round ran out of time, whichever stage was running: the
|
||||
// error's shape (a dial that saw the ctx end, an abandoned
|
||||
// call) says nothing the admin should act on.
|
||||
ts.Reason = provider.ReasonTimeout
|
||||
case errors.As(err, &inactive):
|
||||
ts.Reason = reasonPoolNotActive
|
||||
} else {
|
||||
default:
|
||||
ts.Reason = checkReason(err)
|
||||
}
|
||||
ts.Message = err.Error()
|
||||
@@ -121,10 +127,11 @@ func (e *poolNotActiveError) Error() string {
|
||||
return fmt.Sprintf("storage pool %q is %s", e.pool, e.state)
|
||||
}
|
||||
|
||||
// checkReason maps a failed probe to a reason. A dial failure is judged by
|
||||
// its stage, so a TCP connect or an SSH handshake that ran out of time is
|
||||
// Unreachable, not Timeout: the host did not answer. Timeout is for ctx
|
||||
// ending or a budget expiring once the connection was up.
|
||||
// checkReason maps a failed probe to a reason once checkHost has ruled out
|
||||
// the round's ctx ending. A dial failure is judged by its stage, so a TCP
|
||||
// connect or an SSH handshake that ran out of the dial budget on its own is
|
||||
// Unreachable, not Timeout: the host did not answer. Timeout is for a
|
||||
// budget expiring once the connection was up.
|
||||
func checkReason(err error) string {
|
||||
var de *dialError
|
||||
if errors.As(err, &de) {
|
||||
|
||||
@@ -33,7 +33,9 @@ func checkContext() (context.Context, *logCapture) {
|
||||
}
|
||||
|
||||
// checkDialer hands out a distinct fake per host name, fails the hosts in
|
||||
// errs, counts every dial, and, with block set, waits for ctx to end.
|
||||
// errs, counts every dial, and, with block set, waits for ctx to end and
|
||||
// then fails the way dialSSHSocket does when the TCP connect sees the ctx
|
||||
// end: a dialError at the tcp stage wrapping the ctx's error.
|
||||
type checkDialer struct {
|
||||
backends map[string]*fakeBackend
|
||||
errs map[string]error
|
||||
@@ -46,7 +48,7 @@ func (d *checkDialer) dial(ctx context.Context, host provider.KVMHost) (backend,
|
||||
d.dials.Add(1)
|
||||
if d.block {
|
||||
<-ctx.Done()
|
||||
return nil, fmt.Errorf("%w: dialing %s: %w", provider.ErrTransient, host.Name, ctx.Err())
|
||||
return nil, &dialError{stage: stageTCP, host: host.Name, err: fmt.Errorf("dial tcp %s: %w", host.Address, ctx.Err())}
|
||||
}
|
||||
if err := d.errs[host.Name]; err != nil {
|
||||
return nil, err
|
||||
@@ -140,13 +142,18 @@ func TestCheck_hostsCheckedConcurrently(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCheck_expiredContextReturnsPromptly drives the production shape of a
|
||||
// round that ran out of time during the dial: the dialer fails at the tcp
|
||||
// stage wrapping context.DeadlineExceeded, which checkReason alone would
|
||||
// call Unreachable; the verdict must still be Timeout because the round's
|
||||
// ctx ended.
|
||||
func TestCheck_expiredContextReturnsPromptly(t *testing.T) {
|
||||
t.Parallel()
|
||||
cfg := twoHosts()
|
||||
p := newProvider(testProvider, cfg, &checkDialer{block: true})
|
||||
ctx, _ := checkContext()
|
||||
ctx, cancel := context.WithCancel(ctx)
|
||||
cancel()
|
||||
ctx, cancel := context.WithTimeout(ctx, 0)
|
||||
defer cancel()
|
||||
start := time.Now()
|
||||
got := p.Check(ctx)
|
||||
if elapsed := time.Since(start); elapsed > time.Second {
|
||||
@@ -159,8 +166,8 @@ func TestCheck_expiredContextReturnsPromptly(t *testing.T) {
|
||||
if ts.Ready || ts.Reason != provider.ReasonTimeout {
|
||||
t.Errorf("target %s = ready %v reason %s, want not ready, Timeout", ts.Name, ts.Ready, ts.Reason)
|
||||
}
|
||||
if ts.Message == "" {
|
||||
t.Errorf("target %s has no message", ts.Name)
|
||||
if !strings.Contains(ts.Message, context.DeadlineExceeded.Error()) || !strings.Contains(ts.Message, string(stageTCP)) {
|
||||
t.Errorf("target %s message = %q, want the tcp stage and the deadline named", ts.Name, ts.Message)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -217,7 +224,7 @@ func TestCheck_reasonsTable(t *testing.T) {
|
||||
wantReason: provider.ReasonUnreachable,
|
||||
},
|
||||
{
|
||||
name: "tcp timed out",
|
||||
name: "tcp timed out on the dial budget but ctx is fine",
|
||||
dialErr: func(*testing.T) error {
|
||||
return &dialError{stage: stageTCP, host: testHost, err: fmt.Errorf("dial tcp 192.0.2.1:22: %w", context.DeadlineExceeded)}
|
||||
},
|
||||
@@ -248,7 +255,7 @@ func TestCheck_reasonsTable(t *testing.T) {
|
||||
wantReason: provider.ReasonUnreachable,
|
||||
},
|
||||
{
|
||||
name: "ssh handshake timed out",
|
||||
name: "ssh handshake timed out on the dial budget but ctx is fine",
|
||||
dialErr: func(*testing.T) error { return sshHandshake(context.DeadlineExceeded) },
|
||||
wantReason: provider.ReasonUnreachable,
|
||||
},
|
||||
|
||||
@@ -15,9 +15,11 @@ import (
|
||||
|
||||
// The wait after DomainCreate (spec entry 18): Create returns only once the
|
||||
// domain is running, so Get never sees a domain kvm2 defined but did not
|
||||
// start (spec entry 9).
|
||||
// start (spec entry 9). DomainCreate is synchronous, so the first read
|
||||
// normally succeeds; the cap only covers state settling and must not pin a
|
||||
// reconcile worker for long.
|
||||
const (
|
||||
defaultRunningWait = 2 * time.Minute
|
||||
defaultRunningWait = 15 * time.Second
|
||||
defaultRunningPoll = 500 * time.Millisecond
|
||||
)
|
||||
|
||||
@@ -75,6 +77,9 @@ func (p *Provider) create(ctx context.Context, log logr.Logger, be backend, host
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !meta.managed() {
|
||||
return permanentf("domain %s exists but carries no kvm2 mark; it was not created by this provider", req.Name)
|
||||
}
|
||||
if meta.UID != "" && meta.UID != req.UID {
|
||||
return permanentf("domain %s belongs to proxy uid %s, not %s", req.Name, meta.UID, req.UID)
|
||||
}
|
||||
@@ -211,14 +216,19 @@ func (p *Provider) writeSeed(ctx context.Context, log logr.Logger, be backend, p
|
||||
|
||||
// ensureRunning starts the domain unless it is active already, then waits
|
||||
// until libvirt reports it running. ErrOperationInvalid from DomainCreate
|
||||
// means another caller started it between the state read and the call.
|
||||
// means another caller started it between the state read and the call. A
|
||||
// paused or pmsuspended domain is an admin's doing and DomainCreate would
|
||||
// refuse it; the verdict is Permanent so the reconciler does not retry it.
|
||||
func (p *Provider) ensureRunning(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) error {
|
||||
state, _, err := be.DomainGetState(ctx, dom, 0)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
|
||||
}
|
||||
if isActive(libvirt.DomainState(state)) {
|
||||
switch s := libvirt.DomainState(state); {
|
||||
case isActive(s):
|
||||
return nil
|
||||
case s == libvirt.DomainPaused || s == libvirt.DomainPmsuspended:
|
||||
return permanentf("domain %s is %s and cannot be started by the operator; resume or destroy it on the host", dom.Name, s)
|
||||
}
|
||||
switch err := be.DomainCreate(ctx, dom); {
|
||||
case err == nil:
|
||||
@@ -233,7 +243,9 @@ func (p *Provider) ensureRunning(ctx context.Context, log logr.Logger, be backen
|
||||
|
||||
// waitRunning polls DomainGetState until the domain is active, the cap
|
||||
// passes (Transient: the next Create attempt finds the domain defined and
|
||||
// tries again) or ctx ends.
|
||||
// tries again) or ctx ends. Only nostate is worth another poll; a domain
|
||||
// that is shut off, crashed or paused right after its start is not going
|
||||
// to come up on its own, so that is Transient at once.
|
||||
func (p *Provider) waitRunning(ctx context.Context, be backend, dom libvirt.Domain) error {
|
||||
deadline := time.NewTimer(p.runningWait)
|
||||
defer deadline.Stop()
|
||||
@@ -244,8 +256,11 @@ func (p *Provider) waitRunning(ctx context.Context, be backend, dom libvirt.Doma
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
|
||||
}
|
||||
if isActive(libvirt.DomainState(state)) {
|
||||
switch s := libvirt.DomainState(state); {
|
||||
case isActive(s):
|
||||
return nil
|
||||
case s != libvirt.DomainNostate:
|
||||
return transientf("domain %s went %s right after start", dom.Name, s)
|
||||
}
|
||||
select {
|
||||
case <-ticker.C:
|
||||
|
||||
@@ -182,6 +182,64 @@ func TestCreate_returnsOnlyWhenRunning(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
func TestCreate_domainThatDiesAfterStartIsTransientAtOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
for _, state := range []libvirt.DomainState{libvirt.DomainCrashed, libvirt.DomainShutoff, libvirt.DomainPaused} {
|
||||
t.Run(state.String(), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
// The cap is well over the bound asserted below, so a poll that
|
||||
// ran to the cap would fail the test.
|
||||
fx := newFixture(t, 1, withRunningWait(time.Millisecond, 10*time.Second))
|
||||
fx.fake.stateAfterStart = state
|
||||
start := time.Now()
|
||||
id, err := fx.p.Create(fx.ctx, fx.req(testName))
|
||||
elapsed := time.Since(start)
|
||||
if id != "" {
|
||||
t.Errorf("Create = %q, want no ID", id)
|
||||
}
|
||||
assertProviderError(t, err, "create", provider.ErrTransient)
|
||||
if !strings.Contains(err.Error(), "right after start") {
|
||||
t.Errorf("error = %v, want the state after start named", err)
|
||||
}
|
||||
if elapsed > time.Second {
|
||||
t.Errorf("Create took %v, want it to give up at once, not poll to the cap", elapsed)
|
||||
}
|
||||
if got := fx.fake.callCount("DomainCreate"); got != 1 {
|
||||
t.Errorf("DomainCreate calls = %d, want 1", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreate_pausedDomainIsPermanent(t *testing.T) {
|
||||
t.Parallel()
|
||||
for _, state := range []libvirt.DomainState{libvirt.DomainPaused, libvirt.DomainPmsuspended} {
|
||||
t.Run(state.String(), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
fx := newFixture(t, 1)
|
||||
fx.fake.addDomain(testName, state, managedMetadata(testUID, time.Now()))
|
||||
id, err := fx.p.Create(fx.ctx, fx.req(testName))
|
||||
if id != "" {
|
||||
t.Errorf("Create = %q, want no ID", id)
|
||||
}
|
||||
assertProviderError(t, err, "create", provider.ErrPermanent)
|
||||
if !strings.Contains(err.Error(), "cannot be started by the operator") {
|
||||
t.Errorf("error = %v, want the admin action named", err)
|
||||
}
|
||||
for _, rpc := range []string{"DomainCreate", "DomainDefineXML", "DomainDestroy"} {
|
||||
if got := fx.fake.callCount(rpc); got != 0 {
|
||||
t.Errorf("%s calls = %d, want 0", rpc, got)
|
||||
}
|
||||
}
|
||||
fx.withDomain(testName, func(d *fakeDomain) {
|
||||
if d.state != state {
|
||||
t.Errorf("domain state = %s, want it left %s", d.state, state)
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreate_rejectsUnknownPlacement(t *testing.T) {
|
||||
t.Parallel()
|
||||
tests := []struct {
|
||||
@@ -248,24 +306,42 @@ func TestCreate_adoptionWithForeignUIDIsPermanent(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreate_adoptsMatchingOrEmptyUID(t *testing.T) {
|
||||
func TestCreate_adoptsOnlyMarkedDomains(t *testing.T) {
|
||||
t.Parallel()
|
||||
tests := []struct {
|
||||
name string
|
||||
metadata string
|
||||
wantAdopt bool
|
||||
}{
|
||||
{name: "matching uid", metadata: managedMetadata(testUID, time.Now())},
|
||||
{name: "empty uid", metadata: managedMetadata("", time.Now())},
|
||||
{name: "matching uid", metadata: managedMetadata(testUID, time.Now()), wantAdopt: true},
|
||||
{name: "managed, empty uid", metadata: managedMetadata("", time.Now()), wantAdopt: true},
|
||||
{name: "no metadata element", metadata: ""},
|
||||
{name: "element without the managed mark", metadata: `<instance><uid>` + testUID + `</uid></instance>`},
|
||||
{name: "managed false", metadata: `<instance><managed>false</managed><uid>` + testUID + `</uid></instance>`},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
fx := newFixture(t, 1)
|
||||
fx.fake.addDomain(testName, libvirt.DomainRunning, tc.metadata)
|
||||
fx.create(testName)
|
||||
if got := fx.fake.callCount("DomainDefineXML"); got != 0 {
|
||||
t.Errorf("DomainDefineXML calls = %d, want 0 (adopted)", got)
|
||||
id, err := fx.p.Create(fx.ctx, fx.req(testName))
|
||||
if tc.wantAdopt {
|
||||
if err != nil || id != formatProviderID(testHost, testName) {
|
||||
t.Fatalf("Create = %q, %v; want the existing domain adopted", id, err)
|
||||
}
|
||||
} else {
|
||||
if id != "" {
|
||||
t.Errorf("Create = %q, want no ID", id)
|
||||
}
|
||||
assertProviderError(t, err, "create", provider.ErrPermanent)
|
||||
if !strings.Contains(err.Error(), "carries no kvm2 mark") {
|
||||
t.Errorf("error = %v, want the missing mark named", err)
|
||||
}
|
||||
}
|
||||
for _, rpc := range []string{"DomainDefineXML", "DomainCreate", "StorageVolCreateXML", "DomainDestroy"} {
|
||||
if got := fx.fake.callCount(rpc); got != 0 {
|
||||
t.Errorf("%s calls = %d, want 0", rpc, got)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -26,7 +26,7 @@ func (p *Provider) Delete(ctx context.Context, providerID string) error {
|
||||
log := p.logger(ctx).WithValues("host", hostName, "name", name)
|
||||
host, err := p.host(hostName)
|
||||
if err != nil {
|
||||
log.Info("host is not configured; delete has nothing to clean up", "id", providerID)
|
||||
log.V(1).Info("host is not configured; delete has nothing to clean up", "id", providerID)
|
||||
return nil
|
||||
}
|
||||
unlock, err := p.lockName(ctx, name)
|
||||
@@ -82,7 +82,7 @@ func removeDomain(ctx context.Context, log logr.Logger, be backend, name string)
|
||||
func removeVolumes(ctx context.Context, log logr.Logger, be backend, poolName, name string) error {
|
||||
pool, err := be.StoragePoolLookupByName(ctx, poolName)
|
||||
if hasCode(err, libvirt.ErrNoStoragePool) {
|
||||
log.Info("pool not found, no volumes to delete", "pool", poolName)
|
||||
log.V(1).Info("pool not found, no volumes to delete", "pool", poolName)
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
|
||||
@@ -128,7 +128,7 @@ func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
|
||||
})
|
||||
t.Run("pool gone", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
fx := newFixture(t, 0)
|
||||
fx := newFixture(t, 1)
|
||||
fx.p.cfg.Hosts[0].Pool = "retired"
|
||||
fx.fake.addDomain(testName, libvirt.DomainShutoff, managedMetadata(testUID, time.Now()))
|
||||
if err := fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)); err != nil {
|
||||
@@ -138,7 +138,7 @@ func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
|
||||
t.Error("domain still defined after Delete")
|
||||
}
|
||||
if got := fx.logs.matching("pool not found, no volumes to delete"); len(got) != 1 || !strings.Contains(got[0], `"pool"="retired"`) {
|
||||
t.Errorf("'pool not found' Info line = %v", got)
|
||||
t.Errorf("'pool not found' V(1) line = %v", got)
|
||||
}
|
||||
})
|
||||
t.Run("nothing at all", func(t *testing.T) {
|
||||
@@ -152,7 +152,7 @@ func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
|
||||
|
||||
func TestDelete_unconfiguredHostIsNoOp(t *testing.T) {
|
||||
t.Parallel()
|
||||
fx := newFixture(t, 0)
|
||||
fx := newFixture(t, 1)
|
||||
id := formatProviderID("nas9", testName)
|
||||
if err := fx.p.Delete(fx.ctx, id); err != nil {
|
||||
t.Fatalf("Delete() error = %v, want nil", err)
|
||||
@@ -164,7 +164,21 @@ func TestDelete_unconfiguredHostIsNoOp(t *testing.T) {
|
||||
t.Errorf("backend calls = %v, want none", got)
|
||||
}
|
||||
if got := fx.logs.matching("host is not configured; delete has nothing to clean up"); len(got) != 1 || !strings.Contains(got[0], `"host"="nas9"`) {
|
||||
t.Errorf("Info line = %v, want one naming the host", got)
|
||||
t.Errorf("V(1) line = %v, want one naming the host", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDelete_noOpLinesAreSilentAtV0(t *testing.T) {
|
||||
t.Parallel()
|
||||
fx := newFixture(t, 0)
|
||||
fx.p.cfg.Hosts[0].Pool = "retired"
|
||||
for _, id := range []string{formatProviderID("nas9", testName), formatProviderID(testHost, testName)} {
|
||||
if err := fx.p.Delete(fx.ctx, id); err != nil {
|
||||
t.Fatalf("Delete(%s) error = %v, want nil", id, err)
|
||||
}
|
||||
}
|
||||
if lines := fx.logs.all(); len(lines) != 0 {
|
||||
t.Errorf("Delete logged %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -36,10 +36,13 @@ type fakeBackend struct {
|
||||
// sha256 in fixed chunks instead of holding the bytes, for tests that
|
||||
// push more through the fake than they want in memory.
|
||||
discardUploads bool
|
||||
// startDelayPolls, when set, makes DomainCreate leave the domain shut
|
||||
// off; it turns running on the startDelayPolls-th DomainGetState after
|
||||
// that, as a real host that is still booting QEMU would.
|
||||
// startDelayPolls, when set, makes DomainCreate leave the domain in
|
||||
// nostate; it turns running on the startDelayPolls-th DomainGetState
|
||||
// after that, as a host whose QEMU is still settling would.
|
||||
startDelayPolls int
|
||||
// stateAfterStart, when not nostate, is what DomainCreate leaves the
|
||||
// domain in instead of running, as a guest that dies at once would.
|
||||
stateAfterStart libvirt.DomainState
|
||||
// undefineFlags records the flags of the last DomainUndefineFlags per
|
||||
// domain name.
|
||||
undefineFlags map[string]libvirt.DomainUndefineFlagsValues
|
||||
@@ -291,9 +294,14 @@ func (f *fakeBackend) DomainCreate(ctx context.Context, dom libvirt.Domain) erro
|
||||
if isActive(d.state) {
|
||||
return fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is already running")
|
||||
}
|
||||
if f.startDelayPolls > 0 {
|
||||
switch {
|
||||
case f.startDelayPolls > 0:
|
||||
d.state = libvirt.DomainNostate
|
||||
d.pendingStart = f.startDelayPolls
|
||||
return nil
|
||||
case f.stateAfterStart != libvirt.DomainNostate:
|
||||
d.state = f.stateAfterStart
|
||||
return nil
|
||||
}
|
||||
d.state = libvirt.DomainRunning
|
||||
d.id = int32(len(f.domains))
|
||||
|
||||
@@ -42,6 +42,8 @@ func (p *Provider) Get(ctx context.Context, providerID string) (*provider.Instan
|
||||
return inst, nil
|
||||
}
|
||||
|
||||
// get builds the Instance from the domain's metadata, its state and, for an
|
||||
// active domain, its discovered address.
|
||||
func get(ctx context.Context, log logr.Logger, be backend, hostName, name string) (*provider.Instance, error) {
|
||||
dom, err := be.DomainLookupByName(ctx, name)
|
||||
if err != nil {
|
||||
@@ -51,11 +53,24 @@ func get(ctx context.Context, log logr.Logger, be backend, hostName, name string
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
inst, err := inspect(ctx, log, be, hostName, dom, meta)
|
||||
state, _, err := be.DomainGetState(ctx, dom, 0)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
|
||||
}
|
||||
s := libvirt.DomainState(state)
|
||||
var ip string
|
||||
if isActive(s) {
|
||||
if ip, err = address(ctx, log, be, dom); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &inst, nil
|
||||
}
|
||||
return &provider.Instance{
|
||||
ID: formatProviderID(hostName, dom.Name),
|
||||
IP: ip,
|
||||
State: mapState(s, ip),
|
||||
UID: meta.UID,
|
||||
CreatedAt: meta.createdAt(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// readMetadata returns the kvm2 mark of a domain, or the zero value (not
|
||||
@@ -72,29 +87,6 @@ func readMetadata(ctx context.Context, be backend, dom libvirt.Domain) (instance
|
||||
return parseInstanceMetadata(raw)
|
||||
}
|
||||
|
||||
// inspect builds the Instance from the backend's state and, for an active
|
||||
// domain, its discovered address.
|
||||
func inspect(ctx context.Context, log logr.Logger, be backend, hostName string, dom libvirt.Domain, meta instanceMetadata) (provider.Instance, error) {
|
||||
state, _, err := be.DomainGetState(ctx, dom, 0)
|
||||
if err != nil {
|
||||
return provider.Instance{}, fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
|
||||
}
|
||||
s := libvirt.DomainState(state)
|
||||
var ip string
|
||||
if isActive(s) {
|
||||
if ip, err = address(ctx, log, be, dom); err != nil {
|
||||
return provider.Instance{}, err
|
||||
}
|
||||
}
|
||||
return provider.Instance{
|
||||
ID: formatProviderID(hostName, dom.Name),
|
||||
IP: ip,
|
||||
State: mapState(s, ip),
|
||||
UID: meta.UID,
|
||||
CreatedAt: meta.createdAt(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// addressSources is the discovery order of spec entry 7: the guest agent
|
||||
// answers where the host's ARP table never sees guest traffic.
|
||||
var addressSources = []struct {
|
||||
|
||||
@@ -2,7 +2,6 @@ package kvm2
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -93,26 +92,6 @@ func (t *hostTable) dialFresh(ctx context.Context, host provider.KVMHost) (backe
|
||||
return t.dialer.dial(ctx, host)
|
||||
}
|
||||
|
||||
// closeAll closes every pooled connection, waiting for in-flight dials.
|
||||
func (t *hostTable) closeAll() error {
|
||||
t.mu.Lock()
|
||||
entries := make([]*hostEntry, 0, len(t.hosts))
|
||||
for _, e := range t.hosts {
|
||||
entries = append(entries, e)
|
||||
}
|
||||
t.mu.Unlock()
|
||||
var errs []error
|
||||
for _, e := range entries {
|
||||
e.sem <- struct{}{}
|
||||
if e.backend != nil {
|
||||
errs = append(errs, e.backend.Close())
|
||||
e.backend = nil
|
||||
}
|
||||
<-e.sem
|
||||
}
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
// libvirtDialer is the production dialer: SSH transport, go-libvirt over
|
||||
// the forwarded socket, the connect handshake, then the observe decorator,
|
||||
// so provider code never holds an undecorated backend.
|
||||
|
||||
@@ -176,23 +176,6 @@ func TestHosts_dialErrorIsNotCached(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestHosts_closeAll(t *testing.T) {
|
||||
t.Parallel()
|
||||
table := newHostTable(&stubDialer{fresh: func() backend { return newFakeBackend() }})
|
||||
a, _ := table.get(context.Background(), hostA)
|
||||
b, _ := table.get(context.Background(), hostB)
|
||||
if err := table.closeAll(); err != nil {
|
||||
t.Fatalf("closeAll error = %v", err)
|
||||
}
|
||||
if a.(*fakeBackend).closed != 1 || b.(*fakeBackend).closed != 1 {
|
||||
t.Errorf("closed counts = %d, %d, want 1, 1", a.(*fakeBackend).closed, b.(*fakeBackend).closed)
|
||||
}
|
||||
again, _ := table.get(context.Background(), hostA)
|
||||
if again == a {
|
||||
t.Error("get after closeAll returned the closed backend")
|
||||
}
|
||||
}
|
||||
|
||||
// perHostDialer routes each host to its own stub.
|
||||
type perHostDialer map[string]*stubDialer
|
||||
|
||||
|
||||
@@ -66,24 +66,15 @@ func resolveImage(cfg provider.KVMConfig, image string) (imageSource, error) {
|
||||
}
|
||||
|
||||
// lockImage takes the per-(host, volume) lock of spec entry 19, or gives up
|
||||
// with ctx. The lock is a one-slot channel rather than a mutex so a caller
|
||||
// whose ctx ends while another caller's download runs returns promptly
|
||||
// (contract §1.5) instead of waiting out a 20-minute transfer.
|
||||
// with ctx: a caller whose ctx ends while another caller's download runs
|
||||
// returns promptly (contract §1.5) instead of waiting out a 20-minute
|
||||
// transfer.
|
||||
func (p *Provider) lockImage(ctx context.Context, hostName, volume string) (func(), error) {
|
||||
key := hostName + "/" + volume
|
||||
p.imageMu.Lock()
|
||||
sem, ok := p.imageLocks[key]
|
||||
if !ok {
|
||||
sem = make(chan struct{}, 1)
|
||||
p.imageLocks[key] = sem
|
||||
}
|
||||
p.imageMu.Unlock()
|
||||
select {
|
||||
case sem <- struct{}{}:
|
||||
return func() { <-sem }, nil
|
||||
case <-ctx.Done():
|
||||
return nil, fmt.Errorf("%w: waiting for image %s on host %s: %w", provider.ErrTransient, volume, hostName, ctx.Err())
|
||||
release, err := p.imageLocks.acquire(ctx, hostName+"/"+volume)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: waiting for image %s on host %s: %w", provider.ErrTransient, volume, hostName, err)
|
||||
}
|
||||
return release, nil
|
||||
}
|
||||
|
||||
// ensureImage makes the base image present and complete in the host's pool
|
||||
@@ -208,7 +199,7 @@ func deleteVolumeIfPresent(ctx context.Context, be backend, pool libvirt.Storage
|
||||
// finding no marker, and removing it now frees the space at once.
|
||||
func (p *Provider) fetchImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, be backend, src imageSource) (libvirt.StorageVol, error) {
|
||||
name := src.volumeName()
|
||||
ctx, cancel := context.WithTimeout(ctx, p.budgets.transfer)
|
||||
ctx, cancel := context.WithTimeout(ctx, defaultBudgets.transfer)
|
||||
defer cancel()
|
||||
ctx, span := tracing.StartSpan(ctx, tracing.Tracer(p.tracing...), "kvm2.download",
|
||||
trace.WithSpanKind(trace.SpanKindClient),
|
||||
@@ -337,7 +328,7 @@ func (c *countingReader) state() (int64, error) {
|
||||
// budget bounds it on its own. Failure is logged, not returned: the next
|
||||
// attempt finds the volume without a marker and deletes it then.
|
||||
func (p *Provider) discardVolume(ctx context.Context, log logr.Logger, be backend, pool libvirt.StoragePool, name string) {
|
||||
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), p.budgets.rpc)
|
||||
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), defaultBudgets.rpc)
|
||||
defer cancel()
|
||||
if err := deleteVolumeIfPresent(ctx, be, pool, name); err != nil {
|
||||
log.V(1).Info("incomplete base volume left behind; the next attempt removes it", "error", err.Error())
|
||||
@@ -421,8 +412,10 @@ func urlHost(rawURL string) string {
|
||||
// imageSizeIfUncached returns the image's Content-Length when the base is
|
||||
// not cached on the host, and 0 when the marker is present, for the quota
|
||||
// precheck of spec entry 13. It never downloads: HEAD first, and a GET whose
|
||||
// body is closed at once when the server does not allow HEAD. A marker is
|
||||
// taken at face value here; ensureImage decides whether it still matches.
|
||||
// body is closed at once when HEAD answers anything but 2xx, since servers
|
||||
// refuse HEAD with 405, 403 or 501 alike; the GET's status is what gets
|
||||
// classified. A marker is taken at face value here; ensureImage decides
|
||||
// whether it still matches.
|
||||
func (p *Provider) imageSizeIfUncached(ctx context.Context, be backend, pool libvirt.StoragePool, src imageSource) (int64, error) {
|
||||
_, err := be.StorageVolLookupByName(ctx, pool, src.markerName())
|
||||
switch {
|
||||
@@ -431,14 +424,14 @@ func (p *Provider) imageSizeIfUncached(ctx context.Context, be backend, pool lib
|
||||
case !hasCode(err, libvirt.ErrNoStorageVol):
|
||||
return 0, fmt.Errorf("looking up marker %s: %w", src.markerName(), err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, p.budgets.rpc)
|
||||
ctx, cancel := context.WithTimeout(ctx, defaultBudgets.rpc)
|
||||
defer cancel()
|
||||
resp, err := p.do(ctx, http.MethodHead, src.URL)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode == http.StatusMethodNotAllowed {
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
if resp, err = p.do(ctx, http.MethodGet, src.URL); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
@@ -152,6 +152,17 @@ func serveStatus(code int) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, _ *http.Request) { http.Error(w, http.StatusText(code), code) }
|
||||
}
|
||||
|
||||
// headFails answers HEAD with code and hands every other method to next.
|
||||
func headFails(code int, next http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method == http.MethodHead {
|
||||
http.Error(w, http.StatusText(code), code)
|
||||
return
|
||||
}
|
||||
next(w, r)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveImage_table(t *testing.T) {
|
||||
t.Parallel()
|
||||
cfg := withDefaults(provider.KVMConfig{Images: map[string]provider.KVMImage{
|
||||
@@ -884,16 +895,29 @@ func TestImageSizeIfUncached_headThenGetFallback(t *testing.T) {
|
||||
},
|
||||
{
|
||||
name: "HEAD not allowed falls back to GET with the body closed",
|
||||
handler: func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method == http.MethodHead {
|
||||
http.Error(w, "nope", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
serveBytes(body)(w, r)
|
||||
},
|
||||
handler: headFails(http.StatusMethodNotAllowed, serveBytes(body)),
|
||||
wantSize: int64(len(body)),
|
||||
wantMethods: []string{http.MethodHead, http.MethodGet},
|
||||
},
|
||||
{
|
||||
name: "HEAD forbidden falls back to GET",
|
||||
handler: headFails(http.StatusForbidden, serveBytes(body)),
|
||||
wantSize: int64(len(body)),
|
||||
wantMethods: []string{http.MethodHead, http.MethodGet},
|
||||
},
|
||||
{
|
||||
name: "HEAD not implemented falls back to GET",
|
||||
handler: headFails(http.StatusNotImplemented, serveBytes(body)),
|
||||
wantSize: int64(len(body)),
|
||||
wantMethods: []string{http.MethodHead, http.MethodGet},
|
||||
},
|
||||
{
|
||||
name: "HEAD refused and GET 404 is permanent from the GET",
|
||||
handler: headFails(http.StatusMethodNotAllowed, serveStatus(http.StatusNotFound)),
|
||||
wantMethods: []string{http.MethodHead, http.MethodGet},
|
||||
wantClass: provider.ErrPermanent,
|
||||
wantErr: "404",
|
||||
},
|
||||
{
|
||||
name: "marker present means nothing to download",
|
||||
handler: serveBytes(body),
|
||||
@@ -902,16 +926,16 @@ func TestImageSizeIfUncached_headThenGetFallback(t *testing.T) {
|
||||
wantMethods: nil,
|
||||
},
|
||||
{
|
||||
name: "404 is permanent",
|
||||
name: "404 on both is permanent",
|
||||
handler: serveStatus(http.StatusNotFound),
|
||||
wantMethods: []string{http.MethodHead},
|
||||
wantMethods: []string{http.MethodHead, http.MethodGet},
|
||||
wantClass: provider.ErrPermanent,
|
||||
wantErr: "404",
|
||||
},
|
||||
{
|
||||
name: "503 is transient",
|
||||
name: "503 on both is transient",
|
||||
handler: serveStatus(http.StatusServiceUnavailable),
|
||||
wantMethods: []string{http.MethodHead},
|
||||
wantMethods: []string{http.MethodHead, http.MethodGet},
|
||||
wantClass: provider.ErrTransient,
|
||||
},
|
||||
{
|
||||
|
||||
@@ -33,7 +33,6 @@ type Provider struct {
|
||||
now func() time.Time
|
||||
hosts *hostTable
|
||||
http *http.Client
|
||||
budgets budgets
|
||||
tracing []tracing.Option
|
||||
|
||||
// runningWait caps the wait for a started domain to report running,
|
||||
@@ -42,28 +41,23 @@ type Provider struct {
|
||||
runningPoll time.Duration
|
||||
|
||||
// imageLocks serialises image fetches per (host, volume), spec entry 19.
|
||||
imageMu sync.Mutex
|
||||
imageLocks map[string]chan struct{}
|
||||
imageLocks *keyedLocks
|
||||
|
||||
// nameLocks serialises Create and Delete per instance name, spec entry
|
||||
// 19, so two Deletes never race on destroy and undefine and a Delete
|
||||
// cannot interleave with a Create's steps.
|
||||
nameMu sync.Mutex
|
||||
nameLocks map[string]chan struct{}
|
||||
nameLocks *keyedLocks
|
||||
}
|
||||
|
||||
// providerOption adjusts what newProvider builds; tests inject an HTTP
|
||||
// client bound to a test server, a span recorder and shorter budgets.
|
||||
// client bound to a test server, a span recorder and a shorter running
|
||||
// wait.
|
||||
type providerOption func(*Provider)
|
||||
|
||||
func withHTTPClient(c *http.Client) providerOption {
|
||||
return func(p *Provider) { p.http = c }
|
||||
}
|
||||
|
||||
func withBudgets(b budgets) providerOption {
|
||||
return func(p *Provider) { p.budgets = b }
|
||||
}
|
||||
|
||||
func withTracing(opts ...tracing.Option) providerOption {
|
||||
return func(p *Provider) { p.tracing = opts }
|
||||
}
|
||||
@@ -104,7 +98,7 @@ func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, er
|
||||
// newProvider is the constructor tests use: d hands out the backends, the
|
||||
// production one being newLibvirtDialer. Without options the image
|
||||
// downloads use http.DefaultClient (system CA roots, no client timeout: the
|
||||
// per-request ctx carries the transfer budget) and the default budgets.
|
||||
// per-request ctx carries the transfer budget).
|
||||
func newProvider(name string, cfg provider.KVMConfig, d dialer, opts ...providerOption) *Provider {
|
||||
p := &Provider{
|
||||
name: name,
|
||||
@@ -112,11 +106,10 @@ func newProvider(name string, cfg provider.KVMConfig, d dialer, opts ...provider
|
||||
now: time.Now,
|
||||
hosts: newHostTable(d),
|
||||
http: http.DefaultClient,
|
||||
budgets: defaultBudgets,
|
||||
runningWait: defaultRunningWait,
|
||||
runningPoll: defaultRunningPoll,
|
||||
imageLocks: make(map[string]chan struct{}),
|
||||
nameLocks: make(map[string]chan struct{}),
|
||||
imageLocks: newKeyedLocks(),
|
||||
nameLocks: newKeyedLocks(),
|
||||
}
|
||||
for _, opt := range opts {
|
||||
opt(p)
|
||||
@@ -187,21 +180,64 @@ func (p *Provider) wrapErr(op, id string, err error) error {
|
||||
return provider.Wrap(classify(op, err), op, p.name, id, err)
|
||||
}
|
||||
|
||||
// lockName takes the per-name lock of spec entry 19, or gives up with ctx.
|
||||
// A one-slot channel rather than a mutex, so a caller whose ctx ends while
|
||||
// another caller's Create is downloading an image returns promptly.
|
||||
// lockName takes the per-name lock of spec entry 19, or gives up with ctx,
|
||||
// so a caller whose ctx ends while another caller's Create is downloading
|
||||
// an image returns promptly.
|
||||
func (p *Provider) lockName(ctx context.Context, name string) (func(), error) {
|
||||
p.nameMu.Lock()
|
||||
sem, ok := p.nameLocks[name]
|
||||
if !ok {
|
||||
sem = make(chan struct{}, 1)
|
||||
p.nameLocks[name] = sem
|
||||
release, err := p.nameLocks.acquire(ctx, name)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, err)
|
||||
}
|
||||
p.nameMu.Unlock()
|
||||
return release, nil
|
||||
}
|
||||
|
||||
// keyedLocks is a set of one-slot semaphores by key (spec entry 19): a
|
||||
// semaphore rather than a mutex so a waiter can give up with its ctx. Each
|
||||
// entry counts the callers holding or waiting for it and is dropped from
|
||||
// the map at zero, so the map does not grow with every name ever seen while
|
||||
// two waiters on one key still always share one semaphore.
|
||||
type keyedLocks struct {
|
||||
mu sync.Mutex
|
||||
locks map[string]*keyedLock
|
||||
}
|
||||
|
||||
type keyedLock struct {
|
||||
sem chan struct{}
|
||||
refs int
|
||||
}
|
||||
|
||||
func newKeyedLocks() *keyedLocks {
|
||||
return &keyedLocks{locks: make(map[string]*keyedLock)}
|
||||
}
|
||||
|
||||
// acquire takes the lock for key, or gives up with ctx's error. The
|
||||
// returned func releases it and must be called exactly once.
|
||||
func (k *keyedLocks) acquire(ctx context.Context, key string) (func(), error) {
|
||||
k.mu.Lock()
|
||||
l, ok := k.locks[key]
|
||||
if !ok {
|
||||
l = &keyedLock{sem: make(chan struct{}, 1)}
|
||||
k.locks[key] = l
|
||||
}
|
||||
l.refs++
|
||||
k.mu.Unlock()
|
||||
select {
|
||||
case sem <- struct{}{}:
|
||||
return func() { <-sem }, nil
|
||||
case l.sem <- struct{}{}:
|
||||
return func() {
|
||||
<-l.sem
|
||||
k.unref(key, l)
|
||||
}, nil
|
||||
case <-ctx.Done():
|
||||
return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, ctx.Err())
|
||||
k.unref(key, l)
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func (k *keyedLocks) unref(key string, l *keyedLock) {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
l.refs--
|
||||
if l.refs == 0 {
|
||||
delete(k.locks, key)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"errors"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -254,6 +255,86 @@ func TestWithDefaults_doesNotAliasInput(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func lockCount(k *keyedLocks) int {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return len(k.locks)
|
||||
}
|
||||
|
||||
func TestKeyedLocks_entryRemovedWhenUncontended(t *testing.T) {
|
||||
t.Parallel()
|
||||
k := newKeyedLocks()
|
||||
release, err := k.acquire(context.Background(), "a")
|
||||
if err != nil {
|
||||
t.Fatalf("acquire error = %v", err)
|
||||
}
|
||||
if lockCount(k) != 1 {
|
||||
t.Errorf("entries while held = %d, want 1", lockCount(k))
|
||||
}
|
||||
release()
|
||||
if lockCount(k) != 0 {
|
||||
t.Errorf("entries after release = %d, want 0", lockCount(k))
|
||||
}
|
||||
if _, err := k.acquire(context.Background(), "a"); err != nil {
|
||||
t.Fatalf("acquire after release error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKeyedLocks_waitersShareOneLock(t *testing.T) {
|
||||
t.Parallel()
|
||||
k := newKeyedLocks()
|
||||
const n = 20
|
||||
var inside, overlaps atomic.Int32
|
||||
var wg sync.WaitGroup
|
||||
for range n {
|
||||
wg.Go(func() {
|
||||
release, err := k.acquire(context.Background(), "shared")
|
||||
if err != nil {
|
||||
t.Errorf("acquire error = %v", err)
|
||||
return
|
||||
}
|
||||
if inside.Add(1) > 1 {
|
||||
overlaps.Add(1)
|
||||
}
|
||||
time.Sleep(time.Millisecond)
|
||||
inside.Add(-1)
|
||||
release()
|
||||
})
|
||||
}
|
||||
wg.Wait()
|
||||
if overlaps.Load() != 0 {
|
||||
t.Errorf("%d holders overlapped on one key", overlaps.Load())
|
||||
}
|
||||
if lockCount(k) != 0 {
|
||||
t.Errorf("entries after every holder released = %d, want 0", lockCount(k))
|
||||
}
|
||||
}
|
||||
|
||||
func TestKeyedLocks_waiterGivesUpWithContext(t *testing.T) {
|
||||
t.Parallel()
|
||||
k := newKeyedLocks()
|
||||
release, err := k.acquire(context.Background(), "a")
|
||||
if err != nil {
|
||||
t.Fatalf("acquire error = %v", err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
|
||||
defer cancel()
|
||||
start := time.Now()
|
||||
if _, err := k.acquire(ctx, "a"); !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("waiter error = %v, want DeadlineExceeded", err)
|
||||
}
|
||||
if time.Since(start) > time.Second {
|
||||
t.Errorf("waiter took %v, want to give up with its ctx", time.Since(start))
|
||||
}
|
||||
if lockCount(k) != 1 {
|
||||
t.Errorf("entries with the holder still in = %d, want 1: the waiter's exit must not drop the holder's entry", lockCount(k))
|
||||
}
|
||||
release()
|
||||
if lockCount(k) != 0 {
|
||||
t.Errorf("entries after the holder released = %d, want 0", lockCount(k))
|
||||
}
|
||||
}
|
||||
|
||||
func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) {
|
||||
t.Parallel()
|
||||
p := newProvider("kvm2-home", twoHosts(), &stubDialer{})
|
||||
|
||||
@@ -10,10 +10,13 @@ import (
|
||||
)
|
||||
|
||||
// ListByTag is spec entry 12: every domain on every configured host, in
|
||||
// config order, filtered client-side on the kvm2 mark. A host that cannot
|
||||
// be reached or listed fails the whole call (fail-whole); a single domain
|
||||
// that cannot be read is skipped, unless ctx ended, which fails the call
|
||||
// rather than pass off an empty view as the truth.
|
||||
// config order, filtered client-side on the kvm2 mark. GC uses only ID, UID
|
||||
// and CreatedAt (contract §2.4), all of which the name and the metadata
|
||||
// element give, so no state or address RPC is made and State and IP stay
|
||||
// empty. A host that cannot be reached or listed fails the whole call
|
||||
// (fail-whole); a single domain whose metadata cannot be read is skipped,
|
||||
// unless ctx ended, which fails the call rather than pass off an empty view
|
||||
// as the truth.
|
||||
func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
|
||||
log := p.logger(ctx)
|
||||
var out []provider.Instance
|
||||
@@ -43,15 +46,8 @@ func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
|
||||
if !meta.managed() {
|
||||
continue
|
||||
}
|
||||
inst, err := inspect(ctx, dlog, be, host.Name, dom, meta)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil, p.wrapErr("list", "", err)
|
||||
}
|
||||
dlog.V(1).Info("skipping domain that could not be inspected", "error", err.Error())
|
||||
continue
|
||||
}
|
||||
dlog.V(2).Info("kvm2 listed instance", "id", inst.ID, "state", inst.State, "uid", inst.UID, "createdAt", inst.CreatedAt)
|
||||
inst := provider.Instance{ID: formatProviderID(host.Name, dom.Name), UID: meta.UID, CreatedAt: meta.createdAt()}
|
||||
dlog.V(2).Info("kvm2 listed instance", "id", inst.ID, "uid", inst.UID, "createdAt", inst.CreatedAt)
|
||||
out = append(out, inst)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -72,8 +72,43 @@ func TestListByTag_acrossHostsInConfigOrder(t *testing.T) {
|
||||
if got := listIDs(list); strings.Join(got, ",") != strings.Join(want, ",") {
|
||||
t.Errorf("ListByTag = %v, want %v (config order, then name order)", got, want)
|
||||
}
|
||||
if list[0].State != provider.StateStopped || list[1].State != provider.StateProvisioning {
|
||||
t.Errorf("states = %s, %s, want Stopped, Provisioning", list[0].State, list[1].State)
|
||||
if list[0].UID != "uid-a" || list[1].UID != "uid-z" || list[2].UID != "uid-b" {
|
||||
t.Errorf("uids = %s, %s, %s, want uid-a, uid-z, uid-b", list[0].UID, list[1].UID, list[2].UID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListByTag_doesNotReadStateOrAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
fx := newFixture(t, 2)
|
||||
id := fx.create(testName)
|
||||
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcAgent, "10.0.0.5")
|
||||
fx.fake.addDomain("proxy-off", libvirt.DomainShutoff, managedMetadata("uid-off", time.Now()))
|
||||
before := len(fx.fake.callLog())
|
||||
|
||||
list, err := fx.p.ListByTag(fx.ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("ListByTag() error = %v", err)
|
||||
}
|
||||
if len(list) != 2 || list[0].ID != id || list[1].ID != formatProviderID(testHost, "proxy-off") {
|
||||
t.Fatalf("ListByTag = %v, want the two managed domains", listIDs(list))
|
||||
}
|
||||
for _, inst := range list {
|
||||
if inst.State != "" || inst.IP != "" {
|
||||
t.Errorf("listed %s = state %q ip %q, want both empty: GC does not use them", inst.ID, inst.State, inst.IP)
|
||||
}
|
||||
if inst.UID == "" || inst.CreatedAt.IsZero() {
|
||||
t.Errorf("listed %s lacks uid or createdAt: %+v", inst.ID, inst)
|
||||
}
|
||||
}
|
||||
for _, rpc := range fx.fake.callLog()[before:] {
|
||||
switch rpc {
|
||||
case "ConnectListAllDomains", "DomainGetMetadata":
|
||||
default:
|
||||
t.Errorf("ListByTag made %s; want only the listing and one metadata read per domain", rpc)
|
||||
}
|
||||
}
|
||||
if got := fx.logs.matching("kvm2 listed instance"); len(got) != 2 || strings.Contains(got[0], `"state"=`) {
|
||||
t.Errorf("listed instance lines = %v, want two without a state", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -126,12 +161,12 @@ func TestListByTag_unreadableDomainSkipped(t *testing.T) {
|
||||
line: "skipping domain with unreadable metadata",
|
||||
},
|
||||
{
|
||||
name: "state rpc fails for the first domain only",
|
||||
name: "domain gone between listing and metadata read",
|
||||
setup: func(fx *fixture) {
|
||||
fx.fake.addDomain("proxy-bad", libvirt.DomainRunning, managedMetadata("uid-bad", time.Now()))
|
||||
var calls int
|
||||
fx.fake.mu.Lock()
|
||||
fx.fake.hooks["DomainGetState"] = func() error {
|
||||
fx.fake.hooks["DomainGetMetadata"] = func() error {
|
||||
calls++
|
||||
if calls == 1 {
|
||||
return fakeErr(libvirt.ErrNoDomain, "Domain not found")
|
||||
@@ -140,7 +175,7 @@ func TestListByTag_unreadableDomainSkipped(t *testing.T) {
|
||||
}
|
||||
fx.fake.mu.Unlock()
|
||||
},
|
||||
line: "skipping domain that could not be inspected",
|
||||
line: "skipping domain with unreadable metadata",
|
||||
},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
@@ -172,7 +207,7 @@ func TestListByTag_includesInactiveDomains(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("ListByTag() error = %v", err)
|
||||
}
|
||||
if len(list) != 1 || list[0].ID != id || list[0].State != provider.StateStopped {
|
||||
t.Errorf("ListByTag = %+v, want the shut-off domain as Stopped", list)
|
||||
if len(list) != 1 || list[0].ID != id || list[0].UID != testUID {
|
||||
t.Errorf("ListByTag = %+v, want the shut-off domain listed with its uid", list)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user