kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

kvm2: apply the milestone review

Browse Source 
Shorter start wait with immediate verdicts for dead or paused domains,
ListByTag without state or address RPCs, Timeout for an expired check
round, no adoption of unmarked domains, reference-counted keyed locks,
dead options removed, HEAD fallback on any non-2xx.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-16 21:19:50 +00:00
co-authored by Claude Fable 5.1
parent fcbf8ddc91
commit 2510842396
21 changed files with 586 additions and 254 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/kvm2-provider.md
+27 -20
View File
@@ -235,10 +235,12 @@ Its `<metadata>` carries one element in kvm2's own namespace,
That is how orphan GC recognises its own VMs, since libvirt has no
labels, and how a repeat `Create` knows whether an existing domain of the
same name belongs to this Proxy: a different non-empty `uid` is a
permanent error naming both UIDs; an empty one is adopted. The namespace
is kvm2's own, so the `kvm` provider's domains are invisible to it (see
*Migrating from kvm*).
same name belongs to this Proxy: a domain without the element is a
permanent error (`carries no kvm2 mark; it was not created by this
provider`), a different non-empty `uid` is a permanent error naming both
UIDs, and a marked domain with a matching or empty `uid` is adopted. The
namespace is kvm2's own, so the `kvm` provider's domains are invisible to
it (see *Migrating from kvm*).
The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled
`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
@@ -321,8 +323,10 @@ Before creating any volume, `Create` reads `StoragePoolGetInfo` and
compares the pool's free space with what it is about to commit
(`checkPoolSpace` and `neededBytes` in `image.go`): the image's
`Content-Length` when the base is not cached yet (found with a `HEAD`
request, or a `GET` whose body is closed at once when the server refuses
`HEAD`), plus `diskSizeGb` in bytes for the overlay, plus 16 MiB for the
request, or a `GET` whose body is closed at once when `HEAD` answers
anything but 2xx, since mirrors refuse it with 405, 403 or 501 alike; the
`GET`'s status is then what counts), plus `diskSizeGb` in bytes for the
overlay, plus 16 MiB for the
seed. Below that it returns `ErrQuotaExceeded` with a message naming the
pool, the bytes available and the bytes needed. The reconciler answers
quota with a wait, not a failure: the Proxy shows `Provisioned=False`
@@ -347,9 +351,12 @@ by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The
What does not port is the VMs. kvm2 marks its domains in its own metadata
namespace and neither lists nor adopts domains the `kvm` provider made:
`ListByTag` skips them, so orphan GC never sees them, and a Proxy's
`spec.provider` is immutable, so an existing Proxy cannot be moved to the
new entry anyway. Do it in this order:
`ListByTag` skips them, so orphan GC never sees them, and `Create` refuses
a domain of the same name that lacks the kvm2 mark with a permanent error
rather than start it (a Proxy recreated under the new entry gets the same
deterministic name, so this is what happens when step 1 below is skipped).
A Proxy's `spec.provider` is immutable, so an existing Proxy cannot be
moved to the new entry anyway. Do it in this order:
1. Delete the Proxies on the `kvm` entry and wait for them to go; the
`kvm` provider removes their domains and volumes.
@@ -437,7 +444,7 @@ applied by `call`):
| `DomainCreate` | 90 s |
| `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s |
| `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier |
| wait for running after `DomainCreate` | 2 m, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`) |
| wait for running after `DomainCreate` | 15 s, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`); `DomainCreate` is synchronous, so the first poll normally answers running, and a domain that is shut off, crashed or paused right after the start fails at once rather than waiting out the cap |
| the `HEAD` request of the pool-space precheck | 30 s |
| the courtesy `ConnectClose` when a connection is closed | 5 s |
@@ -499,10 +506,10 @@ A failing target says where the check broke (`checkReason` in `check.go`):
|---|---|---|
| `Reachable` | ready | |
| `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host |
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget while the round still had time, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
| `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) |
| `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group |
| `Timeout` | the connection was up, but the monitor's `--provider-check-timeout` (default 15s) or an RPC's budget ran out before the check finished | a hung `libvirtd` or a saturated link |
| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out before the check finished, whichever stage was running, the dial included; or the connection was up and an RPC's own budget ran out | a hung `libvirtd`, a saturated link, or a round timeout shorter than the dial takes |
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
| `APIError` | any other libvirt error | the message carries libvirt's own text |
@@ -518,10 +525,10 @@ waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
| Class | What lands there |
|---|---|
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
| `ErrQuotaExceeded` | the pool-space precheck |
| `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring |
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring or the domain going shut off or crashed right after its start |
Error strings name paths, hosts, volumes and libvirt's own text. They
never contain key bytes, the known_hosts content or cloud-init.
@@ -556,12 +563,12 @@ never contain key bytes, the known_hosts content or cloud-init.
- **`Content-Length` is required** for image downloads; a mirror that
only chunks is rejected permanently. Point the catalogue at one that
sends the length.
- **A paused or pmsuspended domain cannot be adopted.** A repeat `Create`
for a domain in that state calls `DomainCreate`, which libvirt refuses
as "already running"; the provider then waits for the domain to report
running, the 2-minute wait expires, and the call fails transiently on
every retry. Resume the domain on the host (`virsh resume <name>`) or
delete the Proxy.
- **A paused or pmsuspended domain is not started.** A repeat `Create`
for a domain in that state fails permanently (`is DomainPaused and
cannot be started by the operator; resume or destroy it on the host`),
so the Proxy shows `Failed` until its spec changes; the operator never
resumes a VM an admin paused. Resume it on the host
(`virsh resume <name>`) and touch the spec, or delete the Proxy.
- **Parser text in credential errors.** A key or known_hosts file that
does not parse fails with the parser's own message, which can quote a
PEM header type or a malformed known_hosts token. Key bytes are never