kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

kvm2: name the cached base image after the provider entry (H4)

Browse Source 
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm
provider builds in the same pool -- for URL images too, since kvm takes
hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with
a completely separate lock table. The shipped config has a live kvm-home
entry and a commented-out kvm2-home entry on the same host, the same key
debian-13 and the same default pool, and the comment invites the operator
to enable it. Both integration tests use key test-image. On beelink both
live overlays back onto kvm's base-debian-13.qcow2.

Since cachedImage accepts any 64-hex marker when no sha256 is configured
and never re-hashes the base, that meant two entries pointing one key at
different URLs silently shared a file; a concurrent cold fetch let one
entry delete the other's half-uploaded base and boot a guest from the
remains; and a sha256 edit deleted the base in place under live overlays,
which libvirt permits without complaint.

The name now carries the provider entry: base-<entry>-<key>.<format>.
imageSource gained an Entry field and resolveImage takes the provider
name, so every production call site follows from imageSource.volumeName().
That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2
alike, and turns the per-Provider imageLocks from a latent bug into a
correct one: with names unique per entry, two entries cannot contend for a
file, so no process-wide lock table is needed.

The entry name now reaches a file name, so New checks it against the
existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather
than duplicated. Provider names are otherwise validated for all types at
once and only for being non-empty and unique, so a name with a slash would
have produced a volume nothing could address.

One companion fix, because the naming alone does not stop it:
cachedImage's digest-mismatch branch deleted the marker and the base and
refetched. It now returns ErrPermanent naming both digests and telling the
operator what to do. That branch was the remaining path that unlinks a
good file under running domains, and the contract already said "add a new
key instead" -- the code was contradicting it. The no-marker and
non-hex-marker branches keep deleting, since there the base is debris.

Not fixed, and said so in the docs rather than left to look closed:
editing images.<key>.url is still a silent no-op, since nothing records
the URL; and cross-process fencing still needs the deferred atomic publish
(upload to a temp name, verify, publish), which lands where H2 put the
discard.

Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's,
kvm keeps using it untouched, and kvm2 downloads its own copy once per
host. Nothing is replaced under a running guest; the pool needs room for
both, which the prerequisites now say.

The eviction recipe in the guide named base-debian-13.qcow2, which after
this change is kvm's file -- following it would have deleted the wrong
provider's base. It now greps for the prefix and deletes the marker first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-17 17:57:02 +00:00
co-authored by Claude Opus 5
parent 9682737c69
commit c10f98a352
17 changed files with 258 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/kvm2-provider.md
+40 -12
View File
@@ -37,7 +37,8 @@ Per host, once:
API. Nothing on the host filters the port; the provider does not use
`spec.port` at all.
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
with room for one base image per distro (a few hundred MB) plus the
with room for one base image per distro and provider entry (a few
hundred MB each) plus the
per-VM overlays (`diskSizeGb` each, thin-provisioned). The pool must be
running: a stopped pool fails the health check with `PoolNotActive` and
fails `Create` permanently.
@@ -202,13 +203,22 @@ host's pool:
| Object | Name | Lifetime |
|---|---|---|
| base image | `base-<key>.<format>` (for a URL image, `<key>` is the first 16 hex characters of the URL's sha256) | cached; survives Proxy deletion |
| image marker | `base-<key>.<format>.sha256`, a 64-byte raw volume holding the digest | with the base image |
| base image | `base-<entry>-<key>.<format>`, where `<entry>` is the provider entry's name (for a URL image, `<key>` is the first 16 hex characters of the URL's sha256) | cached; survives Proxy deletion |
| image marker | `base-<entry>-<key>.<format>.sha256`, a 64-byte raw volume holding the digest | with the base image |
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base file, `diskSizeGb` capacity | with the Proxy |
| seed ISO | `proxy-abc…-seed.iso`, raw | with the Proxy |
| domain | `proxy-abc…` | with the Proxy |
The names are built in `internal/provider/kvm2/volume.go`.
The names are built in `internal/provider/kvm2/volume.go`. The base image
carries the **provider entry's name** because the file is shared by every
Proxy on that host but must not be shared by two entries: the `kvm`
provider builds `base-<key>.<format>` in the same pool from the same
catalogue shape, and two kvm2 entries pointing one key at different URLs
would otherwise agree on the name and disagree on the contents. The cost
is one copy of each image per entry per host. For that reason the entry's
name must itself be a usable file name — letters, digits, dots, dashes and
underscores — which `New` checks at startup, since provider names are
otherwise only required to be non-empty and unique.
The domain (`buildDomainXML` in `xml.go`) is a `q35` machine with a
host-passthrough CPU, ACPI and APIC, memory in MiB and vcpus from the
@@ -318,16 +328,27 @@ the rest find the marker.
To evict manually:
```sh
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2
ssh novakj@beelink virsh -c qemu:///system vol-list --pool default | grep base-kvm2-home-
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-kvm2-home-debian-13.qcow2.sha256
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-kvm2-home-debian-13.qcow2
```
**Warning: do not change an image entry's `url` or `sha256`, and do not
evict its base volume, while Proxies using that image exist on the
host.** The overlays back onto the base file; deleting or replacing it
corrupts every VM that uses it. Delete those Proxies first, or add the
new image under a new key and move Proxies over by changing their
`placement.image` (which replaces the VM).
Marker first: a base without its marker is refetched, a marker without its
base is cleaned up, and neither leaves a Proxy pointing at nothing.
**Warning: do not evict a base volume while Proxies using that image exist
on the host.** The overlays back onto the base file, and libvirt will
unlink it under running domains without complaint; the guests keep running
on the open file descriptor and fail at their next start. Delete those
Proxies first.
Changing `sha256` on an entry whose base is already cached is **refused**,
permanently and by name, rather than resolved by replacing the file — that
replacement was the same corruption, just automated. Add the image under a
new key and move Proxies over by changing their `placement.image` (which
replaces the VM), or delete the Proxies and evict the base by hand.
Changing `url` alone does nothing at all: the cache is keyed by entry and
key, so the new URL is never fetched. Add a new key for that too.
## Quota: the pool-space precheck
@@ -381,6 +402,13 @@ A `kvm` domain left behind on a host is invisible to kvm2 forever; clean
it up on the host with `virsh undefine --nvram <name>` and
`virsh vol-delete` for its `<name>.qcow2` and `<name>-seed.iso`.
The base image cache is **not** shared between the two types, and that is
deliberate: kvm names its copy `base-<key>.<format>` while kvm2 names its
own `base-<entry>-<key>.<format>`, so kvm2 downloads the image once more
per host rather than adopting a file another provider maintains. Nothing
needs deleting; the `kvm` base stays where it is and keeps serving the
overlays that back onto it. Budget the extra few hundred MB in the pool.
## Logging
The provider logs through controller-runtime's logger, named `kvm2`, with