kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

kvm2: name the cached base image after the provider entry (H4)

Browse Source 
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm
provider builds in the same pool -- for URL images too, since kvm takes
hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with
a completely separate lock table. The shipped config has a live kvm-home
entry and a commented-out kvm2-home entry on the same host, the same key
debian-13 and the same default pool, and the comment invites the operator
to enable it. Both integration tests use key test-image. On beelink both
live overlays back onto kvm's base-debian-13.qcow2.

Since cachedImage accepts any 64-hex marker when no sha256 is configured
and never re-hashes the base, that meant two entries pointing one key at
different URLs silently shared a file; a concurrent cold fetch let one
entry delete the other's half-uploaded base and boot a guest from the
remains; and a sha256 edit deleted the base in place under live overlays,
which libvirt permits without complaint.

The name now carries the provider entry: base-<entry>-<key>.<format>.
imageSource gained an Entry field and resolveImage takes the provider
name, so every production call site follows from imageSource.volumeName().
That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2
alike, and turns the per-Provider imageLocks from a latent bug into a
correct one: with names unique per entry, two entries cannot contend for a
file, so no process-wide lock table is needed.

The entry name now reaches a file name, so New checks it against the
existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather
than duplicated. Provider names are otherwise validated for all types at
once and only for being non-empty and unique, so a name with a slash would
have produced a volume nothing could address.

One companion fix, because the naming alone does not stop it:
cachedImage's digest-mismatch branch deleted the marker and the base and
refetched. It now returns ErrPermanent naming both digests and telling the
operator what to do. That branch was the remaining path that unlinks a
good file under running domains, and the contract already said "add a new
key instead" -- the code was contradicting it. The no-marker and
non-hex-marker branches keep deleting, since there the base is debris.

Not fixed, and said so in the docs rather than left to look closed:
editing images.<key>.url is still a silent no-op, since nothing records
the URL; and cross-process fencing still needs the deferred atomic publish
(upload to a temp name, verify, publish), which lands where H2 put the
discard.

Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's,
kvm keeps using it untouched, and kvm2 downloads its own copy once per
host. Nothing is replaced under a running guest; the pool needs room for
both, which the prerequisites now say.

The eviction recipe in the guide named base-debian-13.qcow2, which after
this change is kvm's file -- following it would have deleted the wrong
provider's base. It now greps for the prefix and deletes the marker first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-17 17:57:02 +00:00
co-authored by Claude Opus 5
parent 9682737c69
commit c10f98a352
17 changed files with 258 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/provider/config_kvm.go
+10 -4
View File
@@ -108,11 +108,17 @@ var sha256Hex = regexp.MustCompile(`^[0-9a-fA-F]{64}$`)
// kvmIdentifier constrains the config names that are pasted into
// identifiers elsewhere: a host name goes into the providerID
// hosts/<name>/domains/<vm>, which is split on "/" to parse, and an image
// key becomes the file name of the cached base volume. A slash in either
// produces an ID or a path that never resolves again, so the VM leaks with
// nothing able to address it.
// key becomes part of the file name of the cached base volume. A slash in
// either produces an ID or a path that never resolves again, so the VM
// leaks with nothing able to address it.
var kvmIdentifier = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*$`)
// IsKVMIdentifier reports whether s is safe to paste into a providerID or a
// volume name. The provider entry's own name reaches a volume name too, and
// that name is validated for every provider type at once (only non-empty
// and unique), so the kvm2 provider checks its own at construction.
func IsKVMIdentifier(s string) bool { return kvmIdentifier.MatchString(s) }
func validateKVM(prefix string, c *KVMConfig) error {
return validateKVMBlock(prefix, "kvm", c)
}
@@ -137,7 +143,7 @@ func validateKVMBlock(prefix, block string, c *KVMConfig) error {
p := fmt.Sprintf("%s: %s.images[%q]", prefix, block, name)
if !kvmIdentifier.MatchString(name) {
return fmt.Errorf("%s: key must be letters, digits, dots, dashes or underscores"+
" — it becomes the cached volume name base-<key>.<format>", p)
" — it becomes part of the cached base volume's name", p)
}
if !IsHTTPURL(img.URL) {
return fmt.Errorf("%s.url must be an http(s) URL (got %q)", p, img.URL)
internal/provider/kvm2/create.go
+1 -1
View File
@@ -38,7 +38,7 @@ func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (stri
if !ok {
return "", p.wrapErr("create", "", permanentf("placement.machineType %q is not in kvm2.machineTypes", req.Placement.MachineType))
}
src, err := resolveImage(p.cfg, req.Placement.Image)
src, err := resolveImage(p.name, p.cfg, req.Placement.Image)
if err != nil {
return "", p.wrapErr("create", "", err)
}
internal/provider/kvm2/create_test.go
+3 -3
View File
@@ -61,13 +61,13 @@ func TestCreate_resumesAfterEachPartialStep(t *testing.T) {
{
name: "base without marker is refetched",
setup: func(fx *fixture) {
fx.putVol("base-debian-13.qcow2", []byte("truncated"))
fx.putVol(fx.baseName(), []byte("truncated"))
},
check: func(t *testing.T, fx *fixture) {
if got := fx.fake.vol("base-debian-13.qcow2"); got == nil || !bytes.Equal(got.content, fx.image) {
if got := fx.fake.vol(fx.baseName()); got == nil || !bytes.Equal(got.content, fx.image) {
t.Errorf("base volume not refetched")
}
if fx.fake.vol("base-debian-13.qcow2.sha256") == nil {
if fx.fake.vol(fx.baseName()+".sha256") == nil {
t.Errorf("marker not written")
}
},
internal/provider/kvm2/delete_test.go
+2 -1
View File
@@ -64,7 +64,8 @@ func TestDelete_removesOverlayAndSeed(t *testing.T) {
t.Errorf("volume %s still present after Delete", name)
}
}
for _, name := range []string{"base-debian-13.qcow2", "base-debian-13.qcow2.sha256"} {
base := "base-" + testProvider + "-debian-13.qcow2"
for _, name := range []string{base, base + ".sha256"} {
if fx.fake.vol(name) == nil {
t.Errorf("shared base volume %s removed by Delete", name)
}
internal/provider/kvm2/image.go
+13 -6
View File
@@ -42,22 +42,25 @@ var hexDigest = regexp.MustCompile(`^[0-9a-f]{64}$`)
// catalogue entry, or a URL given directly with the catalogue defaults, plus
// the key its base volume is named after.
type imageSource struct {
Key string
Entry string
Key string
provider.KVMImage
}
func (s imageSource) volumeName() string { return baseVolumeName(s.Key, s.Format) }
func (s imageSource) volumeName() string { return baseVolumeName(s.Entry, s.Key, s.Format) }
func (s imageSource) markerName() string { return markerVolumeName(s.volumeName()) }
// resolveImage maps placement.image to a source. cfg is already defaulted,
// so a catalogue entry carries its format and firmware; a URL gets qcow2
// and uefi.
func resolveImage(cfg provider.KVMConfig, image string) (imageSource, error) {
// and uefi. entry is the provider entry's name, which the base volume is
// named after so two entries never share one file.
func resolveImage(entry string, cfg provider.KVMConfig, image string) (imageSource, error) {
if img, ok := cfg.Images[image]; ok {
return imageSource{Key: image, KVMImage: img}, nil
return imageSource{Entry: entry, Key: image, KVMImage: img}, nil
}
if provider.IsHTTPURL(image) {
return imageSource{
Entry: entry,
Key: urlImageKey(image),
KVMImage: provider.KVMImage{URL: image, Format: provider.KVMFormatQcow2, Firmware: provider.KVMFirmwareUEFI},
}, nil
@@ -127,7 +130,11 @@ func cachedImage(ctx context.Context, log logr.Logger, pool libvirt.StoragePool,
case !hexDigest.MatchString(digest):
log.V(1).Info("base image marker holds no digest; refetching")
case src.SHA256 != "" && !strings.EqualFold(digest, src.SHA256):
log.V(1).Info("base image digest differs from the configured sha256; refetching", "cached", digest, "configured", strings.ToLower(src.SHA256))
// Refusing rather than replacing: every overlay on this host names
// the base as its backing file, and libvirt will unlink it under
// running domains without a word.
return "", permanentf("base image %s is cached with sha256 %s but %s is configured; add the image under a new key and move Proxies to it, or delete the Proxies using it and evict the base by hand",
name, digest, strings.ToLower(src.SHA256))
default:
base, err := be.StorageVolLookupByName(ctx, pool, name)
switch {
internal/provider/kvm2/image_test.go
+28 -26
View File
@@ -100,7 +100,7 @@ func (fx *imageFixture) sameConnection() {
func (fx *imageFixture) src() imageSource {
fx.t.Helper()
src, err := resolveImage(fx.p.cfg, imageKey)
src, err := resolveImage(fx.p.name, fx.p.cfg, imageKey)
if err != nil {
fx.t.Fatalf("resolveImage(%q) error = %v", imageKey, err)
}
@@ -180,20 +180,20 @@ func TestResolveImage_table(t *testing.T) {
{
name: "catalogue key with defaults applied",
image: "debian-13",
want: imageSource{Key: "debian-13", KVMImage: provider.KVMImage{URL: "https://cloud.debian.org/debian-13.qcow2", SHA256: "ABCD", Format: "qcow2", Firmware: "uefi"}},
wantVolume: "base-debian-13.qcow2",
want: imageSource{Entry: testProvider, Key: "debian-13", KVMImage: provider.KVMImage{URL: "https://cloud.debian.org/debian-13.qcow2", SHA256: "ABCD", Format: "qcow2", Firmware: "uefi"}},
wantVolume: "base-" + testProvider + "-debian-13.qcow2",
},
{
name: "catalogue key with explicit format and firmware",
image: "alpine",
want: imageSource{Key: "alpine", KVMImage: provider.KVMImage{URL: "https://alpine.example/alpine.img", Format: "raw", Firmware: "bios"}},
wantVolume: "base-alpine.raw",
want: imageSource{Entry: testProvider, Key: "alpine", KVMImage: provider.KVMImage{URL: "https://alpine.example/alpine.img", Format: "raw", Firmware: "bios"}},
wantVolume: "base-" + testProvider + "-alpine.raw",
},
{
name: "http URL used directly with defaults",
image: rawURL,
want: imageSource{Key: urlImageKey(rawURL), KVMImage: provider.KVMImage{URL: rawURL, Format: "qcow2", Firmware: "uefi"}},
wantVolume: "base-" + urlImageKey(rawURL) + ".qcow2",
want: imageSource{Entry: testProvider, Key: urlImageKey(rawURL), KVMImage: provider.KVMImage{URL: rawURL, Format: "qcow2", Firmware: "uefi"}},
wantVolume: "base-" + testProvider + "-" + urlImageKey(rawURL) + ".qcow2",
},
{
name: "unknown key is permanent and names the value",
@@ -214,7 +214,7 @@ func TestResolveImage_table(t *testing.T) {
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, err := resolveImage(cfg, tc.image)
got, err := resolveImage(testProvider, cfg, tc.image)
if tc.wantErr != "" {
if err == nil || !errors.Is(err, provider.ErrPermanent) || !strings.Contains(err.Error(), tc.wantErr) {
t.Fatalf("resolveImage(%q) = %+v, %v; want ErrPermanent containing %q", tc.image, got, err, tc.wantErr)
@@ -278,7 +278,7 @@ func TestEnsureImage_downloadsAndWritesMarkerLast(t *testing.T) {
t.Errorf("got %d lines with %s, want 1:\n%s", len(lines), want, strings.Join(fx.logs.all(), "\n"))
continue
}
for _, kv := range []string{`"provider"="kvm2-home"`, `"host"="beelink"`, `"volume"="base-debian-13.qcow2"`, `"bytes"=` + strconv.Itoa(len(body)), `"traceID"=`} {
for _, kv := range []string{`"provider"="kvm2-home"`, `"host"="beelink"`, `"volume"="base-` + testProvider + `-debian-13.qcow2"`, `"bytes"=` + strconv.Itoa(len(body)), `"traceID"=`} {
if !strings.Contains(lines[0], kv) {
t.Errorf("line lacks %s: %s", kv, lines[0])
}
@@ -295,7 +295,7 @@ func TestEnsureImage_downloadsAndWritesMarkerLast(t *testing.T) {
attrs := spans[0].Attributes()
for _, want := range []attribute.KeyValue{
attribute.String("kvm2.host", "beelink"),
attribute.String("kvm2.volume", "base-debian-13.qcow2"),
attribute.String("kvm2.volume", "base-"+testProvider+"-debian-13.qcow2"),
attribute.String("kvm2.url_host", strings.TrimPrefix(fx.srv.URL, "http://")),
attribute.Int64("kvm2.bytes", int64(len(body))),
attribute.Bool("kvm2.sha256_configured", false),
@@ -391,7 +391,13 @@ func TestEnsureImage_missingMarkerRedownloads(t *testing.T) {
}
}
func TestEnsureImage_sha256ChangeRedownloads(t *testing.T) {
// TestEnsureImage_sha256ChangeRefusesInsteadOfReplacing pins the rule the
// contract already stated and the code contradicted. Every overlay on the
// host names the base as its backing file, and libvirt unlinks it under
// running domains without complaint, so a digest that no longer matches is
// for the operator to resolve, not for the next Create to resolve by
// deleting.
func TestEnsureImage_sha256ChangeRefusesInsteadOfReplacing(t *testing.T) {
t.Parallel()
body := randomBody(t, 8192)
fx := newImageFixture(t, serveBytes(body))
@@ -401,25 +407,21 @@ func TestEnsureImage_sha256ChangeRedownloads(t *testing.T) {
fx.putVol(src.volumeName(), []byte("the old image"))
fx.putVol(src.markerName(), []byte(digestA))
if _, err := fx.ensure(fx.ctx, src); err != nil {
t.Fatalf("ensureImage() error = %v", err)
_, err := fx.ensure(fx.ctx, src)
if !errors.Is(err, provider.ErrPermanent) || !strings.Contains(err.Error(), "add the image under a new key") {
t.Fatalf("ensureImage() error = %v, want ErrPermanent telling the operator what to do", err)
}
if fx.hits.Load() != 1 {
t.Errorf("server hits = %d, want 1", fx.hits.Load())
if fx.hits.Load() != 0 {
t.Errorf("server hits = %d, want 0: nothing should be refetched", fx.hits.Load())
}
if n := fx.pooled.callCount("StorageVolDelete"); n != 2 {
t.Errorf("StorageVolDelete called %d times, want 2 (marker and base)", n)
if n := fx.pooled.callCount("StorageVolDelete"); n != 0 {
t.Errorf("StorageVolDelete called %d times, want 0: the base backs live overlays", n)
}
base := fx.pooled.vol(src.volumeName())
if base == nil || !bytes.Equal(base.content, body) {
t.Errorf("base volume was not replaced by the download")
if base := fx.pooled.vol(src.volumeName()); base == nil || !bytes.Equal(base.content, []byte("the old image")) {
t.Errorf("base volume = %+v, want the cached one untouched", base)
}
marker := fx.pooled.vol(src.markerName())
if marker == nil || string(marker.content) != digestOf(body) {
t.Errorf("marker = %+v, want the new digest %s", marker, digestOf(body))
}
if lines := fx.logs.matching("differs from the configured sha256"); len(lines) != 1 {
t.Errorf("got %d refetch lines, want 1", len(lines))
if marker := fx.pooled.vol(src.markerName()); marker == nil || string(marker.content) != digestA {
t.Errorf("marker = %+v, want the cached digest untouched", marker)
}
}
internal/provider/kvm2/kvm2.go
+3
View File
@@ -74,6 +74,9 @@ var _ provider.Provider = (*Provider)(nil)
// files are only checked for existence, and a missing one is a warning, not
// an error, because the Secret mount is optional and may land later.
func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, error) {
if !provider.IsKVMIdentifier(pc.Name) {
return nil, permanentf("provider name %q: kvm2 names its cached base images after the entry, so the name must be letters, digits, dots, dashes or underscores", pc.Name)
}
cfg := provider.KVMConfig{}
if pc.KVM2 != nil {
cfg = *pc.KVM2
internal/provider/kvm2/kvm2_integration_test.go
+1 -1
View File
@@ -481,7 +481,7 @@ func TestKVM2_lifecycle(t *testing.T) {
// 10. The pool keeps the shared base image and its marker, and holds
// no per-VM volume.
start = time.Now()
base := baseVolumeName(integrationImage, provider.KVMFormatQcow2)
base := baseVolumeName(integrationProvider, integrationImage, provider.KVMFormatQcow2)
marker := markerVolumeName(base)
overlay, seed := overlayVolumeName(name), seedVolumeName(name)
present := poolVolumes(ctx, t, p, host, base, marker, overlay, seed)
internal/provider/kvm2/kvm2_test.go
+12
View File
@@ -110,6 +110,18 @@ func TestNew_nilBlockDoesNotPanic(t *testing.T) {
}
}
// TestNew_rejectsUnsafeEntryName guards the one config value that reaches a
// volume name without passing the kvm validator: the provider entry's own
// name, which is checked only for being non-empty and unique.
func TestNew_rejectsUnsafeEntryName(t *testing.T) {
t.Parallel()
ctx, _ := captureContext()
_, err := New(ctx, provider.ProviderConfig{Name: "kvm2/home", Type: "kvm2"})
if !errors.Is(err, provider.ErrPermanent) || !strings.Contains(err.Error(), "kvm2/home") {
t.Fatalf("New() error = %v, want ErrPermanent naming the entry", err)
}
}
func TestHost_table(t *testing.T) {
t.Parallel()
tests := []struct {
internal/provider/kvm2/provider_test.go
+13 -2
View File
@@ -129,10 +129,21 @@ func (fx *fixture) putVol(name string, content []byte) *fakeVol {
return v
}
// baseName is the cached base image's volume name for this fixture's
// provider entry and image key.
func (fx *fixture) baseName() string {
fx.t.Helper()
src, err := resolveImage(fx.p.name, fx.p.cfg, imageKey)
if err != nil {
fx.t.Fatal(err)
}
return src.volumeName()
}
// cacheImage places a complete base image and its marker, as a previous
// Create on the host would have.
func (fx *fixture) cacheImage() {
src, err := resolveImage(fx.p.cfg, imageKey)
src, err := resolveImage(fx.p.name, fx.p.cfg, imageKey)
if err != nil {
fx.t.Fatal(err)
}
@@ -406,7 +417,7 @@ func TestLogging_v0SilentV1OutcomeV2Detail(t *testing.T) {
if len(resolved) != 1 {
t.Fatalf("V(2) lines matching resolved = %d, want 1", len(resolved))
}
for _, key := range []string{`"cloudInitBytes"=`, `"image"="base-debian-13.qcow2"`, `"firmware"="uefi"`, `"vcpus"=1`, `"memoryMiB"=1024`, `"machineType"="small"`, `"bridge"="br0"`, `"pool"="default"`} {
for _, key := range []string{`"cloudInitBytes"=`, `"image"="base-` + testProvider + `-debian-13.qcow2"`, `"firmware"="uefi"`, `"vcpus"=1`, `"memoryMiB"=1024`, `"machineType"="small"`, `"bridge"="br0"`, `"pool"="default"`} {
if !strings.Contains(resolved[0], key) {
t.Errorf("resolved line lacks %s: %s", key, resolved[0])
}
internal/provider/kvm2/volume.go
+8 -1
View File
@@ -14,7 +14,14 @@ import (
func overlayVolumeName(name string) string { return name + ".qcow2" }
func seedVolumeName(name string) string { return name + "-seed.iso" }
func baseVolumeName(key, format string) string { return "base-" + key + "." + format }
// baseVolumeName carries the provider entry because the base is shared by
// every Proxy on a host but must not be shared by two entries: the kvm
// provider builds base-<key>.<format> in the same pool from the same
// catalogue shape, and two kvm2 entries pointing one key at different URLs
// would otherwise agree on the name and disagree on the contents.
func baseVolumeName(entry, key, format string) string {
return "base-" + entry + "-" + key + "." + format
}
func markerVolumeName(baseName string) string { return baseName + ".sha256" }
// urlImageKey stands in for the catalogue key of an image given as a URL:
internal/provider/kvm2/volume_test.go
+7 -3
View File
@@ -119,10 +119,14 @@ func TestVolumeNames(t *testing.T) {
}{
{name: "overlay", got: overlayVolumeName("proxy-abc"), want: "proxy-abc.qcow2"},
{name: "seed", got: seedVolumeName("proxy-abc"), want: "proxy-abc-seed.iso"},
{name: "base qcow2", got: baseVolumeName("debian-13", provider.KVMFormatQcow2), want: "base-debian-13.qcow2"},
{name: "base raw", got: baseVolumeName("alpine", provider.KVMFormatRaw), want: "base-alpine.raw"},
{name: "base qcow2", got: baseVolumeName("kvm2-home", "debian-13", provider.KVMFormatQcow2), want: "base-kvm2-home-debian-13.qcow2"},
{name: "base raw", got: baseVolumeName("kvm2-home", "alpine", provider.KVMFormatRaw), want: "base-kvm2-home-alpine.raw"},
{name: "marker", got: markerVolumeName("base-debian-13.qcow2"), want: "base-debian-13.qcow2.sha256"},
{name: "marker of url base", got: markerVolumeName(baseVolumeName(urlImageKey("https://x/y.img"), "qcow2")), want: "base-" + urlImageKey("https://x/y.img") + ".qcow2.sha256"},
{name: "marker of url base", got: markerVolumeName(baseVolumeName("kvm2-home", urlImageKey("https://x/y.img"), "qcow2")), want: "base-kvm2-home-" + urlImageKey("https://x/y.img") + ".qcow2.sha256"},
// Two entries pointing one key at different images must not agree
// on a file name, and neither may collide with the kvm provider's
// base-<key>.<format> in the same pool.
{name: "another entry, same key", got: baseVolumeName("kvm2-lab", "debian-13", provider.KVMFormatQcow2), want: "base-kvm2-lab-debian-13.qcow2"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {