Files
egress-proxies-operator/internal/provider/config_kvm.go
T
kacerrandClaude Opus 5 c10f98a352 kvm2: name the cached base image after the provider entry (H4)
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm
provider builds in the same pool -- for URL images too, since kvm takes
hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with
a completely separate lock table. The shipped config has a live kvm-home
entry and a commented-out kvm2-home entry on the same host, the same key
debian-13 and the same default pool, and the comment invites the operator
to enable it. Both integration tests use key test-image. On beelink both
live overlays back onto kvm's base-debian-13.qcow2.

Since cachedImage accepts any 64-hex marker when no sha256 is configured
and never re-hashes the base, that meant two entries pointing one key at
different URLs silently shared a file; a concurrent cold fetch let one
entry delete the other's half-uploaded base and boot a guest from the
remains; and a sha256 edit deleted the base in place under live overlays,
which libvirt permits without complaint.

The name now carries the provider entry: base-<entry>-<key>.<format>.
imageSource gained an Entry field and resolveImage takes the provider
name, so every production call site follows from imageSource.volumeName().
That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2
alike, and turns the per-Provider imageLocks from a latent bug into a
correct one: with names unique per entry, two entries cannot contend for a
file, so no process-wide lock table is needed.

The entry name now reaches a file name, so New checks it against the
existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather
than duplicated. Provider names are otherwise validated for all types at
once and only for being non-empty and unique, so a name with a slash would
have produced a volume nothing could address.

One companion fix, because the naming alone does not stop it:
cachedImage's digest-mismatch branch deleted the marker and the base and
refetched. It now returns ErrPermanent naming both digests and telling the
operator what to do. That branch was the remaining path that unlinks a
good file under running domains, and the contract already said "add a new
key instead" -- the code was contradicting it. The no-marker and
non-hex-marker branches keep deleting, since there the base is debris.

Not fixed, and said so in the docs rather than left to look closed:
editing images.<key>.url is still a silent no-op, since nothing records
the URL; and cross-process fencing still needs the deferred atomic publish
(upload to a temp name, verify, publish), which lands where H2 put the
discard.

Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's,
kvm keeps using it untouched, and kvm2 downloads its own copy once per
host. Nothing is replaced under a running guest; the pool needs room for
both, which the prerequisites now say.

The eviction recipe in the guide named base-debian-13.qcow2, which after
this change is kvm's file -- following it would have deleted the wrong
provider's base. It now greps for the prefix and deletes the marker first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:57:02 +00:00

198 lines
8.3 KiB
Go

package provider
import (
"fmt"
"net/url"
"regexp"
)
// KVMConfig configures a kvm provider instance: one or more libvirt/KVM
// hypervisors reached over SSH (the libvirt unix socket is forwarded through
// the SSH connection, so hosts need no libvirtd TCP/TLS listener).
// placement.zone on a Proxy selects a host by name; with a single host it may
// be omitted.
type KVMConfig struct {
// SSHKeyFile is the path to the private key used for every host. Required.
SSHKeyFile string `json:"sshKeyFile"`
// KnownHostsFile is an OpenSSH known_hosts file used to verify host keys.
// Required unless InsecureIgnoreHostKey is set.
KnownHostsFile string `json:"knownHostsFile,omitempty"`
// InsecureIgnoreHostKey skips host-key verification entirely.
InsecureIgnoreHostKey bool `json:"insecureIgnoreHostKey,omitempty"`
// DiskSizeGB is the capacity of each VM's overlay disk. Default 10. The
// guest's cloud-init growpart expands the root filesystem into it.
DiskSizeGB int64 `json:"diskSizeGb,omitempty"`
// VendorData is optional cloud-init vendor-data placed on every seed next
// to the Proxy's own user-data. Note that cloud-init drops any top-level
// key of a vendor-data cloud-config that the user-data also defines.
// When InstallGuestAgent is on, this document and the agent script are
// carried as separate parts of a MIME multipart vendor-data.
VendorData string `json:"vendorData,omitempty"`
// InstallGuestAgent adds a vendor-data script that installs and starts
// qemu-guest-agent, which is how the provider learns a VM's address
// before the host's ARP table has seen traffic from it. Default true.
// Turn it off for images that already ship the agent, or that cannot
// reach a package mirror; discovery then depends on ARP alone.
InstallGuestAgent *bool `json:"installGuestAgent,omitempty"`
// Images are named base images; placement.image names a key here (or is
// an http(s) URL directly). The provider downloads each image into a
// host's storage pool on first use and caches it there.
Images map[string]KVMImage `json:"images,omitempty"`
// MachineTypes are named size profiles; placement.machineType names a
// key here. Required, at least one.
MachineTypes map[string]KVMMachineType `json:"machineTypes"`
// Hosts are the hypervisors. Required, at least one, unique names.
Hosts []KVMHost `json:"hosts"`
}
// KVMImage is one downloadable base image (a distro cloud image with
// cloud-init preinstalled, e.g. Debian's generic qcow2).
type KVMImage struct {
// URL is where to fetch the image from. Required, http or https.
URL string `json:"url"`
// SHA256 is the expected hex digest of the file. Optional; when set it is
// verified after download and a later change triggers a re-download.
SHA256 string `json:"sha256,omitempty"`
// Format is the image's disk format, "qcow2" (default) or "raw".
Format string `json:"format,omitempty"`
// Firmware is how VMs from this image boot: "uefi" (default; OVMF with
// secure boot off) or "bios" (SeaBIOS) for images with no EFI system
// partition. Current Debian, Ubuntu, Fedora and Rocky cloud images all
// boot UEFI, and Debian 13's BIOS path was found not to boot at all.
Firmware string `json:"firmware,omitempty"`
}
// KVMMachineType is a VM size profile.
type KVMMachineType struct {
VCPUs int32 `json:"vcpus"`
MemoryMiB int64 `json:"memoryMiB"`
}
// KVMHost is one hypervisor.
type KVMHost struct {
// Name is the placement.zone value that selects this host. Required.
Name string `json:"name"`
// Address is the SSH endpoint, host:port. Required.
Address string `json:"address"`
// User is the SSH user; it must be allowed to use the libvirt socket
// (typically a member of the libvirt group). Required.
User string `json:"user"`
// Socket is the libvirt unix socket path on the host.
// Default /var/run/libvirt/libvirt-sock.
Socket string `json:"socket,omitempty"`
// Pool is the libvirt storage pool for base images, overlay disks and
// seed ISOs. Default "default".
Pool string `json:"pool,omitempty"`
// Bridge is the host bridge the VM's NIC attaches to; it must lead to a
// network the cluster can reach and that hands out DHCP leases. Required.
Bridge string `json:"bridge"`
}
// Accepted KVMImage.Format and KVMImage.Firmware values.
const (
KVMFormatQcow2 = "qcow2"
KVMFormatRaw = "raw"
KVMFirmwareUEFI = "uefi"
KVMFirmwareBIOS = "bios"
)
// IsHTTPURL reports whether s is an absolute http(s) URL with a host, the
// only kind of image source the kvm provider can fetch.
func IsHTTPURL(s string) bool {
u, err := url.Parse(s)
return err == nil && (u.Scheme == "http" || u.Scheme == "https") && u.Host != ""
}
var sha256Hex = regexp.MustCompile(`^[0-9a-fA-F]{64}$`)
// kvmIdentifier constrains the config names that are pasted into
// identifiers elsewhere: a host name goes into the providerID
// hosts/<name>/domains/<vm>, which is split on "/" to parse, and an image
// key becomes part of the file name of the cached base volume. A slash in
// either produces an ID or a path that never resolves again, so the VM
// leaks with nothing able to address it.
var kvmIdentifier = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*$`)
// IsKVMIdentifier reports whether s is safe to paste into a providerID or a
// volume name. The provider entry's own name reaches a volume name too, and
// that name is validated for every provider type at once (only non-empty
// and unique), so the kvm2 provider checks its own at construction.
func IsKVMIdentifier(s string) bool { return kvmIdentifier.MatchString(s) }
func validateKVM(prefix string, c *KVMConfig) error {
return validateKVMBlock(prefix, "kvm", c)
}
// validateKVMBlock checks a KVMConfig under the given block name, so the
// kvm and kvm2 provider types share one validator and each reports its own
// block in messages.
func validateKVMBlock(prefix, block string, c *KVMConfig) error {
if c == nil {
return fmt.Errorf("%s: %s block is required", prefix, block)
}
if c.SSHKeyFile == "" {
return fmt.Errorf("%s: %s.sshKeyFile is required", prefix, block)
}
if c.KnownHostsFile == "" && !c.InsecureIgnoreHostKey {
return fmt.Errorf("%s: %s.knownHostsFile is required unless %s.insecureIgnoreHostKey is true", prefix, block, block)
}
if c.DiskSizeGB < 0 {
return fmt.Errorf("%s: %s.diskSizeGb must not be negative", prefix, block)
}
for name, img := range c.Images {
p := fmt.Sprintf("%s: %s.images[%q]", prefix, block, name)
if !kvmIdentifier.MatchString(name) {
return fmt.Errorf("%s: key must be letters, digits, dots, dashes or underscores"+
" — it becomes part of the cached base volume's name", p)
}
if !IsHTTPURL(img.URL) {
return fmt.Errorf("%s.url must be an http(s) URL (got %q)", p, img.URL)
}
if img.SHA256 != "" && !sha256Hex.MatchString(img.SHA256) {
return fmt.Errorf("%s.sha256 must be 64 hex characters", p)
}
if img.Format != "" && img.Format != KVMFormatQcow2 && img.Format != KVMFormatRaw {
return fmt.Errorf("%s.format must be %s or %s (got %q)", p, KVMFormatQcow2, KVMFormatRaw, img.Format)
}
if img.Firmware != "" && img.Firmware != KVMFirmwareUEFI && img.Firmware != KVMFirmwareBIOS {
return fmt.Errorf("%s.firmware must be %s or %s (got %q)", p, KVMFirmwareUEFI, KVMFirmwareBIOS, img.Firmware)
}
}
if len(c.MachineTypes) == 0 {
return fmt.Errorf("%s: %s.machineTypes must define at least one profile", prefix, block)
}
for name, mt := range c.MachineTypes {
if mt.VCPUs <= 0 || mt.MemoryMiB <= 0 {
return fmt.Errorf("%s: %s.machineTypes[%q] needs vcpus > 0 and memoryMiB > 0", prefix, block, name)
}
}
if len(c.Hosts) == 0 {
return fmt.Errorf("%s: %s.hosts must list at least one host", prefix, block)
}
seen := make(map[string]bool, len(c.Hosts))
for i, h := range c.Hosts {
p := fmt.Sprintf("%s: %s.hosts[%d]", prefix, block, i)
if h.Name == "" {
return fmt.Errorf("%s: name is required", p)
}
if !kvmIdentifier.MatchString(h.Name) {
return fmt.Errorf("%s: name %q must be letters, digits, dots, dashes or underscores"+
" — it becomes part of the providerID hosts/<name>/domains/<vm>", p, h.Name)
}
if seen[h.Name] {
return fmt.Errorf("%s: duplicate host name %q", p, h.Name)
}
seen[h.Name] = true
if h.Address == "" {
return fmt.Errorf("%s %q: address is required", p, h.Name)
}
if h.User == "" {
return fmt.Errorf("%s %q: user is required", p, h.Name)
}
if h.Bridge == "" {
return fmt.Errorf("%s %q: bridge is required", p, h.Name)
}
}
return nil
}