egress-proxies-operator/docs/kvm2-provider-spec.md

Provider spec: kvm2

Filled from provider-spec-template.md against provider-contract.md. This is the design the kvm2 package is built from. Where an entry names an axis in backticks, the answer is one of that axis's profiles in contract §4. Nothing in this spec was taken from the code of the existing kvm provider; backend facts come from kvm-provider.md, the execution log of the first build, the go-libvirt module in the module cache, and the beelink host.

Header

• Type: kvm2
• Example configured name: kvm2-home
• Backend and API version this spec was written against: libvirt 11.6.0 and QEMU 10.1.0 on Ubuntu (host beelink), reached with go-libvirt github.com/digitalocean/go-libvirt v0.0.0-20260814190004-1a83157e1858 over an SSH-forwarded unix socket.
• Client: rpc-over-ssh. go-libvirt's RPC client (libvirt.New(conn) then Connect()) over a direct-streamlocal channel opened with golang.org/x/crypto/ssh to the host's libvirt socket. Both modules are already direct requires in go.mod; no new dependency.
• Author and date: Claude (orchestrator session), 2026-09-16.

1. Identity and providerID

Answer: hosts/<host>/domains/<name>, where <host> is hosts[].name from the config block and <name> is req.Name. Parsing is strict: exactly four segments, literal hosts and domains, non-empty host and name, no further slashes; anything else is ErrPermanent from Get and Delete. Host names are validated at config load by the shared kvmIdentifier regexp in internal/provider/config_kvm.go (letters, digits, dots, dashes, underscores), so a host name can never contain the separator. Instances do not migrate between hosts (libvirt domains are per host), so the location belongs in the ID.

Evidence: contract §6; kvmIdentifier in internal/provider/config_kvm.go.

2. Primary key (primary-key)

Answer: name-is-key. libvirt addresses domains by name (DomainLookupByName); req.Name is used verbatim as the domain name and as the base of the per-VM volume names. Create looks the name up first and adopts an existing domain only if it carries the kvm2 mark (managed true, entry 8) and its metadata uid is empty or equals req.UID. A domain without the mark returns ErrPermanent saying it was not created by this provider: kvm2 never starts or takes over a VM someone else defined under that name. A different non-empty uid returns ErrPermanent naming both UIDs. This closes the contract's "adoption is not verified" limit for this provider.

Evidence: DomainLookupByName(Name string) (Domain, error) in the go-libvirt module; contract §2.1 idempotency and §13.

3. Placement mapping

Answer: three fields are catalogue keys checked before any backend call, one is ignored:

Field	Meaning	Required	Unknown value
Zone	a hosts[].name	only when more than one host is configured; with exactly one host an empty zone selects it	ErrPermanent: placement.zone "x" is not in kvm2.hosts
MachineType	a key of machineTypes (vcpus, memoryMiB)	yes	ErrPermanent: placement.machineType "x" is not in kvm2.machineTypes
Image	a key of images, or an absolute http(s) URL used directly	yes	ErrPermanent: placement.image "x" is neither a key in kvm2.images nor an http(s) URL
Region	ignored		still part of the spec hash: changing it replaces the VM

All three checks run before any backend call. The guide documents the ignored field and that catalogue edits affect only new VMs.

Evidence: contract §2.1 (reject early, MAY ignore with documentation); IsHTTPURL in internal/provider/config_kvm.go.

4. Image source

Answer: a catalogue in the config block (images.<key>.url, optional sha256, format qcow2 or raw, firmware uefi or bios) or a URL given directly in placement.image. The provider downloads the image into the host's storage pool on first use and caches it there as a base volume named base-<entry>-<key>.<format> (for a URL: <key> is the first 16 hex of sha256(url)), where <entry> is the provider entry's name, with a sidecar marker volume base-<entry>-<key>.<format>.sha256 holding the digest of the verified upload, written last. The entry is in the name because the kvm provider builds base-<key>.<format> in the same pool from the same catalogue shape, and two entries pointing one key at different URLs would otherwise share a file; the cost is one copy per entry per host, and the entry's name is checked against kvmIdentifier at construction. A base volume without a marker is incomplete and is deleted and redone. When sha256 is configured and differs from the marker's content, the image is not replaced: that is ErrPermanent naming both digests, because overlays back onto the base file and libvirt unlinks it under running domains without complaint. The download streams from the URL through the operator into StorageVolUpload with a running sha256; nothing is buffered. Content-Length is required because the volume's capacity must be declared before the upload; a response without it is ErrPermanent naming the URL. Per (host, volume) a keyed mutex makes concurrent Creates download once. The download runs on a dedicated connection to the host, not the pooled one, so a cancelled poll on another Proxy cannot cut it off. After a successful upload StoragePoolRefresh makes the file usable as a backing image.

Catalogue edit warning for the guide: overlays back onto the base file, so evicting a base volume while Proxies use it corrupts those VMs. A sha256 change is refused for that reason; a url change is a silent no-op, since the cache is keyed by entry and key. Add a new key for either.

Evidence: kvm-provider.md "The image cache" and "Known gaps"; execution log of 2026-09-04 (raw qcow2 upload plus pool refresh accepted by libvirt 11.6; 435 MB trixie image); contract §1.4 (stream, no buffering) and §2.1 (catalogue changes).

5. Cloud-init delivery

Answer: a NoCloud seed ISO built in-process with github.com/kdomanski/iso9660, volume label cidata, files user-data (the Proxy's CloudInit, byte for byte, never parsed), meta-data (instance-id: <name> and local-hostname: <name>, so a reboot is never taken for a new instance) and vendor-data. The ISO is written into a temp directory created under the default temp dir (/tmp, a 64Mi emptyDir in the manager), uploaded as a raw volume <name>-seed.iso and attached as a CD-ROM. The seed is rebuilt on every Create attempt: a previous attempt may have been cut off mid-upload, and a truncated seed would boot a VM without cloud-init.

vendor-data carries a shell script that installs and starts qemu-guest-agent with whichever of apt-get, dnf, yum, zypper or apk the image has. It is a script and not a cloud-config on purpose: cloud-init merges a vendor-data cloud-config under the user-data with mergers that keep the user-data's value for every top-level key both define and never append lists, so a Proxy with its own packages: would silently lose the agent. Scripts bypass the merge. When the config's vendorData is set, the two travel as separate parts of a MIME multipart (text/plain for the configured document, text/x-shellscript for the agent script), headed by MIME-Version within the first bytes. With installGuestAgent: false and no vendorData the file is empty. Size: cloud-init has no practical limit for NoCloud; the ISO is a few hundred KB.

Evidence: kvm-provider.md "What lands on the host"; execution log run 7 (vendor-data as YAML silently dropped) and run 8 (script survives); contract §2.1 (byte for byte, never logged) and §11 (sentinel test).

6. Machine type mapping

Answer: a catalogue: machineTypes.<key> is {vcpus, memoryMiB}, validated at load (both greater than zero, at least one profile). Disk size is provider-wide: diskSizeGb (default 10) is the capacity of every VM's overlay volume, thin-provisioned; cloud-init's growpart expands the root filesystem into it.

Evidence: KVMMachineType and validateKVM in internal/provider/config_kvm.go.

7. Network and IP discovery (readiness, ip-source)

Answer: readiness: provisioning-window (boot plus cloud-init, plus the image download on a cold pool). ip-source: discovered. The VM gets one virtio NIC on hosts[].bridge, which must lead to a LAN with DHCP that the cluster can route to. The address comes from DomainInterfaceAddresses, guest agent source first, then the ARP table (on beelink the agent answered about 20 s after start where ARP never did, because the host itself does not talk to the guest). The first IPv4 that is not loopback, link-local or unspecified is reported. One source failing means "no address yet" and Get returns StateProvisioning, which is the ordinary boot on beelink: the agent is not up while ARP answers with nothing. A connection already gone, or every source failing, is ErrTransient instead — a running VM whose address cannot be read is not a VM without an address, and reporting it as one would make the reconciler clear its IP and the health engine drop its counters. The guest needs the agent: it is installed by the vendor-data script (entry 5) unless installGuestAgent: false, which needs an image that ships it or a network where ARP sees guest traffic. Port is not used by the provider; nothing on the host filters it. Reachability is a prerequisite in the guide: the operator's pod network and the clients must route to the bridge's LAN.

Evidence: kvm-provider.md "Hypervisor prerequisites" and "Known gaps" (IP discovery); execution log run 6 (agent 20 s, ARP never); contract §1.3 and §2.2 (Running only with IP).

8. GC marks and CreatedAt (gc-mark, created-at)

Answer: gc-mark: written-metadata. created-at: written-at-create. The domain XML passed to DomainDefineXML carries, under <metadata>, an element <instance xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2"> with children <managed>true</managed>, <uid>req.UID</uid> and <createdAt>RFC3339 UTC</createdAt>. The mark lands in the same call that defines the domain, so there is no mark window. Read back with DomainGetMetadata(dom, DomainMetadataElement, uri, DomainAffectCurrent); libvirt 11.6 returns the element without its namespace declaration, so the parser matches on the local name instance only (the URI was already used to select it). A domain without the element is not managed and reports an empty UID. A createdAt that does not parse gives the zero time, which GC treats as old; that is accepted. The volumes carry no mark (documented GC blind spot, entry 24). The namespace is kvm2's own, so kvm2 never lists or adopts domains made by the kvm provider; the two types may share a host.

Evidence: execution log run 3 (namespace asymmetry); DomainGetMetadata(Dom Domain, Type int32, Uri OptString, Flags DomainModificationImpact) in the go-libvirt module; contract §2.1 (mark in the create call) and §4.5.

9. State mapping (provisioning-state)

Answer: provisioning-state: create-blocks. Create defines and starts the domain in one call and returns only once DomainGetState reports running, so Get never sees a domain kvm2 defined but did not start. A domain that is shut off later stopped on its own or was stopped by someone, and Stopped is the right answer.

Backend state (DomainGetState)	Lock or task	Provider state
nostate (0)	none	Provisioning (libvirt could not determine it; next poll decides)
running (1), blocked (2)	no IP found	Provisioning
running (1), blocked (2)	IP found	Running
paused (3), shutdown (4), shutoff (5), crashed (6), pmsuspended (7)	none	Stopped
any other code	none	Stopped

libvirt has no locks or tasks; every RPC is synchronous.

Evidence: go-libvirt DomainState constants; contract §2.2 and §4.11.

10. Create: blocking and task tracking (create-return, async-ops)

Answer: create-return: blocking. async-ops: none. Create returns when the domain is running, holding one reconcile worker for the duration, including the first image download on a host (minutes on a cold pool; the guide says so). The wait after DomainCreate is capped at 15 seconds: DomainCreate is synchronous in libvirt, so the first DomainGetState normally answers running and the cap only covers state settling. Two rules keep the wait from pinning a worker: a state after the start that is neither active nor nostate (shutoff, crashed, paused, pmsuspended, shutdown) is ErrTransient at once, since polling will not bring that guest up; and a domain found paused or pmsuspended before the start is ErrPermanent (cannot be started by the operator; resume or destroy it on the host), an admin's doing that DomainCreate would refuse and the reconciler must not retry. Every RPC has its own deadline (entry 18).

Steps, each idempotent, in order, all under the per-name lock (entry 19):

1. Validate placement (entry 3). No backend call yet.
2. DomainLookupByName. Found: adoption guard (entry 2), then start it if it is not running (paused or pmsuspended is ErrPermanent; DomainCreate; ErrOperationInvalid means already running), poll as in step 9, return the ID.
3. Pool space precheck (entry 13, quota).
4. Ensure the base image (entry 4): marker present and matching means done.
5. Overlay volume <name>.qcow2, qcow2 backed by the base file with diskSizeGb capacity; ErrStorageVolExist means reuse (creation is atomic, so an existing overlay is complete).
6. Seed volume <name>-seed.iso: delete if it exists, create, upload (entry 5). Always rebuilt.
7. StorageVolGetPath for the overlay and the seed.
8. DomainDefineXML with file-type disks (entry 15's AppArmor note), the metadata element, firmware per image.
9. DomainCreate, then poll DomainGetState until running: nostate is polled again, any other inactive state is ErrTransient at once, and the 15-second cap is ErrTransient too.
10. Return hosts/<host>/domains/<name>.

A retry after a crash finds whatever the previous attempt got to: an existing overlay is reused, a seed is rebuilt, a defined domain is started, a running domain is returned. ErrNoDomain and ErrNoStorageVol inside Create (a volume vanishing between steps) are ErrTransient, never ErrNotFound. The log line for a Create records cloudInitBytes, never the content.

Evidence: contract §2.1, §4.2, §5 ("do not return NotFound from Create"); execution log run 1 (AppArmor).

11. Delete and secondary resources (delete-return, secondary-resources)

Answer: delete-return: waits. secondary-resources: explicit. Under the per-name lock:

1. Parse the ID (ErrPermanent if malformed). A host no longer in the config is a logged no-op success: the provider cannot reach it and refusing would pin the finalizer; the guide tells admins to delete Proxies before retiring a host.
2. DomainLookupByName; ErrNoDomain means the domain is already gone, skip to step 5.
3. DomainDestroy; ErrOperationInvalid (not running) and ErrNoDomain are success.
4. DomainUndefineFlags with DomainUndefineNvram, DomainUndefineManagedSave and DomainUndefineSnapshotsMetadata (UEFI domains keep NVRAM; undefine refuses without the flag); ErrNoDomain is success.
5. Delete <name>.qcow2 and <name>-seed.iso by name from the host's pool; ErrNoStorageVol is success, and ErrNoStoragePool means nothing is left to clean.
6. Return nil. The base image and its marker stay (shared cache).

Everything is gone when Delete returns nil, so Get afterwards is NotFound. A second Delete during the first waits on the lock and then finds nothing. Volumes that exist without a domain (a crash between step 5 and 8 of Create) are removed by the same Proxy's next Delete but are invisible to GC (entry 24).

Evidence: kvm-provider.md "Known gaps"; execution log run 4 (NVRAM); contract §2.3.

12. ListByTag scope and partial failure (list-partial)

Answer: fail-whole, per host, with single unreadable domains skipped. Scope: every host in hosts, in config order, ConnectListAllDomains with active and inactive domains. For each domain, read the metadata element; keep those with managed true and build the Instance from the domain name and the element alone: ID, UID and CreatedAt, which is all GC reads (contract §2.4). State and IP are left empty; no state or address RPC is made. A domain whose metadata cannot be read (mid teardown, corrupt element) is skipped with a V(1) line, unless ctx ended, which fails the call; a host that cannot be dialled or whose listing fails fails the whole call, so GC skips this provider for the sweep rather than act on a partial view. Filtering is client-side; libvirt has no label index. Cost: exactly one DomainGetMetadata per domain per sweep, about half a millisecond each over the tunnel.

Evidence: execution log run 6 (per-domain metadata cost); contract §2.4 and §4.7.

13. Error classification (not-found, quota)

Answer: not-found: error-code. quota: detected, from a pool-space precheck rather than a backend error: before creating any volume, Create reads StoragePoolGetInfo and compares available with the bytes it is about to commit: the image's Content-Length when the base is not cached, plus diskSizeGb in bytes, plus 16 MiB for the seed. Below that it returns ErrQuotaExceeded naming the pool, available and needed, and the reconciler waits QuotaRetry.

One classify(op string, err error):

Backend response	Class	Why
libvirt.Error code ErrNoDomain, ErrNoStorageVol	NotFound (Transient when op == "create")	the object is gone; inside Create it vanished mid-call
ErrNoStoragePool	Permanent	config names a pool that does not exist
ErrXMLError, ErrNoSupport, ErrInvalidArg	Permanent	the request itself is wrong
ErrAuthFailed, ErrAccessDenied	Permanent	the SSH user may not use the socket
ErrOperationInvalid	Transient (Delete and Create treat the specific cases as success before classify)	state race; the next poll decides
any other libvirt.Error	Transient	unknown means retry
SSH: ssh: unable to authenticate (from x/crypto/ssh)	Permanent	the key is not in authorized_keys
SSH: knownhosts.KeyError with a known key that differs, or with no known key	Permanent	host key changed or missing from known_hosts; the admin must act
SSH key or known_hosts file missing (fs.ErrNotExist)	Transient	the Secret has not landed yet
SSH key present but does not parse	Permanent	wrong file in the Secret
context.Canceled, context.DeadlineExceeded, net.OpError, io.EOF, io.ErrUnexpectedEOF	Transient	network or cancellation
HTTP 404, 403, 410 on the image URL; missing Content-Length; sha256 mismatch	Permanent	the catalogue entry is wrong
HTTP 5xx, 429, network errors on the image URL	Transient	retry
an error already wrapped with a class (errors.Is any sentinel)	that class	validation and builders classify at the source
anything else	Transient	contract default

The unit test table uses a libvirt.Error per code, the real SSH error strings produced by x/crypto/ssh against the in-process server, and a real knownhosts.KeyError.

Evidence: go-libvirt ErrorNumber constants and Error{Code uint32, Message} (value receiver, so errors.As uses a libvirt.Error value); contract §5; kvm-provider.md "Known gaps" (SSH auth retried forever in v1: closed here).

14. Credentials and TLS (credentials)

Answer: mounted-files. The config block names sshKeyFile (private key, OpenSSH or PEM, parsed with ssh.ParsePrivateKey) and knownHostsFile (OpenSSH known_hosts, parsed with knownhosts.New), both expected under the existing optional kvm-ssh Secret mount at /etc/proxy-operator/kvm (config/manager/manager.yaml; the mount is shared with the kvm provider and its comment says so). insecureIgnoreHostKey: true replaces known_hosts with ssh.InsecureIgnoreHostKey() and is logged as a warning by New. Both files are read on every dial, so rotating the Secret takes effect on the next connection without a restart. New tries to read them once to log a warning when they are unusable, but never fails on them and never dials.

Minimum privilege: the SSH user must be able to open the libvirt socket (member of the libvirt group); libvirt itself is used as qemu:///system through the socket, so no further host permissions.

TLS: none (SSH provides transport security); system-roots for the image download's HTTPS, which uses the distroless image's CA bundle.

Hygiene: the private key bytes and the known_hosts content never appear in logs, spans or error strings; errors from parsing say "parsing ssh key file " plus the parser's message. Tested with a sentinel placed in a bad key file.

Evidence: kvm-provider.md "Cluster-side setup"; execution log run 8 (lazy load: a missing Secret must not abort the provider set); contract §7.

15. Config block

Answer: the block is kvm2: and reuses the Go type provider.KVMConfig from internal/provider/config_kvm.go unchanged, so an existing kvm-home entry ports by changing type: kvm to type: kvm2 and kvm: to kvm2:. Validation calls the shared validator with the block name kvm2 so messages read kvm2.hosts and not kvm.hosts.

Field	Required	Default	Validation	Ends up in
sshKeyFile	yes		non-empty	dial
knownHostsFile	unless insecureIgnoreHostKey		non-empty	dial
insecureIgnoreHostKey		false		dial, warning at New
diskSizeGb		10	not negative	overlay capacity, quota precheck
vendorData				seed vendor-data
installGuestAgent		true		seed vendor-data
images.<key>			key matches kvmIdentifier	volume name base-<entry>-<key>.<format>
images.<key>.url	yes per entry		http(s) URL	download
images.<key>.sha256			64 hex	marker comparison
images.<key>.format		qcow2	qcow2 or raw	base volume XML, overlay backing format
images.<key>.firmware		uefi	uefi or bios	domain XML <os>
machineTypes.<key>	at least one		vcpus and memoryMiB greater than zero	domain XML
hosts[].name	yes		unique, matches kvmIdentifier	providerID segment
hosts[].address	yes		non-empty host:port	TCP dial
hosts[].user	yes		non-empty	SSH
hosts[].socket		/var/run/libvirt/libvirt-sock		SSH channel target
hosts[].pool		default		every storage RPC
hosts[].bridge	yes		non-empty	domain XML NIC

Wiring: ProviderConfig.KVM2 *KVMConfig with tag kvm2,omitempty; setBlocks reports kvm2; the allow-list in Config.validate and the constructor map in cmd/main.go both gain kvm2 in the same commit.

Evidence: KVMConfig, validateKVM, kvmIdentifier in internal/provider/config_kvm.go; contract §10.

16. Checker

Answer: one target per hosts[] entry, Name the host name, Address the SSH address, checked concurrently in config order on a fresh dedicated connection each time (dial, ConnectGetLibVersion, ConnectGetHostname, StoragePoolLookupByName, StoragePoolGetInfo, close). It never touches the pooled connection, so a check that runs out of time closes only its own connection and cannot disrupt a reconcile. Cost: one SSH handshake and four cheap RPCs per host per round.

Reasons: Reachable; Timeout (the round's ctx ended before the check finished, whichever stage was running, the dial included, or an RPC's own budget ran out once the connection was up); Unreachable (TCP connect or the SSH transport failed, including on the dial's own 10 s budget while the round still had time, or the link died mid-check); Unauthorized (SSH authentication refused, host key unknown or changed); Forbidden (ErrAccessDenied or ErrAuthFailed from libvirt: the socket refused the user); APIError (any other libvirt error); CredentialsUnavailable (the key or known_hosts file is missing or unparsable, i.e. the Secret is absent or wrong: fixed in the cluster, not on the host); PoolNotFound; PoolNotActive.

Details on success: user, hostname, libvirtVersion (decoded major.minor.micro), pool, poolState, poolCapacityGiB, poolAvailableGiB (the closest thing to a capacity signal, and the same number the quota precheck reads). The status.type comment in api/v1alpha1/providerhealth_types.go gains kvm2 and the CRD is regenerated.

Evidence: contract §8; StoragePoolGetInfo returning state, capacity, allocation, available; ConnectGetLibVersion returning major1e6 + minor1e3

• micro.

17. Client and child spans (client-kind, child-spans)

Answer: client-kind: rpc-over-ssh. child-spans: span-decorator.

The seam is an interface over exactly the go-libvirt methods the provider uses, with ctx as the first parameter of every method (go-libvirt's own methods take none and block): DomainLookupByName, DomainDefineXML, DomainCreate, DomainDestroy, DomainUndefineFlags, DomainGetState, DomainGetMetadata, DomainInterfaceAddresses, ConnectListAllDomains, StoragePoolLookupByName, StoragePoolRefresh, StoragePoolGetInfo, StorageVolLookupByName, StorageVolGetPath, StorageVolCreateXML, StorageVolUpload, StorageVolDownload, StorageVolDelete, ConnectGetLibVersion, ConnectGetHostname, plus Close. One production adapter wraps *libvirt.Libvirt and applies the per-call deadline (entry 18); one in-memory fake implements the seam for unit tests. A dialer seam (dial(ctx, host) (backend, error)) separates the SSH transport so tests inject fakes and an in-process SSH server test covers the real dialer.

The decorator is applied once, inside the real dialer, at the point where a connection is created, so provider code never sees an undecorated backend. Per call it starts a span kvm2.<Method> (for example kvm2.DomainDefineXML) with attributes kvm2.host and the identifiers passed (kvm2.domain, kvm2.pool, kvm2.volume), records the libvirt error code on failure, and inside the span writes the V(1) log line, so the line carries that span's ID. Two more spans: kvm2.dial around the SSH handshake and libvirt connect (attributes address and user) and kvm2.download around an image fetch (attributes URL host, bytes, whether a sha256 was configured). All spans are children of the decorator's provider.create, provider.get, provider.delete, provider.list or the monitor's providerhealth.check. No CloudInit, key material or upload bodies in attributes. A reflection test walks the seam interface and asserts every method produces exactly one span and one log line.

Evidence: contract §4.15, §4.18, §9; tracing.Start in internal/tracing/logger.go.

18. Timeouts (timeouts)

Answer: per-call. The adapter runs each blocking go-libvirt call in a goroutine and waits on the result, ctx.Done() and a timer:

Call	Budget
TCP connect plus SSH handshake plus libvirt connect	10 s
ordinary RPC	30 s
DomainCreate	90 s
StorageVolUpload (seed)	60 s
StorageVolUpload (image), HTTP download	ctx, capped at 20 m
wait for running after DomainCreate	15 s (entry 10)
HTTP client for images	no client timeout; the per-request ctx carries the cap

When a call is abandoned (ctx cancelled or budget exceeded), the adapter closes the underlying connection, because that is the only way to unblock go-libvirt, and marks it dead so the next call redials; the abandoned call returns ErrTransient wrapping the ctx or deadline error. Consequence, documented in the guide: other RPCs in flight on that host's pooled connection fail once with Transient and their callers retry. The image download uses a dedicated connection so the expensive case (a cancelled poll killing a 435 MB upload) cannot happen. The keepalive (keepalive@openssh.com request every 15 s) marks the connection dead on failure.

Evidence: contract §1.5 (no caller deadline; a hung ListByTag stalls GC), §4.16; go-libvirt methods take no ctx.

19. Concurrency

Answer: shared state is the host connection table (one pooled connection per host, dialled lazily, under a mutex with a per-host dial mutex so concurrent callers dial once and share the result), the per-(host, volume) download locks, and a lock per instance name. Both lock sets are one instance each of keyedLocks: a one-slot semaphore per key, so a waiter gives up when its ctx ends, reference-counted so an entry is removed from the map once no caller holds or waits for it and the map does not grow with every name ever seen, while two waiters on one key always share one semaphore. Create and Delete for the same name run under that name's lock, so two Deletes (GC and the reconciler) never race on destroy and undefine, and a Delete cannot interleave with a Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes RPCs on one connection, so calls for different names proceed concurrently on the shared connection. Three reconciles, a GC sweep and a check round at once are safe; the check uses its own connections.

Evidence: contract §1.2 (concurrency), §4.15.

20. Logging (logging)

Answer: tiered. Logger from logf.FromContext(ctx) on every call, named kvm2, with provider on every line and host and name on every line about an instance.

• V(1): one line per operation outcome (kvm2 domain ensured running, kvm2 domain fetched with state, ip and uid, kvm2 domain deleted, kvm2 domains listed with counts), one libvirt call line per RPC from the decorator (rpc, identifiers, duration, on failure error, libvirtCode and libvirtError name), connection events (dialled, dropped with reason), and a line per failed backend call with the raw error.
• V(2): the same lines also carry domain and volume XML, the metadata element, interface addresses, listed domain names, and the resolved Create parameters (cloudInitBytes, never the content).
• No V(5) wire tier; go-libvirt exposes no hook for it.
• Never logged at any level: CloudInit, key material, known_hosts content, upload or download bodies.

Evidence: contract §4.17, §9.

21. Tests (backend-test)

Answer: backend-test: integration-tag. Unit tests run over the in-memory fake behind the seam (and an in-process x/crypto/ssh server for the dialer); the integration test is kvm2_integration_test.go behind //go:build integration, skipped unless KVM2_TEST_HOST, KVM2_TEST_USER, KVM2_TEST_SSH_KEY, KVM2_TEST_BRIDGE and KVM2_TEST_IMAGE_URL are set, optional KVM2_TEST_KNOWN_HOSTS (without it host keys are not verified and the log says so), KVM2_TEST_IMAGE_SHA256, KVM2_TEST_POOL, KVM2_TEST_SOCKET, KVM2_TEST_TIMEOUT (default 20m), KVM2_TEST_KEEP, KVM2_TEST_LOG_LEVEL; run with make test-integration-kvm2. It asserts, in order: New returns without dialling; Check reports the host Ready with details; Create returns hosts/<host>/domains/<name>; a second Create returns the same ID and the host's domain count is unchanged; Get reaches Running with an IPv4 within the budget and its UID and CreatedAt equal what Create wrote; ListByTag contains it; Delete returns nil; Get is ErrNotFound; Delete again returns nil; the pool holds the base image and its marker and no <name>.qcow2 or <name>-seed.iso; with an in-memory span recorder, kvm2.DomainDefineXML and kvm2.dial spans are children of provider.create.

Contract §11 list, with the test that covers each item:

• repeat Create: TestCreate_repeatReturnsSameIDAndCreatesNothing
• Get after Delete: TestGet_afterDeleteReturnsNotFound
• Delete twice: TestDelete_twiceSucceeds
• malformed ID: TestGet_malformedIDIsPermanent, TestDelete_malformedIDIsPermanent, TestParseProviderID_malformed
• unknown state maps to StateStopped: TestMapState_table
• StateRunning requires an IP: TestGet_runningWithoutIPIsProvisioning, TestGet_runningWithIPIsRunning
• UID and CreatedAt read back: TestGet_readsUIDAndCreatedAtFromBackend
• ListByTag excludes unmarked instances: TestListByTag_excludesUnmarkedDomains, TestListByTag_emptyUIDStillListed
• Get during an in-progress delete: TestGet_duringDeleteStillReturnsInstance
• Get during an asynchronous operation or lock: TestGet_backendBusyIsTransientNeverNotFound
• never started is never StateStopped: TestCreate_returnsOnlyWhenRunning
• cancelled ctx returns promptly: TestProvider_cancelledContextReturnsPromptly, TestCall_cancelUnblocksRPC
• classify table with captured responses: TestClassify_table
• Checker reasons, expired ctx, stable order: TestCheck_reasonsTable, TestCheck_expiredContextReturnsPromptly, TestCheck_oneEntryPerHostInConfigOrder
• sentinel tests for CloudInit and the credential: TestCreate_cloudInitSentinelNeverLoggedOrInErrors, TestSSH_keyMaterialNeverInErrorsOrLogs
• child spans under the operation span: TestObserve_childSpansUnderParent, TestObserve_everySeamMethodInstrumented
• real-backend lifecycle test: TestKVM2_lifecycle and TestKVM2Check_realHost in kvm2_integration_test.go

Further tests per file are listed in the plan (docs/plans/2026-09-16-1333-kvm2-provider.md).

Evidence: contract §11; docs/testing.md conventions.

22. Deployment

Answer: the existing kvm-ssh mount is reused; the rest is samples and a Makefile target:

• config/manager/manager.yaml: no new mount; the existing optional kvm-ssh Secret at /etc/proxy-operator/kvm serves both kvm and kvm2, and its comment says so. /tmp is already an emptyDir.
• config/manager/providers_config.yaml and config/samples/providers-config.yaml: a commented kvm2-home entry, the same block as kvm-home under kvm2:.
• config/samples/proxy_kvm2.yaml, listed in config/samples/kustomization.yaml: provider kvm2-home, zone beelink (matching the shipped config, unlike the kvm sample's nas1), machine type small, image debian-13, a cloud-init that installs Squid.
• Makefile: test-integration-kvm2 (go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM2 ./internal/provider/kvm2/). The Secret helper make kvm-ssh-secret already exists and is reused.
• RBAC: none; the provider does not talk to the cluster's API.
• hack/providers-dev.yaml: no entry (needs credentials).

Evidence: contract §12; config/manager/manager.yaml.

23. Docs

Answer: docs/kvm2-provider.md (hypervisor prerequisites including OVMF and the libvirt group and a bridge with DHCP; cluster-side setup reusing the kvm-ssh Secret; config reference; placement mapping; what lands on the host including the metadata namespace; the image cache and the quota precheck; the health check reason table; logging tiers and span names; migrating an entry from kvm, which requires deleting its Proxies first because kvm2 does not adopt v1 domains; known gaps). A fourth column in the three tables of docs/providers-existing.md and a kvm2 section. docs/architecture.md: components row, a kvm2 block in §5, the span topology in §10, a §11 row, and Decisions bullets (dedicated download connection, checks on fresh connections, own metadata namespace, quota precheck, SSH auth failures Permanent). A README row. docs/testing.md rows and a kvm2 integration section. A CHANGELOG.md entry after the user confirms.

Evidence: contract §12.

24. Known limits accepted at design time

Answer: nine limits, each with its workaround:

• Create blocks, including the first image download per host, holding one of three reconcile workers. Workaround: warm the cache with one Proxy per image per host before creating many.
• Volumes without a domain are invisible to GC (crash between volume creation and define). The same Proxy's next reconcile removes them; a Proxy deleted for good leaves them for virsh vol-list.
• A host removed from the config orphans its VMs: Delete is a no-op success for it and GC does not sweep it. Delete the Proxies first.
• No adoption of kvm (v1) domains: switching an entry's type requires deleting its Proxies first, or the old domains stay behind.
• Cancelling one call drops the host's pooled connection, so other RPCs in flight on that host fail once with Transient and retry. The download is isolated on its own connection.
• Content-Length is required for image downloads; a chunked mirror is rejected as Permanent.
• A paused or pmsuspended domain is not started. A repeat Create for a domain in that state is ErrPermanent (entry 10), so the Proxy latches as Failed until its spec changes, rather than retrying a DomainCreate that libvirt refuses. Resume the domain on the host (virsh resume) and touch the spec, or delete the Proxy.
• No v1 log-format compatibility: the libvirt call lines have the same keys where the same facts exist, but dashboards keyed on kvm logger names need kvm2.
• Contract §13 limits that apply unchanged: config read once per process (adding a host needs a restart; credentials do not), no installation identity in GC, an unconfigured provider latches the Proxy.

Evidence: contract §13; entries 4, 8, 11, 18 above.