egress-proxies-operator/docs/plans/2026-09-16-1333-kvm2-provider.md

Plan: kvm2, a clean-room libvirt/KVM provider built from the provider contract

Created: 2026-09-16 13:33

Context

The provider documentation was just split into a generic contract (docs/provider-contract.md), a spec template (docs/provider-spec-template.md) and an appendix on the existing providers (docs/providers-existing.md), so that a new provider can be specified and built from those files alone. Before using them for Proxmox, the user wants a dry run on a known backend: rebuild the libvirt/KVM provider from scratch as a second type, kvm2, following the documents, without touching or reading the existing internal/provider/kvm/ code. The result is a validated workflow (spec first, then implementation by fresh agents against the contract) and a v2 provider that can replace v1 once it passes on the real hypervisor.

User decisions:

• Placement: new package internal/provider/kvm2, type kvm2, config block kvm2:, side by side with kvm. The Go config struct reuses provider.KVMConfig (internal/provider/config_kvm.go is shared code, not the kvm package), so an entry ports by changing type and the block name.
• Clean room: implementers never open, grep or import internal/provider/kvm/. Docs (docs/kvm-provider.md) and the operational lessons from the first build's execution log are allowed.
• Implementer: the orchestrator (this session) writes the spec and the step briefs; each implementation step runs in a fresh general-purpose subagent given only the three docs, the filled spec, the backend facts and its brief. The orchestrator reviews each result against the contract, runs the tests and commits.
• Profiles: create-return blocking, provisioning-state create-blocks, child-spans span-decorator, quota detected (pool free space precheck), credentials mounted-files (the existing kvm-ssh Secret), client-kind rpc-over-ssh, gc-mark written-metadata, created-at written-at-create, ip-source discovered (agent, then ARP), list-partial fail-whole per host with unreadable domains skipped, not-found error-code, delete-return waits, secondary-resources explicit, timeouts per-call, logging tiered, backend-test integration-tag.

Backend facts the spec carries (allowed lessons)

• Real host beelink: libvirt 11.6, QEMU 10.1, pool default at /srv/vms/images, bridge br0, beelink.tail8c69c.ts.net:22, user novakj, key ~/.ssh/proxy-operator_ed25519, known_hosts ~/.ssh/kvm_known_hosts. The Debian trixie qcow2 is about 435 MB.
• AppArmor confines QEMU from the domain XML: only <disk type="file"> with a <source file> path gets a rule, so volume paths are resolved with StorageVolGetPath and emitted as file disks.
• Debian 13 hangs in GRUB under BIOS with a serial console and no video: UEFI by default (<os firmware="efi">, secure boot feature disabled), firmware: bios per image; OVMF is a host prerequisite; undefine needs the NVRAM flag.
• libvirt hands the metadata element back without its namespace declaration: parse by local name.
• Vendor-data must be a shell script, not cloud-config, to survive cloud-init's merge with user-data; it installs qemu-guest-agent. The agent answered after about 20 s where ARP never did.
• Read-only rootfs: iso9660.NewWriter needs a temp dir under /tmp (64Mi).
• go-libvirt (github.com/digitalocean/go-libvirt, in go.mod): every method blocks and takes no ctx; libvirt.Error{Code uint32, Message} has a value receiver; needed ErrorNumber constants exist (ErrNoDomain, ErrNoStorageVol, ErrNoStoragePool, ErrOperationInvalid, ErrNoDomainMetadata, ErrStorageVolExist, ErrXMLError, ErrNoSupport, ErrInvalidArg, ErrAuthFailed, ErrAccessDenied). libvirt.New(net.Conn) then Connect(). No new dependencies: x/crypto/ssh, kdomanski/iso9660 are already direct requires; use slices, never sort (depguard).

Design decisions recorded in the spec (step 1)

• Package layout (internal/provider/kvm2/): kvm2.go (type, New, logger, per-name keyed lock), id.go, state.go, errors.go, xml.go, seed.go, backend.go (ctx-first seam over the 20 go-libvirt calls, adapter with a per-RPC deadline), ssh.go, hosts.go (one cached connection per host, lazy dial, dialer seam), observe.go (the single decorator: one span kvm2.<Call> per RPC plus the V(1)/V(2) log line), image.go, create.go, get.go, delete.go, list.go, check.go, plus fake_test.go, sshserver_test.go, kvm2_integration_test.go. Names are suggestions for the briefs; the implementer may choose others by responsibility.
• Seam: every seam method takes ctx; the adapter runs the blocking call in a goroutine and selects on result, ctx.Done() and a timer (dial 10s, RPC 30s, DomainCreate 90s, upload/download ctx plus a 20m cap). On abandon it closes the connection and marks it dead; the next call redials. The image download runs on a dedicated connection so a cancelled poll cannot cut it off.
• Decorator wrapped once: applied inside the real dialer where a connection is born, so provider code never sees an undecorated backend. Spans kvm2.dial (in the dialer) and kvm2.download (in image.go) are the two non-RPC spans. A reflection test asserts every seam method is instrumented.
• Metadata: own namespace https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2, element <instance> with managed, uid, createdAt. kvm2 never lists or adopts v1 domains; a kvm and a kvm2 entry may share a host.
• Adoption guard: an existing domain whose metadata uid is non-empty and differs from req.UID makes Create return ErrPermanent.
• Quota precheck: before creating volumes, StoragePoolGetInfo; needed = image Content-Length (if not cached) + diskSizeGb + 16 MiB; below that returns ErrQuotaExceeded naming pool, available and needed.
• Download requires Content-Length (Permanent otherwise), streams through sha256 into StorageVolUpload, refreshes the pool, writes the marker volume last; a failed attempt deletes the partial volume.
• Classify: ErrNoDomain, ErrNoStorageVol NotFound, but Transient inside Create; ErrNoStoragePool, ErrXMLError, ErrNoSupport, ErrInvalidArg, ErrAuthFailed, ErrAccessDenied Permanent; SSH authentication failures and host-key mismatches or unknown host keys Permanent; missing key or known_hosts file Transient (Secret pending), unparsable key Permanent; cancellation, deadlines, net.OpError, io.EOF and anything else Transient.
• Same-name concurrency: a keyed mutex per instance name serialises Create and Delete for one name.
• State: nostate Provisioning; running or blocked without IP Provisioning, with IP Running; everything else, including unknown codes, Stopped. IP from the guest agent first, then ARP, first non-loopback non-link-local IPv4.
• Checker: per host concurrently on a fresh dedicated connection (dial, ConnectGetLibVersion, ConnectGetHostname, pool lookup and info, close), so it never touches the pooled connection. Reasons: shared ones plus PoolNotFound, PoolNotActive, CredentialsUnavailable. Details user, hostname, libvirtVersion, pool, poolState, poolCapacityGiB, poolAvailableGiB.
• Rotation: SSH material is read on every dial, so Secret rotation needs no restart.
• Config validation messages: validateKVM is refactored into validateKVMBlock(prefix, block string, c *KVMConfig) so messages say kvm2. for the new block; validateKVM stays as a wrapper for kvm.

Steps (one commit each, on feat/kvm2-provider off main)

PR #7 is merged, so main carries the three docs. Step 0 begins with git fetch origin && git switch main && git pull --ff-only.

Every subagent brief opens and closes with: "Do not open, grep, list or import anything under internal/provider/kvm/. If a detail seems to need that code, stop and report the question instead." Each brief names the files it may create or edit and nothing else, lists the inputs (the three docs, the spec, docs/kvm-provider.md for backend facts, CLAUDE.md conventions), the acceptance tests by name and the commands to run, and asks the agent to end with "files read outside the allowed set: none".

• Step 0 (orchestrator): copy this plan to docs/plans/<ts>-kvm2-provider.md with the real timestamp, commit alone, create the branch. Execution summary docs/plans-executions/<ts>-kvm2-provider.md with the checklist after step 1.
• Step 1 (orchestrator): the spec. Copy docs/provider-spec-template.md to docs/kvm2-provider-spec.md and answer all 24 entries with the profiles and decisions above; every Evidence line cites the user guide, a go-libvirt symbol, or "verified on beelink in step 11". Entry 21's checklist lists the concrete test names from steps 4 to 7. Verify: no blank Answer lines, every axis code used matches contract §4.20, no reference to internal/provider/kvm/.
• Step 2 (subagent): config wiring, IDs, state, classify. Edits internal/provider/config.go (KVM2 *KVMConfig field, setBlocks, allow-list, dispatch), config_kvm.go (validateKVMBlock), config_test.go and config_kvm_test.go rows. Creates kvm2.go (type, assertions, New that never dials and warns on unreadable SSH files, stub methods), id.go, state.go, errors.go with tests: TestFormatProviderID_roundTrip, TestParseProviderID_malformed, TestMapState_table, TestFirstIPv4_filters, TestClassify_table. Also registers "kvm2": kvm2.New in cmd/main.go so the two type lists change in one commit. Verify: three "kvm2" sites in config.go, no import of provider/kvm, go test -race ./internal/provider/... && go vet ./....
• Step 3 (subagent): pure builders. xml.go, seed.go and tests: TestDomainXML_fileDisksWithResolvedPaths, TestDomainXML_uefiDefaultSecureBootOff, TestDomainXML_biosEscape, TestDomainXML_metadataElement, TestDomainXML_guestAgentChannel, TestParseMetadata_withNamespace, TestParseMetadata_withoutNamespaceDeclaration, TestParseMetadata_absentGivesEmptyUID, TestVolumeXML_overlayBackingStore, TestSeedISO_filesAndLabel, TestSeedISO_userDataByteForByte, TestSeedISO_tempDirRemoved, TestVendorData_isShellScriptNotCloudConfig, TestVendorData_multipartWhenExtraProvided. Verify: no type="volume" in xml.go; temp dir under /tmp with a deferred removal.
• Step 4 (subagent): transport, seam, decorator, fakes. backend.go, ssh.go, hosts.go, observe.go, fake_test.go, sshserver_test.go and tests: TestDial_forwardsUnixSocket, TestDial_unknownHostKeyRejected, TestDial_mismatchedHostKeyRejected, TestDial_insecureIgnoreHostKeyAccepts, TestDial_missingKeyFileIsTransient, TestDial_unparsableKeyIsPermanent, TestDial_handshakeTimeout, TestCall_deadlineClosesConnectionAndReturnsTransient, TestCall_cancelUnblocksRPC, TestHosts_oneDialPerHostUnderConcurrency, TestHosts_redialsAfterDrop, TestHosts_unconfiguredHostIsPermanent, TestObserve_everySeamMethodInstrumented, TestObserve_childSpansUnderParent, TestObserve_logsOneLinePerCallAtV1, TestObserve_detailOnlyAtV2, TestObserve_uploadBodyNeverLogged, TestSSH_keyMaterialNeverInErrorsOrLogs. Verify: no context.Background() outside tests and the keepalive goroutine; tracing.Start in the decorator and the dialer; go test -race -count=2.
• Step 5 (subagent): image cache and pool-space precheck. image.go and tests: TestEnsureImage_downloadsAndWritesMarkerLast, TestEnsureImage_cachedWithMarkerSkipsDownload, TestEnsureImage_missingMarkerRedownloads, TestEnsureImage_sha256ChangeRedownloads, TestEnsureImage_sha256MismatchIsPermanentAndRemovesVolume, TestEnsureImage_noContentLengthIsPermanent, TestEnsureImage_http404IsPermanent, TestEnsureImage_http503IsTransient, TestEnsureImage_concurrentCallersFetchOnce, TestEnsureImage_cancelledMidStreamReturnsPromptly, TestEnsureImage_streamsWithoutBuffering, TestCheckPoolSpace_belowNeededIsQuota. Verify: no io.ReadAll or bytes.Buffer on the download path; download span.
• Step 6 (subagent): the four contract methods. create.go, get.go, delete.go, list.go, kvm2_test.go, finishing kvm2.go. Tests map 1:1 to contract §11: TestCreate_repeatReturnsSameIDAndCreatesNothing, TestCreate_resumesAfterEachPartialStep, TestCreate_returnsOnlyWhenRunning, TestCreate_rejectsUnknownPlacement, TestCreate_adoptionWithForeignUIDIsPermanent, TestCreate_poolOutOfSpaceIsQuotaExceeded, TestCreate_libvirtNotFoundIsTransientNotNotFound, TestCreate_cloudInitSentinelNeverLoggedOrInErrors, TestGet_afterDeleteReturnsNotFound, TestGet_duringDeleteStillReturnsInstance, TestGet_backendBusyIsTransientNeverNotFound, TestGet_runningWithoutIPIsProvisioning, TestGet_runningWithIPIsRunning, TestGet_agentPreferredOverARP, TestGet_addressErrorIsProvisioning, TestGet_readsUIDAndCreatedAtFromBackend, TestGet_malformedIDIsPermanent, TestGet_unconfiguredHostIsPermanent, TestDelete_malformedIDIsPermanent, TestDelete_twiceSucceeds, TestDelete_removesOverlayAndSeed, TestDelete_runningDomainDestroyedFirst, TestDelete_toleratesMissingVolumesAndPool, TestDelete_unconfiguredHostIsNoOp, TestDelete_concurrentSameNameSafe, TestListByTag_excludesUnmarkedDomains, TestListByTag_emptyUIDStillListed, TestListByTag_acrossHostsInConfigOrder, TestListByTag_unreachableHostFailsWhole, TestListByTag_unreadableDomainSkipped, TestProvider_cancelledContextReturnsPromptly, TestProvider_everyErrorIsWrappedWithConfiguredName, TestLogging_v0SilentV1OutcomeV2Detail. Verify: CloudInit appears in create.go only as len(req.CloudInit); stub strings gone; Wrap always gets the configured name; go test -race -count=2 ./internal/provider/kvm2/.
• Step 7 (subagent): Checker and the CRD comment. check.go, check_test.go: TestCheck_oneEntryPerHostInConfigOrder, TestCheck_hostsCheckedConcurrently, TestCheck_expiredContextReturnsPromptly, TestCheck_reasonsTable, TestCheck_neverUsesPooledConnection, TestCheck_detailsKeys. Edit api/v1alpha1/providerhealth_types.go (status.type comment) and run make manifests. Verify: only the CRD description line changes under config/; go test ./internal/providerhealth/.
• Step 8 (subagent): deployment, Makefile, integration test. Edit config/manager/manager.yaml (comment: the kvm-ssh mount serves kvm and kvm2), config/manager/providers_config.yaml and config/samples/providers-config.yaml (commented kvm2 entry), config/samples/kustomization.yaml, Makefile (test-integration-kvm2, -run TestKVM2, -timeout 40m), docs/testing.md (suite rows and a kvm2 integration section). Create config/samples/proxy_kvm2.yaml (zone beelink, provider kvm2-home) and kvm2_integration_test.go (KVM2_TEST_HOST, _USER, _SSH_KEY, _KNOWN_HOSTS, _BRIDGE, _IMAGE_URL; optional _IMAGE_SHA256, _POOL, _SOCKET, _TIMEOUT, _KEEP, _LOG_LEVEL) asserting in order: New does not dial; Check Ready with details; Create returns hosts/<host>/domains/<name>; second Create same ID, domain count unchanged; Get reaches Running with an IPv4 and the written UID and CreatedAt; ListByTag contains it; Delete nil; Get NotFound; Delete again nil; pool holds base and marker, no per-VM volumes; with an in-memory span recorder, kvm2.DomainDefineXML and kvm2.dial spans are children of provider.create. Verify: go vet -tags=integration ./... && go vet -tags=e2e ./..., bin/kustomize build config/default and config/samples parse, make -n test-integration-kvm2.
• Step 9 (subagent): documentation. Create docs/kvm2-provider.md (prerequisites incl. OVMF, cluster setup reusing kvm-ssh, config reference pointing at the shared KVMConfig table, placement mapping and the add-a-key-never-edit warning, what lands on the host incl. the metadata namespace, image cache and quota precheck, health-check reasons, logging and span names, migration from kvm, known gaps). Edit docs/providers-existing.md (fourth column in the three tables and a kvm2 section), docs/architecture.md (components row, §5 block, §10 span topology, §11 row, Decisions bullets), README.md row, docs/providers.md guide link. Verify with the link and symbol sweep from the docs-split execution log; no em-dashes in new prose.
• Step 10 (orchestrator): milestone review. go-operator-reviewer on the branch with the spec and contract as inputs, then /code-review high, make lint, make test (full, with envtest). Fix accepted findings with regression tests in one commit; record rejected ones in the summary.
• Step 11 (orchestrator with the user): real-host validation, changelog, MR. Hand the user a host checklist (leftover proxy-* domains and volumes from earlier runs, OVMF installed, pool running with free space, br0 up, the known_hosts entry) and the KVM2_TEST_* export lines; do not run remote probes unprompted. With the user's go-ahead run make test-integration-kvm2 (a permission prompt is expected; the allow-list entry make test-integration-kvm * does not match) and, if wanted, the in-cluster pass on kind: build and load the image, deploy, a scratch ConfigMap with the kvm2-home entry, apply proxy_kvm2.yaml, wait for Ready, kubectl get ph kvm2-home -o yaml, a curl -x through the proxy, delete and confirm the domain and volumes are gone; scratch overrides are not committed. Record cold download duration and time to IP. CHANGELOG entry after the user confirms; push with the token helper; MR via the Gitea API with base main; never merge.

Verification

• Per step: the checks listed above, plus go build ./... && go test -short ./... before every commit and git diff --stat reviewed for files outside the step's allowed set.
• Clean-room check on every step's diff: grep -rn 'provider/kvm"' internal/provider/kvm2/ empty; the agent's closing line "files read outside the allowed set: none"; identifiers that coincide with v1's names from the appendix (libvirtAPI, hostPool, imageCache, tracingAPI, lazySSH) are treated as a smell and questioned.
• Contract mapping: after step 7, every rule of contract §2 and §8 has a named test in the package; the §11 list maps 1:1 to test names (recorded in the spec's entry 21).
• End to end: the integration run's ordered assertions (step 8) on beelink, and the optional kind pass (step 11).

Risks

• Subagent scope creep into internal/provider/kvm/: instruction-only guard, mitigated by the closing attestation and the diff greps.
• .claude/settings.json is locally modified and uncommitted; it stays out of feature commits, and the permission prompt for the new make target is accepted rather than pre-approved.
• Content-Length is required for downloads; the Debian and Ubuntu mirrors send it.
• The cancelled-call policy still drops other in-flight RPCs on that host once; the dedicated download connection removes the expensive case and the spec states the rest.
• go-operator-reviewer hunts over-abstraction; the seam, the decorator and the dialer interface each have one production implementation and one fake, which the contract requires (§4.15, §11).