302 lines
19 KiB
Markdown
302 lines
19 KiB
Markdown
# Plan: kvm2, a clean-room libvirt/KVM provider built from the provider contract
|
|
|
|
**Created:** 2026-09-16 13:33
|
|
|
|
## Context
|
|
|
|
The provider documentation was just split into a generic contract
|
|
(`docs/provider-contract.md`), a spec template (`docs/provider-spec-template.md`)
|
|
and an appendix on the existing providers (`docs/providers-existing.md`), so that
|
|
a new provider can be specified and built from those files alone. Before using
|
|
them for Proxmox, the user wants a dry run on a known backend: rebuild the
|
|
libvirt/KVM provider from scratch as a second type, `kvm2`, following the
|
|
documents, without touching or reading the existing `internal/provider/kvm/`
|
|
code. The result is a validated workflow (spec first, then implementation by
|
|
fresh agents against the contract) and a v2 provider that can replace v1 once it
|
|
passes on the real hypervisor.
|
|
|
|
User decisions:
|
|
|
|
- **Placement:** new package `internal/provider/kvm2`, type `kvm2`, config block
|
|
`kvm2:`, side by side with `kvm`. The Go config struct reuses
|
|
`provider.KVMConfig` (`internal/provider/config_kvm.go` is shared code, not
|
|
the kvm package), so an entry ports by changing `type` and the block name.
|
|
- **Clean room:** implementers never open, grep or import
|
|
`internal/provider/kvm/`. Docs (`docs/kvm-provider.md`) and the operational
|
|
lessons from the first build's execution log are allowed.
|
|
- **Implementer:** the orchestrator (this session) writes the spec and the step
|
|
briefs; each implementation step runs in a fresh general-purpose subagent
|
|
given only the three docs, the filled spec, the backend facts and its brief.
|
|
The orchestrator reviews each result against the contract, runs the tests and
|
|
commits.
|
|
- **Profiles:** `create-return` blocking, `provisioning-state` create-blocks,
|
|
`child-spans` span-decorator, `quota` detected (pool free space precheck),
|
|
`credentials` mounted-files (the existing `kvm-ssh` Secret), `client-kind`
|
|
rpc-over-ssh, `gc-mark` written-metadata, `created-at` written-at-create,
|
|
`ip-source` discovered (agent, then ARP), `list-partial` fail-whole per host
|
|
with unreadable domains skipped, `not-found` error-code, `delete-return`
|
|
waits, `secondary-resources` explicit, `timeouts` per-call, `logging` tiered,
|
|
`backend-test` integration-tag.
|
|
|
|
## Backend facts the spec carries (allowed lessons)
|
|
|
|
- Real host `beelink`: libvirt 11.6, QEMU 10.1, pool `default` at
|
|
`/srv/vms/images`, bridge `br0`, `beelink.tail8c69c.ts.net:22`, user `novakj`,
|
|
key `~/.ssh/proxy-operator_ed25519`, known_hosts `~/.ssh/kvm_known_hosts`.
|
|
The Debian trixie qcow2 is about 435 MB.
|
|
- AppArmor confines QEMU from the domain XML: only `<disk type="file">` with a
|
|
`<source file>` path gets a rule, so volume paths are resolved with
|
|
`StorageVolGetPath` and emitted as file disks.
|
|
- Debian 13 hangs in GRUB under BIOS with a serial console and no video: UEFI
|
|
by default (`<os firmware="efi">`, secure boot feature disabled), `firmware:
|
|
bios` per image; OVMF is a host prerequisite; undefine needs the NVRAM flag.
|
|
- libvirt hands the metadata element back without its namespace declaration:
|
|
parse by local name.
|
|
- Vendor-data must be a shell script, not cloud-config, to survive cloud-init's
|
|
merge with user-data; it installs `qemu-guest-agent`. The agent answered after
|
|
about 20 s where ARP never did.
|
|
- Read-only rootfs: `iso9660.NewWriter` needs a temp dir under `/tmp` (64Mi).
|
|
- go-libvirt (`github.com/digitalocean/go-libvirt`, in go.mod): every method
|
|
blocks and takes no ctx; `libvirt.Error{Code uint32, Message}` has a value
|
|
receiver; needed `ErrorNumber` constants exist (`ErrNoDomain`,
|
|
`ErrNoStorageVol`, `ErrNoStoragePool`, `ErrOperationInvalid`,
|
|
`ErrNoDomainMetadata`, `ErrStorageVolExist`, `ErrXMLError`, `ErrNoSupport`,
|
|
`ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied`). `libvirt.New(net.Conn)`
|
|
then `Connect()`. No new dependencies: `x/crypto/ssh`, `kdomanski/iso9660`
|
|
are already direct requires; use `slices`, never `sort` (depguard).
|
|
|
|
## Design decisions recorded in the spec (step 1)
|
|
|
|
- **Package layout** (`internal/provider/kvm2/`): `kvm2.go` (type, `New`,
|
|
logger, per-name keyed lock), `id.go`, `state.go`, `errors.go`, `xml.go`,
|
|
`seed.go`, `backend.go` (ctx-first seam over the 20 go-libvirt calls, adapter
|
|
with a per-RPC deadline), `ssh.go`, `hosts.go` (one cached connection per
|
|
host, lazy dial, dialer seam), `observe.go` (the single decorator: one span
|
|
`kvm2.<Call>` per RPC plus the V(1)/V(2) log line), `image.go`, `create.go`,
|
|
`get.go`, `delete.go`, `list.go`, `check.go`, plus `fake_test.go`,
|
|
`sshserver_test.go`, `kvm2_integration_test.go`. Names are suggestions for
|
|
the briefs; the implementer may choose others by responsibility.
|
|
- **Seam:** every seam method takes `ctx`; the adapter runs the blocking call in
|
|
a goroutine and selects on result, `ctx.Done()` and a timer (dial 10s, RPC
|
|
30s, `DomainCreate` 90s, upload/download ctx plus a 20m cap). On abandon it
|
|
closes the connection and marks it dead; the next call redials. The image
|
|
download runs on a dedicated connection so a cancelled poll cannot cut it off.
|
|
- **Decorator wrapped once:** applied inside the real dialer where a connection
|
|
is born, so provider code never sees an undecorated backend. Spans `kvm2.dial`
|
|
(in the dialer) and `kvm2.download` (in `image.go`) are the two non-RPC spans.
|
|
A reflection test asserts every seam method is instrumented.
|
|
- **Metadata:** own namespace
|
|
`https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2`,
|
|
element `<instance>` with `managed`, `uid`, `createdAt`. kvm2 never lists or
|
|
adopts v1 domains; a `kvm` and a `kvm2` entry may share a host.
|
|
- **Adoption guard:** an existing domain whose metadata `uid` is non-empty and
|
|
differs from `req.UID` makes `Create` return `ErrPermanent`.
|
|
- **Quota precheck:** before creating volumes, `StoragePoolGetInfo`; needed =
|
|
image `Content-Length` (if not cached) + `diskSizeGb` + 16 MiB; below that
|
|
returns `ErrQuotaExceeded` naming pool, available and needed.
|
|
- **Download** requires `Content-Length` (Permanent otherwise), streams through
|
|
`sha256` into `StorageVolUpload`, refreshes the pool, writes the marker
|
|
volume last; a failed attempt deletes the partial volume.
|
|
- **Classify:** `ErrNoDomain`, `ErrNoStorageVol` NotFound, but Transient inside
|
|
`Create`; `ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`,
|
|
`ErrAuthFailed`, `ErrAccessDenied` Permanent; SSH authentication failures and
|
|
host-key mismatches or unknown host keys Permanent; missing key or
|
|
known_hosts file Transient (Secret pending), unparsable key Permanent;
|
|
cancellation, deadlines, `net.OpError`, `io.EOF` and anything else Transient.
|
|
- **Same-name concurrency:** a keyed mutex per instance name serialises Create
|
|
and Delete for one name.
|
|
- **State:** nostate Provisioning; running or blocked without IP Provisioning,
|
|
with IP Running; everything else, including unknown codes, Stopped. IP from
|
|
the guest agent first, then ARP, first non-loopback non-link-local IPv4.
|
|
- **Checker:** per host concurrently on a fresh dedicated connection (dial,
|
|
`ConnectGetLibVersion`, `ConnectGetHostname`, pool lookup and info, close), so
|
|
it never touches the pooled connection. Reasons: shared ones plus
|
|
`PoolNotFound`, `PoolNotActive`, `CredentialsUnavailable`. Details `user`,
|
|
`hostname`, `libvirtVersion`, `pool`, `poolState`, `poolCapacityGiB`,
|
|
`poolAvailableGiB`.
|
|
- **Rotation:** SSH material is read on every dial, so Secret rotation needs no
|
|
restart.
|
|
- **Config validation messages:** `validateKVM` is refactored into
|
|
`validateKVMBlock(prefix, block string, c *KVMConfig)` so messages say
|
|
`kvm2.` for the new block; `validateKVM` stays as a wrapper for `kvm`.
|
|
|
|
## Steps (one commit each, on `feat/kvm2-provider` off `main`)
|
|
|
|
PR #7 is merged, so `main` carries the three docs. Step 0 begins with
|
|
`git fetch origin && git switch main && git pull --ff-only`.
|
|
|
|
Every subagent brief opens and closes with: "Do not open, grep, list or import
|
|
anything under `internal/provider/kvm/`. If a detail seems to need that code,
|
|
stop and report the question instead." Each brief names the files it may create
|
|
or edit and nothing else, lists the inputs (the three docs, the spec,
|
|
`docs/kvm-provider.md` for backend facts, CLAUDE.md conventions), the
|
|
acceptance tests by name and the commands to run, and asks the agent to end
|
|
with "files read outside the allowed set: none".
|
|
|
|
- **Step 0 (orchestrator):** copy this plan to `docs/plans/<ts>-kvm2-provider.md`
|
|
with the real timestamp, commit alone, create the branch. Execution summary
|
|
`docs/plans-executions/<ts>-kvm2-provider.md` with the checklist after step 1.
|
|
- **Step 1 (orchestrator): the spec.** Copy `docs/provider-spec-template.md` to
|
|
`docs/kvm2-provider-spec.md` and answer all 24 entries with the profiles and
|
|
decisions above; every Evidence line cites the user guide, a go-libvirt symbol,
|
|
or "verified on beelink in step 11". Entry 21's checklist lists the concrete
|
|
test names from steps 4 to 7. Verify: no blank Answer lines, every axis code
|
|
used matches contract §4.20, no reference to `internal/provider/kvm/`.
|
|
- **Step 2 (subagent): config wiring, IDs, state, classify.** Edits
|
|
`internal/provider/config.go` (`KVM2 *KVMConfig` field, `setBlocks`,
|
|
allow-list, dispatch), `config_kvm.go` (`validateKVMBlock`),
|
|
`config_test.go` and `config_kvm_test.go` rows. Creates `kvm2.go` (type,
|
|
assertions, `New` that never dials and warns on unreadable SSH files, stub
|
|
methods), `id.go`, `state.go`, `errors.go` with tests:
|
|
`TestFormatProviderID_roundTrip`, `TestParseProviderID_malformed`,
|
|
`TestMapState_table`, `TestFirstIPv4_filters`, `TestClassify_table`. Also
|
|
registers `"kvm2": kvm2.New` in `cmd/main.go` so the two type lists change in
|
|
one commit. Verify: three `"kvm2"` sites in `config.go`, no import of
|
|
`provider/kvm`, `go test -race ./internal/provider/... && go vet ./...`.
|
|
- **Step 3 (subagent): pure builders.** `xml.go`, `seed.go` and tests:
|
|
`TestDomainXML_fileDisksWithResolvedPaths`, `TestDomainXML_uefiDefaultSecureBootOff`,
|
|
`TestDomainXML_biosEscape`, `TestDomainXML_metadataElement`,
|
|
`TestDomainXML_guestAgentChannel`, `TestParseMetadata_withNamespace`,
|
|
`TestParseMetadata_withoutNamespaceDeclaration`,
|
|
`TestParseMetadata_absentGivesEmptyUID`, `TestVolumeXML_overlayBackingStore`,
|
|
`TestSeedISO_filesAndLabel`, `TestSeedISO_userDataByteForByte`,
|
|
`TestSeedISO_tempDirRemoved`, `TestVendorData_isShellScriptNotCloudConfig`,
|
|
`TestVendorData_multipartWhenExtraProvided`. Verify: no `type="volume"` in
|
|
`xml.go`; temp dir under `/tmp` with a deferred removal.
|
|
- **Step 4 (subagent): transport, seam, decorator, fakes.** `backend.go`,
|
|
`ssh.go`, `hosts.go`, `observe.go`, `fake_test.go`, `sshserver_test.go` and
|
|
tests: `TestDial_forwardsUnixSocket`, `TestDial_unknownHostKeyRejected`,
|
|
`TestDial_mismatchedHostKeyRejected`, `TestDial_insecureIgnoreHostKeyAccepts`,
|
|
`TestDial_missingKeyFileIsTransient`, `TestDial_unparsableKeyIsPermanent`,
|
|
`TestDial_handshakeTimeout`, `TestCall_deadlineClosesConnectionAndReturnsTransient`,
|
|
`TestCall_cancelUnblocksRPC`, `TestHosts_oneDialPerHostUnderConcurrency`,
|
|
`TestHosts_redialsAfterDrop`, `TestHosts_unconfiguredHostIsPermanent`,
|
|
`TestObserve_everySeamMethodInstrumented`, `TestObserve_childSpansUnderParent`,
|
|
`TestObserve_logsOneLinePerCallAtV1`, `TestObserve_detailOnlyAtV2`,
|
|
`TestObserve_uploadBodyNeverLogged`, `TestSSH_keyMaterialNeverInErrorsOrLogs`.
|
|
Verify: no `context.Background()` outside tests and the keepalive goroutine;
|
|
`tracing.Start` in the decorator and the dialer; `go test -race -count=2`.
|
|
- **Step 5 (subagent): image cache and pool-space precheck.** `image.go` and
|
|
tests: `TestEnsureImage_downloadsAndWritesMarkerLast`,
|
|
`TestEnsureImage_cachedWithMarkerSkipsDownload`,
|
|
`TestEnsureImage_missingMarkerRedownloads`,
|
|
`TestEnsureImage_sha256ChangeRedownloads`,
|
|
`TestEnsureImage_sha256MismatchIsPermanentAndRemovesVolume`,
|
|
`TestEnsureImage_noContentLengthIsPermanent`, `TestEnsureImage_http404IsPermanent`,
|
|
`TestEnsureImage_http503IsTransient`, `TestEnsureImage_concurrentCallersFetchOnce`,
|
|
`TestEnsureImage_cancelledMidStreamReturnsPromptly`,
|
|
`TestEnsureImage_streamsWithoutBuffering`, `TestCheckPoolSpace_belowNeededIsQuota`.
|
|
Verify: no `io.ReadAll` or `bytes.Buffer` on the download path; download span.
|
|
- **Step 6 (subagent): the four contract methods.** `create.go`, `get.go`,
|
|
`delete.go`, `list.go`, `kvm2_test.go`, finishing `kvm2.go`. Tests map 1:1 to
|
|
contract §11: `TestCreate_repeatReturnsSameIDAndCreatesNothing`,
|
|
`TestCreate_resumesAfterEachPartialStep`, `TestCreate_returnsOnlyWhenRunning`,
|
|
`TestCreate_rejectsUnknownPlacement`, `TestCreate_adoptionWithForeignUIDIsPermanent`,
|
|
`TestCreate_poolOutOfSpaceIsQuotaExceeded`,
|
|
`TestCreate_libvirtNotFoundIsTransientNotNotFound`,
|
|
`TestCreate_cloudInitSentinelNeverLoggedOrInErrors`,
|
|
`TestGet_afterDeleteReturnsNotFound`, `TestGet_duringDeleteStillReturnsInstance`,
|
|
`TestGet_backendBusyIsTransientNeverNotFound`, `TestGet_runningWithoutIPIsProvisioning`,
|
|
`TestGet_runningWithIPIsRunning`, `TestGet_agentPreferredOverARP`,
|
|
`TestGet_addressErrorIsProvisioning`, `TestGet_readsUIDAndCreatedAtFromBackend`,
|
|
`TestGet_malformedIDIsPermanent`, `TestGet_unconfiguredHostIsPermanent`,
|
|
`TestDelete_malformedIDIsPermanent`, `TestDelete_twiceSucceeds`,
|
|
`TestDelete_removesOverlayAndSeed`, `TestDelete_runningDomainDestroyedFirst`,
|
|
`TestDelete_toleratesMissingVolumesAndPool`, `TestDelete_unconfiguredHostIsNoOp`,
|
|
`TestDelete_concurrentSameNameSafe`, `TestListByTag_excludesUnmarkedDomains`,
|
|
`TestListByTag_emptyUIDStillListed`, `TestListByTag_acrossHostsInConfigOrder`,
|
|
`TestListByTag_unreachableHostFailsWhole`, `TestListByTag_unreadableDomainSkipped`,
|
|
`TestProvider_cancelledContextReturnsPromptly`,
|
|
`TestProvider_everyErrorIsWrappedWithConfiguredName`,
|
|
`TestLogging_v0SilentV1OutcomeV2Detail`. Verify: `CloudInit` appears in
|
|
`create.go` only as `len(req.CloudInit)`; stub strings gone; `Wrap` always
|
|
gets the configured name; `go test -race -count=2 ./internal/provider/kvm2/`.
|
|
- **Step 7 (subagent): Checker and the CRD comment.** `check.go`,
|
|
`check_test.go`: `TestCheck_oneEntryPerHostInConfigOrder`,
|
|
`TestCheck_hostsCheckedConcurrently`, `TestCheck_expiredContextReturnsPromptly`,
|
|
`TestCheck_reasonsTable`, `TestCheck_neverUsesPooledConnection`,
|
|
`TestCheck_detailsKeys`. Edit `api/v1alpha1/providerhealth_types.go`
|
|
(`status.type` comment) and run `make manifests`. Verify: only the CRD
|
|
description line changes under `config/`; `go test ./internal/providerhealth/`.
|
|
- **Step 8 (subagent): deployment, Makefile, integration test.** Edit
|
|
`config/manager/manager.yaml` (comment: the `kvm-ssh` mount serves kvm and
|
|
kvm2), `config/manager/providers_config.yaml` and
|
|
`config/samples/providers-config.yaml` (commented `kvm2` entry),
|
|
`config/samples/kustomization.yaml`, `Makefile` (`test-integration-kvm2`,
|
|
`-run TestKVM2`, `-timeout 40m`), `docs/testing.md` (suite rows and a kvm2
|
|
integration section). Create `config/samples/proxy_kvm2.yaml` (zone
|
|
`beelink`, provider `kvm2-home`) and `kvm2_integration_test.go`
|
|
(`KVM2_TEST_HOST`, `_USER`, `_SSH_KEY`, `_KNOWN_HOSTS`, `_BRIDGE`,
|
|
`_IMAGE_URL`; optional `_IMAGE_SHA256`, `_POOL`, `_SOCKET`, `_TIMEOUT`,
|
|
`_KEEP`, `_LOG_LEVEL`) asserting in order: `New` does not dial; `Check`
|
|
Ready with details; `Create` returns `hosts/<host>/domains/<name>`; second
|
|
`Create` same ID, domain count unchanged; `Get` reaches Running with an IPv4
|
|
and the written UID and CreatedAt; `ListByTag` contains it; `Delete` nil;
|
|
`Get` NotFound; `Delete` again nil; pool holds base and marker, no per-VM
|
|
volumes; with an in-memory span recorder, `kvm2.DomainDefineXML` and
|
|
`kvm2.dial` spans are children of `provider.create`. Verify:
|
|
`go vet -tags=integration ./... && go vet -tags=e2e ./...`,
|
|
`bin/kustomize build config/default` and `config/samples` parse,
|
|
`make -n test-integration-kvm2`.
|
|
- **Step 9 (subagent): documentation.** Create `docs/kvm2-provider.md`
|
|
(prerequisites incl. OVMF, cluster setup reusing `kvm-ssh`, config reference
|
|
pointing at the shared `KVMConfig` table, placement mapping and the
|
|
add-a-key-never-edit warning, what lands on the host incl. the metadata
|
|
namespace, image cache and quota precheck, health-check reasons, logging and
|
|
span names, migration from `kvm`, known gaps). Edit
|
|
`docs/providers-existing.md` (fourth column in the three tables and a kvm2
|
|
section), `docs/architecture.md` (components row, §5 block, §10 span
|
|
topology, §11 row, Decisions bullets), `README.md` row, `docs/providers.md`
|
|
guide link. Verify with the link and symbol sweep from the docs-split
|
|
execution log; no em-dashes in new prose.
|
|
- **Step 10 (orchestrator): milestone review.** `go-operator-reviewer` on the
|
|
branch with the spec and contract as inputs, then `/code-review high`,
|
|
`make lint`, `make test` (full, with envtest). Fix accepted findings with
|
|
regression tests in one commit; record rejected ones in the summary.
|
|
- **Step 11 (orchestrator with the user): real-host validation, changelog, MR.**
|
|
Hand the user a host checklist (leftover `proxy-*` domains and volumes from
|
|
earlier runs, OVMF installed, pool running with free space, `br0` up, the
|
|
known_hosts entry) and the `KVM2_TEST_*` export lines; do not run remote
|
|
probes unprompted. With the user's go-ahead run `make test-integration-kvm2`
|
|
(a permission prompt is expected; the allow-list entry
|
|
`make test-integration-kvm *` does not match) and, if wanted, the in-cluster
|
|
pass on kind: build and load the image, deploy, a scratch ConfigMap with the
|
|
`kvm2-home` entry, apply `proxy_kvm2.yaml`, wait for `Ready`,
|
|
`kubectl get ph kvm2-home -o yaml`, a `curl -x` through the proxy, delete
|
|
and confirm the domain and volumes are gone; scratch overrides are not
|
|
committed. Record cold download duration and time to IP. CHANGELOG entry
|
|
after the user confirms; push with the token helper; MR via the Gitea API
|
|
with base `main`; never merge.
|
|
|
|
## Verification
|
|
|
|
- Per step: the checks listed above, plus `go build ./... && go test -short ./...`
|
|
before every commit and `git diff --stat` reviewed for files outside the
|
|
step's allowed set.
|
|
- Clean-room check on every step's diff: `grep -rn 'provider/kvm"' internal/provider/kvm2/`
|
|
empty; the agent's closing line "files read outside the allowed set: none";
|
|
identifiers that coincide with v1's names from the appendix (`libvirtAPI`,
|
|
`hostPool`, `imageCache`, `tracingAPI`, `lazySSH`) are treated as a smell and
|
|
questioned.
|
|
- Contract mapping: after step 7, every rule of contract §2 and §8 has a named
|
|
test in the package; the §11 list maps 1:1 to test names (recorded in the
|
|
spec's entry 21).
|
|
- End to end: the integration run's ordered assertions (step 8) on beelink, and
|
|
the optional kind pass (step 11).
|
|
|
|
## Risks
|
|
|
|
- Subagent scope creep into `internal/provider/kvm/`: instruction-only guard,
|
|
mitigated by the closing attestation and the diff greps.
|
|
- `.claude/settings.json` is locally modified and uncommitted; it stays out of
|
|
feature commits, and the permission prompt for the new make target is
|
|
accepted rather than pre-approved.
|
|
- `Content-Length` is required for downloads; the Debian and Ubuntu mirrors send
|
|
it.
|
|
- The cancelled-call policy still drops other in-flight RPCs on that host once;
|
|
the dedicated download connection removes the expensive case and the spec
|
|
states the rest.
|
|
- `go-operator-reviewer` hunts over-abstraction; the seam, the decorator and
|
|
the dialer interface each have one production implementation and one fake,
|
|
which the contract requires (§4.15, §11).
|