provider: extract nocloud seed builder and keyedlock from kvm2
kvm2's NoCloud seed builder moves to internal/provider/nocloud (exported, with an ErrEmptyName sentinel instead of permanentf) and its keyed locks to internal/provider/keyedlock, tests moved under their existing names. kvm2 behaviour is unchanged. proxmox gains the per-name lock and the pure seed helpers (seedISOName, seedVolID, buildSeed); unlike kvm2 (review finding S5.1), a seed staging failure there is Transient, not Permanent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -11,6 +11,7 @@
|
||||
| gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
|
||||
| kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
|
||||
| kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial |
|
||||
| Shared provider helpers | `internal/provider/nocloud`, `internal/provider/keyedlock` | library | — | the NoCloud seed ISO and vendor-data builder, and one-slot semaphores by key; used by kvm2 and proxmox |
|
||||
| Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events |
|
||||
| Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
|
||||
| Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
|
||||
|
||||
@@ -511,7 +511,7 @@ every 15 s) marks the connection dead on failure.
|
||||
per host, dialled lazily, under a mutex with a per-host dial mutex so
|
||||
concurrent callers dial once and share the result), the per-`(host, volume)`
|
||||
download locks, and a lock per instance name. Both lock sets are one instance
|
||||
each of `keyedLocks`: a one-slot semaphore per key, so a waiter gives up when
|
||||
each of `keyedlock.Locks` (`internal/provider/keyedlock`): a one-slot semaphore per key, so a waiter gives up when
|
||||
its ctx ends, reference-counted so an entry is removed from the map once no
|
||||
caller holds or waits for it and the map does not grow with every name ever
|
||||
seen, while two waiters on one key always share one semaphore. Create and
|
||||
|
||||
@@ -252,7 +252,8 @@ UIDs, and a marked domain with a matching or empty `uid` is adopted. The
|
||||
namespace is kvm2's own, so the `kvm` provider's domains are invisible to
|
||||
it (see *Migrating from kvm*).
|
||||
|
||||
The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled
|
||||
The seed (`nocloud.BuildSeedISO`, shared with the proxmox provider in
|
||||
`internal/provider/nocloud`) is an ISO 9660 image labelled
|
||||
`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
|
||||
for byte**: any format cloud-init accepts works, including a Jinja
|
||||
template, a MIME archive, a shell script or a gzipped payload; the
|
||||
@@ -267,7 +268,7 @@ previous attempt may have been cut off mid-upload and a truncated seed
|
||||
would boot a VM without cloud-init.
|
||||
|
||||
The guest agent rides along in `vendor-data` as a **shell script**
|
||||
(`guestAgentScript` in `seed.go`), not a cloud-config, on purpose:
|
||||
(`nocloud.GuestAgentScript`), not a cloud-config, on purpose:
|
||||
cloud-init merges a vendor-data cloud-config under the user-data with
|
||||
mergers that keep the user-data's value for every top-level key both
|
||||
define and never append lists, so a Proxy carrying its own `packages:`
|
||||
|
||||
@@ -7,7 +7,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18-
|
||||
- [x] Phase 0 — Plan file, branch, host recon, fixtures
|
||||
- [x] Phase 1 — The spec (`docs/proxmox-provider-spec.md`)
|
||||
- [x] Phase 2 — Config block, registration, package skeleton
|
||||
- [ ] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2
|
||||
- [x] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2
|
||||
- [ ] Phase 4 — Seam, HTTP client, decorator, fake
|
||||
- [ ] Phase 5 — Get, ListByTag, task helpers
|
||||
- [ ] Phase 6 — Create
|
||||
@@ -381,3 +381,69 @@ go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 152
|
||||
Worth noting: the `validProxmoxYAML` constant is the shape Phase 9's commented
|
||||
sample entry should take, with the user's values (`192.168.0.8`, `pv1-1`,
|
||||
`proxy-seeds`, `vlanTag: 4`, template 9001).
|
||||
|
||||
## Phase 3 — Extract `nocloud` and `keyedlock` from kvm2
|
||||
|
||||
kvm2's `seed.go` and `seed_test.go` moved to `internal/provider/nocloud`
|
||||
with `git mv`, so history follows them. The package exports `BuildSeedISO`,
|
||||
`BuildVendorData`, `GuestAgentScript`, `SeedLabel` and a new `ErrEmptyName`
|
||||
sentinel in place of kvm2's `permanentf`. kvm2's `keyedLocks` became
|
||||
`internal/provider/keyedlock` (`Locks`, `New`, `Acquire`), and its three
|
||||
`TestKeyedLocks_*` tests moved with it under their old names. kvm2 now
|
||||
imports both packages, and its `create.go` still wraps every seed-build error
|
||||
with `permanentf`, so kvm2's behaviour is unchanged. proxmox gained the
|
||||
`nameLocks` field and `lockName`.
|
||||
|
||||
The test counts match the plan's Verify clause: kvm2 dropped from 465 to 447
|
||||
`=== RUN` entries, which is the 12 moved tests plus the 6 subtests of
|
||||
`TestSeedISO_userDataByteForByte`. `nocloud` (15) and `keyedlock` (3) carry
|
||||
exactly those 18.
|
||||
|
||||
```bash
|
||||
git mv internal/provider/kvm2/seed.go internal/provider/nocloud/seed.go
|
||||
git mv internal/provider/kvm2/seed_test.go internal/provider/nocloud/seed_test.go
|
||||
go test -count=1 -v ./internal/provider/kvm2/ | grep -c '^=== RUN' # 465 before, 447 after
|
||||
grep -rn 'iso9660' internal/provider/kvm2/ # nothing
|
||||
go build ./... && go vet ./... && go vet -tags=integration ./...
|
||||
go test -race -count=1 ./internal/provider/... # all ok, run three times
|
||||
go test -short -count=1 ./... # all ok
|
||||
```
|
||||
|
||||
**Pulled forward from Phase 6 so proxmox imports `nocloud` now**, as the
|
||||
"Done when" line requires: `seed.go` with `seedISOName`, `seedVolID` and
|
||||
`buildSeed`, the pure part of the seed step. `uploadSeed` still belongs to
|
||||
Phase 6, which calls `buildSeed`. The new tests are `TestSeedNames_table`,
|
||||
`TestBuildSeed_table` (4 rows),
|
||||
`TestBuildSeed_emptyNameIsPermanentWithoutPayload`,
|
||||
`TestBuildSeed_stagingFailureIsTransient` and
|
||||
`TestLockName_givesUpWithContextAsTransient`. proxmox went from 152 to 161
|
||||
test entries.
|
||||
|
||||
**proxmox does not copy kvm2's open review finding S5.1**
|
||||
(`docs/reviews/2026-09-17-kvm2-provider-review/sweep.md`). In kvm2, a
|
||||
staging failure such as a missing `/tmp` or a full emptyDir becomes
|
||||
Permanent, and that latches the Proxy. `buildSeed` makes only
|
||||
`ErrEmptyName` Permanent and wraps any other error with `%w`, so
|
||||
`classify` reads it as Transient. The sentinel makes the same three-line
|
||||
fix possible in kvm2, but this phase leaves kvm2's behaviour unchanged.
|
||||
|
||||
Worth noting:
|
||||
|
||||
- The older `kvm` provider has its own `seed.go` with a different
|
||||
signature (`buildSeedISO` returns `[]byte`). The plan scopes the
|
||||
extraction to kvm2, so `kvm` is untouched.
|
||||
- `GuestAgentScript` now says the operator reads the address "from the
|
||||
hypervisor" instead of "through libvirt", in both the Go comment and the
|
||||
script's comment line inside the guest. The script's commands are
|
||||
unchanged.
|
||||
- The docs now name the new packages: `kvm2-provider.md`,
|
||||
`kvm2-provider-spec.md` entry 19, the kvm2 columns and section of
|
||||
`providers-existing.md`, plus a "Shared provider helpers" row in the
|
||||
`architecture.md` component table. `docs/reviews/` keeps the old names,
|
||||
because it records the code as it was reviewed.
|
||||
- `TestCheck_hostsCheckedConcurrently` in kvm2 failed once under the full
|
||||
`-race` suite (latency 199.96 ms against a 200 ms floor). It passed 20
|
||||
runs out of 20 in isolation, with and without this phase's changes. The
|
||||
test starts its 200 ms release timers before `Check` starts its clock, so
|
||||
under load the measured latency can come in just under 200 ms. The flake
|
||||
predates this phase, and the test is not touched here.
|
||||
|
||||
@@ -58,7 +58,7 @@ against.
|
||||
|---|---|---|---|---|
|
||||
| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) |
|
||||
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image |
|
||||
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
|
||||
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) |
|
||||
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no |
|
||||
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped |
|
||||
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined |
|
||||
@@ -68,7 +68,7 @@ against.
|
||||
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) |
|
||||
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) |
|
||||
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter |
|
||||
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
|
||||
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) |
|
||||
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN |
|
||||
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself |
|
||||
|
||||
@@ -347,8 +347,10 @@ same libvirt.
|
||||
production adapter and the budgets), `ssh.go` (credentials, the forwarded
|
||||
socket, keepalive), `hosts.go` (the pooled table and the production
|
||||
dialer), `observe.go` (the span and log decorator), `image.go` (cache,
|
||||
download, quota precheck), `seed.go`, `xml.go`, `volume.go`,
|
||||
`metadata.go`, `id.go`, `state.go`, `errors.go`, `check.go`. The seam is
|
||||
download, quota precheck), `xml.go`, `volume.go`, `metadata.go`,
|
||||
`id.go`, `state.go`, `errors.go`, `check.go`; the seed ISO comes from
|
||||
the shared `internal/provider/nocloud` and both lock sets are
|
||||
`internal/provider/keyedlock`. The seam is
|
||||
`backend`: the twenty go-libvirt calls the provider uses, each with `ctx`
|
||||
first (go-libvirt's own methods take none and block), plus `Close`.
|
||||
`libvirtBackend` adapts `*libvirt.Libvirt` and runs every call through
|
||||
|
||||
Reference in New Issue
Block a user