proxmox: Create by cloning a template, seed upload, config, start
Create validates placement, then under the per-name lock adopts the VM a previous attempt left (by name, or at a taken VMID after the ownership check of spec entry 2) or full-clones the template onto the UID's deterministic VMID with the mark in the clone call. It then uploads the seed over any existing file, writes cores, memory, agent, tag, ide2 and net0 when configured, grows the boot disk when diskGiB is larger, starts, and waits for running/running. waitUnlocked no longer waits on a suspended lock: a hibernated VM is judged at once, so ours is Permanent and a foreign one is probed past. The template is checked only on the allocation path, and its shape (net0, boot disk size) is refused before any clone exists. Spec entries 2, 4 and 10 updated to match. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -10,7 +10,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18-
|
||||
- [x] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2
|
||||
- [x] Phase 4 — Seam, HTTP client, decorator, fake
|
||||
- [x] Phase 5 — Get, ListByTag, task helpers
|
||||
- [ ] Phase 6 — Create
|
||||
- [x] Phase 6 — Create
|
||||
- [ ] Phase 7 — Delete, Checker, CRD comment
|
||||
- [ ] Phase 8 — Contract and observability sweep, integration test, testing.md
|
||||
- [ ] Phase 9 — Deployment and documentation
|
||||
@@ -657,3 +657,135 @@ Worth noting:
|
||||
- Nothing in this phase talked to the host. Every Proxmox message a test
|
||||
relies on comes from `testdata/` or from fake errors that
|
||||
`TestFakeAPI_errorsMatchCaptures` ties to the captures.
|
||||
|
||||
## Phase 6 — Create
|
||||
|
||||
`create.go` holds `Create` and its steps: placement, the by-name lookup,
|
||||
the template check, the VMID probe with its ownership check (`inspect`),
|
||||
the clone, and `provision` (seed, config, resize, start). `configParams`,
|
||||
`rewriteNet0`, `resizeTarget`, `bootDisk` and `diskBytes` are pure helpers
|
||||
with their own table tests. `seed.go` gains `uploadSeed`, which uploads
|
||||
over any existing file with no delete first (F12). `task.go` gains
|
||||
`waitRunning`, built on `pollUntil` with `budgets.running`. It finishes
|
||||
only at `running`/`running`, and a `stopped` status after the start is
|
||||
Transient at once. The Create stub is gone, and `TestStubs_answerTransient`
|
||||
now covers only Delete.
|
||||
|
||||
The spec won over the plan wherever the Phase 6 notes said it would. The
|
||||
plan's allocation loop is replaced by entry 2: a taken VMID and a clone
|
||||
answering F4 both go through the ownership check before any probing, and a
|
||||
locked or nameless entry is waited on with `waitUnlocked` and the clone
|
||||
budget. The seed is uploaded without a delete. The resize is a task. `net0`
|
||||
changes only when `bridge` or `vlanTag` is set, and `tag=` is replaced or
|
||||
appended in place.
|
||||
|
||||
Judgment calls, with spec entries 2, 4 and 10 updated to match:
|
||||
|
||||
- **The Phase 5 open question: a `suspended` lock is not waited on.**
|
||||
`waitUnlocked` now returns at once when it sees `suspended`, because a
|
||||
hibernated VM is not a task and would otherwise burn the 10-minute clone
|
||||
budget on every attempt. The ownership check then judges the VM the way
|
||||
`Get` does. A foreign hibernated VM is probed past. Ours is Permanent
|
||||
("resume or destroy it"), following kvm2's precedent for a paused or
|
||||
pmsuspended domain in Create. Transient was the alternative: it would
|
||||
self-heal once an admin acts, but it would retry indefinitely on
|
||||
something only an admin can fix. `TestWaitUnlocked_table` has a new row,
|
||||
and `TestCreate_hibernatedVMIsNotWaitedOn` covers both cases.
|
||||
- **The template is checked only when a clone is needed.** Spec entry 10
|
||||
read the template's config in step 2, before the by-name lookup. A retry
|
||||
for a VM that already exists would then fail Permanent if the template
|
||||
had been deleted since, which entry 4 explicitly allows. The template
|
||||
check now runs after the by-name lookup misses, so a repeat Create costs
|
||||
three calls: resources, config and status
|
||||
(`TestCreate_adoptionDoesNotNeedTheTemplate`).
|
||||
- **The template's shape is checked before the clone.** No `net0` while
|
||||
`bridge` or `vlanTag` is set, or no boot disk (or none with a readable
|
||||
`size=`) while `diskGiB` is set, is Permanent. Found after the clone, the
|
||||
VM would already carry the bad config, so fixing the template could not
|
||||
help that Proxy's next attempt. `template()` runs the same
|
||||
`configParams` and `resizeTarget` over the template's config that
|
||||
`provision` later runs over the clone's.
|
||||
- **An empty `req.Name` is Permanent at placement.** A nameless request
|
||||
would match nameless mid-clone entries in the by-name lookup.
|
||||
- **Adopted VMs get one `status/current` read** rather than trusting the
|
||||
listing's status, which may be stale after a lock wait. A running VM,
|
||||
whatever its `qmpstatus`, returns its ID at once, as entry 10 says. `Get`
|
||||
then reports prelaunch as Provisioning and paused as Stopped.
|
||||
- **After the clone task, Create reads the config through
|
||||
`waitUnlocked`.** That read supplies the `net0` and boot disk the later
|
||||
steps need, and it turns a clone that ended but kept its lock (F15) into
|
||||
a Transient lock wait rather than a refused config write.
|
||||
- **An F4 whose VMID then resolves to no QEMU VM probes on.** Either the
|
||||
racer's clone failed and removed it, or a container took it.
|
||||
- **Errors carry no ID** (`wrapErr("create", "", …)`), since the VMID is
|
||||
known only after allocation. The messages name the VMID wherever one
|
||||
exists.
|
||||
|
||||
Test names: every plan and entry 21 name exists except
|
||||
`TestCreate_resizeOnlyWhenLarger`. It is covered by the spec's pair:
|
||||
`TestCreate_resizeWaitsForTask` for the larger case and
|
||||
`TestCreate_resizeSkippedWhenNotLarger` for equal, smaller and unset. Tests
|
||||
beyond both lists: `TestCreate_vmThatStopsAfterStartIsTransientAtOnce`,
|
||||
`TestCreate_hibernatedVMIsNotWaitedOn`,
|
||||
`TestCreate_templateShapeRefusedBeforeClone`,
|
||||
`TestCreate_adoptionDoesNotNeedTheTemplate`,
|
||||
`TestCreate_cloneTargetAndStorage` (read from the decorator's V(2) clone
|
||||
line), `TestCreate_adoptedVMStaysOnItsNode`, `TestCreate_outcomeLine`,
|
||||
`TestCreate_quietAtV0`, `TestRewriteNet0_table`, `TestBootDisk_capturedClone`
|
||||
and `TestDiskBytes_table`. The only new test helpers are the small ones in
|
||||
`create_test.go` (`slot`, `newCreateFixture`, `req`/`create`/`createFails`,
|
||||
`assertRunning`, `callsBefore`, `clearLockOnVMConfig`). Everything else
|
||||
reuses the Phase 5 fixture and fake. Two tests check config-write contents
|
||||
without a recording wrapper. `TestCreate_markWrittenInCloneCall` reads the
|
||||
VM at the start of `UploadISO`, which is after the clone and before the
|
||||
config write. `TestCreate_noBridgeOverrideLeavesNet0` changes `net0` at the
|
||||
start of `SetVMConfig`, so a write that carried `net0` would put the old
|
||||
value back.
|
||||
|
||||
A mutation check applied 18 mutations, and each failed at least one test.
|
||||
They were: no lock wait in the ownership check; taken VMIDs never
|
||||
inspected; the F4 race skipping the ownership check; no cloud-init drive
|
||||
check; upload or resize task not waited on; resize at equal size; a
|
||||
CD-ROM counted as a boot disk; prelaunch counted as running; the suspended
|
||||
lock waited on; `net0` always written; `bridge=` appended instead of
|
||||
replaced; duplicate names ignored; a running VM re-provisioned; `target`
|
||||
always set; a foreign UID adopted; the tag dropped; probes not capped by
|
||||
the range. The last one was first **missed**. A 4-VMID range with 32
|
||||
probes wraps, re-inspects each VM eight times and still ends Transient, so
|
||||
`TestCreate_vmidRangeExhaustedIsTransient` now pins five config reads.
|
||||
Five mutations did not compile on the first try and were redone. Every
|
||||
mutation was reverted, and the files were compared identical to backups.
|
||||
|
||||
```bash
|
||||
go test -race -count=2 ./internal/provider/proxmox/ # ok
|
||||
grep -n 'CloudInit' internal/provider/proxmox/create.go internal/provider/proxmox/seed.go
|
||||
# seed.go: the pass-through into nocloud.BuildSeedISO; create.go: len(req.CloudInit)
|
||||
grep -n 'not implemented' internal/provider/proxmox/*.go | grep -v _test # Delete only
|
||||
grep -n 'context.Background()' internal/provider/proxmox/*.go | grep -v _test.go # nothing
|
||||
go build ./... && go vet ./... && go vet -tags=integration ./...
|
||||
go test -short -count=1 ./... # all ok but the parked kvm2 flake, below
|
||||
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 393 before, 519 after
|
||||
bin/golangci-lint run --disable staticcheck ./internal/provider/proxmox/...
|
||||
```
|
||||
|
||||
Worth noting:
|
||||
|
||||
- `go test -short ./...` failed once on the parked kvm2 flake
|
||||
`TestCheck_hostsCheckedConcurrently` (measured 199.98 ms against a
|
||||
200 ms floor). Three reruns passed. This phase does not touch kvm2.
|
||||
- The new lint findings follow the baseline's shape. There are logcheck
|
||||
hits on `create`, `allocate` and `clone` (ctx plus logger, as in kvm2),
|
||||
and goconst on test strings. One more unparam than the notes list:
|
||||
`(*fakeAPI).iso`'s `storage` always receives `fakeISOStorage`, because
|
||||
this phase is its first caller. `addTemplate`'s hit changed from a
|
||||
constant VMID to an unused result.
|
||||
- The config write sets `agent=1` and `tags=proxy_operator` as values. A
|
||||
template's agent options (such as `fstrim_cloned_disks=1`) and its own
|
||||
tags are replaced, not merged. That is what entry 10 specifies, and the
|
||||
guide should say so.
|
||||
- In the fake, a cross-node clone puts the config on the target at once.
|
||||
Real Proxmox may move it at the end of the task, in which case a lock
|
||||
wait on the target node would see `does not exist` and the next attempt
|
||||
would find the VM. This stays untested, like every cross-node path
|
||||
(entry 24).
|
||||
- Nothing in this phase talked to the host.
|
||||
|
||||
@@ -72,7 +72,12 @@ lands on the same number: `vmidFor(uid, attempt)` is
|
||||
seed step (entry 10). It is **foreign** when the name differs: move to
|
||||
the next attempt. The same name with no mark, or with a different
|
||||
non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the
|
||||
operator never takes over a VM it did not create.
|
||||
operator never takes over a VM it did not create. A `suspended` lock (a
|
||||
VM hibernated to disk) is not waited on: it is not a task and never
|
||||
clears by itself. Ownership is judged at once, as `Get` does (entry 9):
|
||||
a foreign hibernated VM is probed past, and ours is `ErrPermanent`
|
||||
(resume or destroy it), like kvm2's paused domain. Container and
|
||||
template entries at a VMID are foreign without a config read.
|
||||
|
||||
The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after
|
||||
a crash mid-clone, the retry's by-name lookup misses the nameless VM, but
|
||||
@@ -113,7 +118,12 @@ template may be edited, replaced or deleted without affecting running VMs.
|
||||
Before cloning, Create reads the template's config and refuses it with
|
||||
`ErrPermanent` when it is not a template (`template` is not 1) or when any
|
||||
disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`),
|
||||
which would compete with the provider's seed (entry 5). The template must
|
||||
which would compete with the provider's seed (entry 5). Because the clone
|
||||
copies the template's config, what the config or resize step would refuse
|
||||
on the clone is refused on the template too, before any VM exists: no
|
||||
`net0` while `bridge` or `vlanTag` is set, and no boot disk, or one
|
||||
without a readable `size=`, while `diskGiB` is set. A Permanent error
|
||||
after the clone would leave a VM that fixing the template cannot help. The template must
|
||||
boot from its first disk, run cloud-init with the NoCloud datasource, and
|
||||
either ship `qemu-guest-agent` or be able to install it (entry 7).
|
||||
|
||||
@@ -320,11 +330,15 @@ waits for the clone task first.
|
||||
Steps, each idempotent, in order, under the per-name lock (entry 19):
|
||||
|
||||
1. Validate placement (entry 3). No backend call yet.
|
||||
2. `GET /cluster/resources`. Find the template's node; a missing template
|
||||
is `ErrPermanent`. Read the template's config and refuse it as entry 4
|
||||
says.
|
||||
3. Find or allocate the VMID (entry 2). An adopted VM that is running
|
||||
returns its ID at once. An adopted VM that is stopped continues at step 5.
|
||||
2. `GET /cluster/resources` once, then look the name up (entry 2).
|
||||
3. When the name finds nothing: find the template's node (a missing
|
||||
template is `ErrPermanent`), read the template's config and refuse it as
|
||||
entry 4 says, then allocate the VMID (entry 2). The template is checked
|
||||
only here, because a VM that already exists no longer depends on it
|
||||
(entry 4): a template deleted after its clone must not fail that
|
||||
clone's adoption. An adopted VM, found by name or by the probe, gets one
|
||||
`status/current` read. Running, it returns its ID at once. Stopped, it
|
||||
continues at step 5.
|
||||
4. Clone and wait (clone budget).
|
||||
5. Build the seed and upload it to the VM's node, overwriting; wait
|
||||
(upload budget).
|
||||
|
||||
@@ -0,0 +1,425 @@
|
||||
package proxmox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"maps"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/go-logr/logr"
|
||||
|
||||
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
|
||||
)
|
||||
|
||||
// maxProbes bounds the VMID probe of spec entry 2.
|
||||
const maxProbes = 32
|
||||
|
||||
// diskKey is a Proxmox disk slot: the keys a boot disk, a CD-ROM or a
|
||||
// cloud-init drive can sit on.
|
||||
var diskKey = regexp.MustCompile(`^(ide|sata|scsi|virtio)[0-9]+$`)
|
||||
|
||||
// ownedVM is a VM Create has cloned or adopted: where it lives and its
|
||||
// config as last read, which the config and resize steps start from.
|
||||
type ownedVM struct {
|
||||
node string
|
||||
vmid int
|
||||
cfg vmConfig
|
||||
}
|
||||
|
||||
// Create is spec entry 10: validate placement, then, under the per-name
|
||||
// lock, adopt the VM a previous attempt left or clone a new one, and carry
|
||||
// it through seed, config, resize and start until it runs.
|
||||
func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (string, error) {
|
||||
node, mt, tmpl, err := p.placement(req)
|
||||
if err != nil {
|
||||
return "", p.wrapErr("create", "", err)
|
||||
}
|
||||
log := p.logger(ctx).WithValues("name", req.Name)
|
||||
log.V(2).Info("proxmox create resolved",
|
||||
"zone", node, "machineType", req.Placement.MachineType, "cores", mt.Cores, "memoryMiB", mt.MemoryMiB,
|
||||
"diskGiB", mt.DiskGiB, "image", req.Placement.Image, "templateVMID", tmpl.VMID, "storage", p.cfg.Storage,
|
||||
"isoStorage", p.cfg.ISOStorage, "bridge", p.cfg.Bridge, "vlanTag", p.cfg.VLANTag, "cloudInitBytes", len(req.CloudInit))
|
||||
|
||||
unlock, err := p.lockName(ctx, req.Name)
|
||||
if err != nil {
|
||||
return "", p.wrapErr("create", "", err)
|
||||
}
|
||||
defer unlock()
|
||||
|
||||
vm, err := p.create(ctx, log, node, mt, tmpl, req)
|
||||
if err != nil {
|
||||
return "", p.wrapErr("create", "", err)
|
||||
}
|
||||
log.V(1).Info("proxmox vm ensured running", "vmid", vm.vmid, "node", vm.node)
|
||||
return formatProviderID(vm.vmid, req.Name), nil
|
||||
}
|
||||
|
||||
// placement is step 1: every catalogue key is checked before any backend
|
||||
// call, and an unknown one is Permanent naming the field and the value
|
||||
// (spec entry 3). Region is ignored.
|
||||
func (p *Provider) placement(req provider.CreateRequest) (string, provider.ProxmoxMachineType, provider.ProxmoxTemplate, error) {
|
||||
var (
|
||||
mt provider.ProxmoxMachineType
|
||||
tmpl provider.ProxmoxTemplate
|
||||
)
|
||||
if req.Name == "" {
|
||||
return "", mt, tmpl, permanentf("the request has no instance name")
|
||||
}
|
||||
node, err := p.node(req.Placement.Zone)
|
||||
if err != nil {
|
||||
return "", mt, tmpl, err
|
||||
}
|
||||
mt, ok := p.cfg.MachineTypes[req.Placement.MachineType]
|
||||
if !ok {
|
||||
return "", mt, tmpl, permanentf("placement.machineType %q is not in proxmox.machineTypes", req.Placement.MachineType)
|
||||
}
|
||||
tmpl, ok = p.cfg.Templates[req.Placement.Image]
|
||||
if !ok {
|
||||
return "", mt, tmpl, permanentf("placement.image %q is not in proxmox.templates", req.Placement.Image)
|
||||
}
|
||||
return node, mt, tmpl, nil
|
||||
}
|
||||
|
||||
// create is steps 2 to 9. The template is checked only when a clone is
|
||||
// needed: a VM a previous attempt created no longer depends on it (spec
|
||||
// entry 4), so a template removed since must not fail its adoption.
|
||||
func (p *Provider) create(ctx context.Context, log logr.Logger, node string, mt provider.ProxmoxMachineType, tmpl provider.ProxmoxTemplate, req provider.CreateRequest) (ownedVM, error) {
|
||||
rs, err := p.api.ClusterResources(ctx)
|
||||
if err != nil {
|
||||
return ownedVM{}, fmt.Errorf("listing cluster resources: %w", err)
|
||||
}
|
||||
vm, adopted, err := p.adoptByName(ctx, rs, req)
|
||||
if err != nil {
|
||||
return ownedVM{}, err
|
||||
}
|
||||
if !adopted {
|
||||
src, err := p.template(ctx, rs, req.Placement.Image, tmpl.VMID, mt)
|
||||
if err != nil {
|
||||
return ownedVM{}, err
|
||||
}
|
||||
if vm, adopted, err = p.allocate(ctx, log, rs, node, src, req); err != nil {
|
||||
return ownedVM{}, err
|
||||
}
|
||||
}
|
||||
log = log.WithValues("vmid", vm.vmid, "node", vm.node)
|
||||
if adopted {
|
||||
st, err := p.api.VMStatus(ctx, vm.node, vm.vmid)
|
||||
if err != nil {
|
||||
return ownedVM{}, fmt.Errorf("reading status of VM %d: %w", vm.vmid, err)
|
||||
}
|
||||
log.V(1).Info("proxmox vm adopted", "status", st.Status, "qmpstatus", st.QMPStatus)
|
||||
if st.Status == "running" {
|
||||
return vm, nil
|
||||
}
|
||||
}
|
||||
return vm, p.provision(ctx, log, vm, mt, req)
|
||||
}
|
||||
|
||||
// adoptByName is the by-name half of spec entry 2: two VMs with the name
|
||||
// are for an admin to sort out, one goes through the ownership check.
|
||||
func (p *Provider) adoptByName(ctx context.Context, rs []vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
|
||||
var found []vmResource
|
||||
for _, r := range rs {
|
||||
if r.Type == "qemu" && r.Template == 0 && r.Name == req.Name {
|
||||
found = append(found, r)
|
||||
}
|
||||
}
|
||||
switch len(found) {
|
||||
case 0:
|
||||
return ownedVM{}, false, nil
|
||||
case 1:
|
||||
return p.inspect(ctx, found[0], req)
|
||||
}
|
||||
vmids := make([]string, len(found))
|
||||
for i, r := range found {
|
||||
vmids[i] = strconv.Itoa(r.VMID)
|
||||
}
|
||||
return ownedVM{}, false, permanentf("VMs %s are all named %s; remove all but one", strings.Join(vmids, ", "), req.Name)
|
||||
}
|
||||
|
||||
// inspect is the ownership check of spec entry 2 on a VMID that is taken.
|
||||
// It waits out a task's lock (a clone in progress is also nameless), then
|
||||
// reads the name and the mark: another name is someone else's VM, and our
|
||||
// name without our mark is never taken over.
|
||||
func (p *Provider) inspect(ctx context.Context, r vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
|
||||
cfg, err := p.waitUnlocked(ctx, r.Node, r.VMID, p.budgets.clone)
|
||||
if err != nil {
|
||||
return ownedVM{}, false, err
|
||||
}
|
||||
if cfg["name"] != req.Name {
|
||||
return ownedVM{}, false, nil
|
||||
}
|
||||
mark := parseDescription(cfg["description"])
|
||||
switch {
|
||||
case !mark.Managed:
|
||||
return ownedVM{}, false, permanentf("VM %d is named %s but carries no proxy-operator mark; it was not created by this operator", r.VMID, req.Name)
|
||||
case mark.UID != "" && mark.UID != req.UID:
|
||||
return ownedVM{}, false, permanentf("VM %d named %s belongs to proxy uid %s, not %s", r.VMID, req.Name, mark.UID, req.UID)
|
||||
case cfg["lock"] == lockSuspended:
|
||||
return ownedVM{}, false, permanentf("VM %d named %s is hibernated (lock suspended); resume or destroy it in Proxmox", r.VMID, req.Name)
|
||||
}
|
||||
return ownedVM{node: r.Node, vmid: r.VMID, cfg: cfg}, true, nil
|
||||
}
|
||||
|
||||
// template finds the clone source and refuses one that is not a template
|
||||
// or brings a Proxmox cloud-init drive, which would compete with the seed
|
||||
// (spec entry 4). The clone copies the template's config, so what the
|
||||
// config or resize step would refuse on the clone is refused here, while no
|
||||
// VM exists that a fixed template could no longer help.
|
||||
func (p *Provider) template(ctx context.Context, rs []vmResource, image string, vmid int, mt provider.ProxmoxMachineType) (vmResource, error) {
|
||||
i := slices.IndexFunc(rs, func(r vmResource) bool { return r.VMID == vmid && r.Type == "qemu" })
|
||||
if i < 0 {
|
||||
return vmResource{}, permanentf("placement.image %q: template VM %d does not exist", image, vmid)
|
||||
}
|
||||
src := rs[i]
|
||||
cfg, err := p.api.VMConfig(ctx, src.Node, vmid)
|
||||
if err != nil {
|
||||
return vmResource{}, fmt.Errorf("reading config of template %d: %w", vmid, err)
|
||||
}
|
||||
if cfg["template"] != "1" {
|
||||
return vmResource{}, permanentf("placement.image %q: VM %d is not a template", image, vmid)
|
||||
}
|
||||
for _, k := range slices.Sorted(maps.Keys(cfg)) {
|
||||
if diskKey.MatchString(k) && strings.Contains(cfg[k], "cloudinit") {
|
||||
return vmResource{}, permanentf("placement.image %q: template %d has a Proxmox cloud-init drive on %s; remove it, the provider attaches its own seed", image, vmid, k)
|
||||
}
|
||||
}
|
||||
shape := ownedVM{node: src.Node, vmid: vmid, cfg: cfg}
|
||||
if _, err := p.configParams(shape, mt, ""); err != nil {
|
||||
return vmResource{}, err
|
||||
}
|
||||
if _, _, _, err := resizeTarget(shape, mt.DiskGiB); err != nil {
|
||||
return vmResource{}, err
|
||||
}
|
||||
return src, nil
|
||||
}
|
||||
|
||||
// allocate is the probe of spec entry 2: the attempt-th VMID for the UID
|
||||
// is cloned onto when the listing shows it free and put through the
|
||||
// ownership check when it is taken, or when the clone finds it taken after
|
||||
// all (F4). The boolean reports an adoption rather than a fresh clone.
|
||||
func (p *Provider) allocate(ctx context.Context, log logr.Logger, rs []vmResource, node string, src vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
|
||||
taken := make(map[int]vmResource, len(rs))
|
||||
for _, r := range rs {
|
||||
taken[r.VMID] = r
|
||||
}
|
||||
rng := p.cfg.VMIDRange
|
||||
probes := min(maxProbes, rng.Max-rng.Min+1)
|
||||
for attempt := range probes {
|
||||
vmid := vmidFor(req.UID, rng.Min, rng.Max, attempt)
|
||||
r, isTaken := taken[vmid]
|
||||
if !isTaken {
|
||||
vm, err := p.clone(ctx, log, node, src, vmid, req)
|
||||
if !isAlreadyExists(err) {
|
||||
return vm, false, err
|
||||
}
|
||||
var found bool
|
||||
if r, found, err = p.resolve(ctx, vmid); err != nil {
|
||||
return ownedVM{}, false, err
|
||||
}
|
||||
if !found {
|
||||
log.V(1).Info("proxmox vmid taken by something other than a VM, probing on", "vmid", vmid, "attempt", attempt)
|
||||
continue
|
||||
}
|
||||
}
|
||||
if r.Type == "qemu" && r.Template == 0 {
|
||||
vm, ours, err := p.inspect(ctx, r, req)
|
||||
if err != nil || ours {
|
||||
return vm, ours, err
|
||||
}
|
||||
}
|
||||
log.V(1).Info("proxmox vmid taken by another vm, probing on", "vmid", vmid, "attempt", attempt)
|
||||
}
|
||||
return ownedVM{}, false, transientf("no free VMID for %s after %d probes in %d..%d", req.Name, probes, rng.Min, rng.Max)
|
||||
}
|
||||
|
||||
// clone is step 4: a full clone carrying the mark, so the VM is never
|
||||
// unmarked (spec entry 8), then its unlocked config. A taken VMID comes
|
||||
// back as the API's error for allocate to judge.
|
||||
func (p *Provider) clone(ctx context.Context, log logr.Logger, node string, src vmResource, vmid int, req provider.CreateRequest) (ownedVM, error) {
|
||||
params := cloneParams{
|
||||
NewID: vmid,
|
||||
Name: req.Name,
|
||||
Description: formatDescription(req.UID, p.now()),
|
||||
Storage: p.cfg.Storage,
|
||||
}
|
||||
if node != src.Node {
|
||||
params.Target = node
|
||||
}
|
||||
upid, err := p.api.Clone(ctx, src.Node, src.VMID, params)
|
||||
if err != nil {
|
||||
return ownedVM{}, fmt.Errorf("cloning template %d to VM %d: %w", src.VMID, vmid, err)
|
||||
}
|
||||
if err := p.waitTask(ctx, log, upid, p.budgets.clone); err != nil {
|
||||
return ownedVM{}, fmt.Errorf("cloning template %d to VM %d: %w", src.VMID, vmid, err)
|
||||
}
|
||||
cfg, err := p.waitUnlocked(ctx, node, vmid, p.budgets.clone)
|
||||
if err != nil {
|
||||
return ownedVM{}, err
|
||||
}
|
||||
return ownedVM{node: node, vmid: vmid, cfg: cfg}, nil
|
||||
}
|
||||
|
||||
// provision is steps 5 to 9, each safe to repeat on a VM a previous attempt
|
||||
// left stopped: the upload overwrites, the config write replaces, and the
|
||||
// resize is skipped once the disk is large enough. Everything that can be
|
||||
// refused from the config alone is checked before the first write.
|
||||
func (p *Provider) provision(ctx context.Context, log logr.Logger, vm ownedVM, mt provider.ProxmoxMachineType, req provider.CreateRequest) error {
|
||||
params, err := p.configParams(vm, mt, req.Name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
disk, size, grow, err := resizeTarget(vm, mt.DiskGiB)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := p.uploadSeed(ctx, log, vm.node, req); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := p.api.SetVMConfig(ctx, vm.node, vm.vmid, params); err != nil {
|
||||
return fmt.Errorf("configuring VM %d: %w", vm.vmid, err)
|
||||
}
|
||||
if grow {
|
||||
upid, err := p.api.Resize(ctx, vm.node, vm.vmid, disk, size)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resizing %s of VM %d to %s: %w", disk, vm.vmid, size, err)
|
||||
}
|
||||
if err := p.waitTask(ctx, log, upid, p.budgets.resize); err != nil {
|
||||
return fmt.Errorf("resizing %s of VM %d to %s: %w", disk, vm.vmid, size, err)
|
||||
}
|
||||
}
|
||||
upid, err := p.api.Start(ctx, vm.node, vm.vmid)
|
||||
if err != nil {
|
||||
return fmt.Errorf("starting VM %d: %w", vm.vmid, err)
|
||||
}
|
||||
if err := p.waitTask(ctx, log, upid, p.budgets.start); err != nil {
|
||||
return fmt.Errorf("starting VM %d: %w", vm.vmid, err)
|
||||
}
|
||||
return p.waitRunning(ctx, vm.node, vm.vmid)
|
||||
}
|
||||
|
||||
// configParams is step 6's write: size, agent, tag and seed, and net0 only
|
||||
// when the config sets a bridge or a VLAN tag (spec entry 7).
|
||||
func (p *Provider) configParams(vm ownedVM, mt provider.ProxmoxMachineType, name string) (map[string]string, error) {
|
||||
params := map[string]string{
|
||||
"cores": strconv.Itoa(int(mt.Cores)),
|
||||
"memory": strconv.FormatInt(mt.MemoryMiB, 10),
|
||||
"agent": "1",
|
||||
"tags": proxyTag,
|
||||
"ide2": seedVolID(p.cfg.ISOStorage, name) + ",media=cdrom",
|
||||
}
|
||||
if p.cfg.Bridge == "" && p.cfg.VLANTag == 0 {
|
||||
return params, nil
|
||||
}
|
||||
net0 := vm.cfg["net0"]
|
||||
if net0 == "" {
|
||||
return nil, permanentf("VM %d has no net0 to apply proxmox.bridge or proxmox.vlanTag to; the template needs a network device", vm.vmid)
|
||||
}
|
||||
params["net0"] = rewriteNet0(net0, p.cfg.Bridge, p.cfg.VLANTag)
|
||||
return params, nil
|
||||
}
|
||||
|
||||
// rewriteNet0 replaces or appends bridge= and tag= in a net0 value, keeping
|
||||
// every other option, the model and the MAC in place. An empty bridge or a
|
||||
// zero tag leaves that option alone.
|
||||
func rewriteNet0(net0, bridge string, tag int) string {
|
||||
want := map[string]string{}
|
||||
if bridge != "" {
|
||||
want["bridge"] = bridge
|
||||
}
|
||||
if tag != 0 {
|
||||
want["tag"] = strconv.Itoa(tag)
|
||||
}
|
||||
opts := strings.Split(net0, ",")
|
||||
for i, opt := range opts {
|
||||
key, _, _ := strings.Cut(opt, "=")
|
||||
if v, ok := want[key]; ok {
|
||||
opts[i] = key + "=" + v
|
||||
delete(want, key)
|
||||
}
|
||||
}
|
||||
for _, key := range []string{"bridge", "tag"} {
|
||||
if v, ok := want[key]; ok {
|
||||
opts = append(opts, key+"="+v)
|
||||
}
|
||||
}
|
||||
return strings.Join(opts, ",")
|
||||
}
|
||||
|
||||
// resizeTarget decides step 7: grow the boot disk to diskGiB when that is
|
||||
// larger than its size= (spec entry 6). A disk is never shrunk.
|
||||
func resizeTarget(vm ownedVM, diskGiB int64) (disk, size string, grow bool, err error) {
|
||||
if diskGiB == 0 {
|
||||
return "", "", false, nil
|
||||
}
|
||||
disk, ok := bootDisk(vm.cfg)
|
||||
if !ok {
|
||||
return "", "", false, permanentf("VM %d has no boot disk to grow to %dG", vm.vmid, diskGiB)
|
||||
}
|
||||
cur, ok := diskBytes(vm.cfg[disk])
|
||||
if !ok {
|
||||
return "", "", false, permanentf("boot disk %s of VM %d has no readable size in %q", disk, vm.vmid, vm.cfg[disk])
|
||||
}
|
||||
if cur >= diskGiB<<30 {
|
||||
return "", "", false, nil
|
||||
}
|
||||
return disk, strconv.FormatInt(diskGiB, 10) + "G", true, nil
|
||||
}
|
||||
|
||||
// bootDisk is the first disk of boot: order=… that is not a CD-ROM, else
|
||||
// the legacy bootdisk key, else the first present of scsi0, virtio0,
|
||||
// sata0 and ide0. Proxmox appends the seed's ide2 to the order (F19), so
|
||||
// CD-ROMs are skipped.
|
||||
func bootDisk(cfg vmConfig) (string, bool) {
|
||||
isDisk := func(k string) bool {
|
||||
v, ok := cfg[k]
|
||||
return ok && diskKey.MatchString(k) && diskOption(v, "media") != "cdrom"
|
||||
}
|
||||
for dev := range strings.SplitSeq(diskOption(cfg["boot"], "order"), ";") {
|
||||
if isDisk(dev) {
|
||||
return dev, true
|
||||
}
|
||||
}
|
||||
if k := cfg["bootdisk"]; isDisk(k) {
|
||||
return k, true
|
||||
}
|
||||
for _, k := range []string{"scsi0", "virtio0", "sata0", "ide0"} {
|
||||
if isDisk(k) {
|
||||
return k, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// diskOption returns the value of key= among a config value's
|
||||
// comma-separated options.
|
||||
func diskOption(value, key string) string {
|
||||
for opt := range strings.SplitSeq(value, ",") {
|
||||
if v, ok := strings.CutPrefix(opt, key+"="); ok {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// diskBytes reads a disk value's size=, in Proxmox's <n>[KMGT] notation
|
||||
// (bytes without a suffix).
|
||||
func diskBytes(disk string) (int64, bool) {
|
||||
s := diskOption(disk, "size")
|
||||
if s == "" {
|
||||
return 0, false
|
||||
}
|
||||
shift := map[byte]uint{'K': 10, 'M': 20, 'G': 30, 'T': 40}
|
||||
unit, ok := shift[s[len(s)-1]]
|
||||
if ok {
|
||||
s = s[:len(s)-1]
|
||||
}
|
||||
n, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil || n < 0 {
|
||||
return 0, false
|
||||
}
|
||||
return int64(n * float64(int64(1)<<unit)), true
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -173,14 +173,8 @@ func (p *Provider) lockName(ctx context.Context, name string) (func(), error) {
|
||||
return release, nil
|
||||
}
|
||||
|
||||
// Create and Delete are stubs until the phases that implement them (Create
|
||||
// in 6, Delete in 7). Each answers Transient, so a Proxy on this provider
|
||||
// retries rather than latching.
|
||||
|
||||
func (p *Provider) Create(_ context.Context, req provider.CreateRequest) (string, error) {
|
||||
return "", p.wrapErr("create", "", transientf("proxmox Create is not implemented yet (request %s)", req.Name))
|
||||
}
|
||||
|
||||
// Delete is a stub until Phase 7. It answers Transient, so a Proxy on this
|
||||
// provider retries rather than latching.
|
||||
func (p *Provider) Delete(_ context.Context, id string) error {
|
||||
return p.wrapErr("delete", id, transientf("proxmox Delete is not implemented yet"))
|
||||
}
|
||||
|
||||
@@ -216,17 +216,12 @@ func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestStubs_answerTransient keeps a Proxy on this provider retrying, not
|
||||
// latched, until Create and Delete are implemented in later phases.
|
||||
// latched, until Delete is implemented in Phase 7.
|
||||
func TestStubs_answerTransient(t *testing.T) {
|
||||
t.Parallel()
|
||||
ctx := context.Background()
|
||||
p := newProvider("proxmox-home", testConfig(), nil)
|
||||
_, createErr := p.Create(ctx, provider.CreateRequest{Name: "proxy-x"})
|
||||
deleteErr := p.Delete(ctx, "vms/150042/proxy-x")
|
||||
for op, err := range map[string]error{"create": createErr, "delete": deleteErr} {
|
||||
if !errors.Is(err, provider.ErrTransient) {
|
||||
t.Errorf("%s error = %v, want ErrTransient", op, err)
|
||||
}
|
||||
if err := p.Delete(context.Background(), "vms/150042/proxy-x"); !errors.Is(err, provider.ErrTransient) {
|
||||
t.Errorf("delete error = %v, want ErrTransient", err)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,9 +2,12 @@ package proxmox
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/go-logr/logr"
|
||||
|
||||
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
|
||||
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud"
|
||||
)
|
||||
@@ -41,3 +44,22 @@ func (p *Provider) buildSeed(req provider.CreateRequest) (*bytes.Buffer, error)
|
||||
}
|
||||
return &iso, nil
|
||||
}
|
||||
|
||||
// uploadSeed puts a freshly built seed on the VM's node and waits for the
|
||||
// copy (spec entry 5). An upload overwrites a file of the same name (F12),
|
||||
// so what a cut-off attempt left is replaced without a delete first.
|
||||
func (p *Provider) uploadSeed(ctx context.Context, log logr.Logger, node string, req provider.CreateRequest) error {
|
||||
iso, err := p.buildSeed(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
volid := seedVolID(p.cfg.ISOStorage, req.Name)
|
||||
upid, err := p.api.UploadISO(ctx, node, p.cfg.ISOStorage, seedISOName(req.Name), iso, int64(iso.Len()))
|
||||
if err != nil {
|
||||
return fmt.Errorf("uploading seed %s: %w", volid, err)
|
||||
}
|
||||
if err := p.waitTask(ctx, log, upid, p.budgets.upload); err != nil {
|
||||
return fmt.Errorf("uploading seed %s: %w", volid, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -89,7 +89,9 @@ func (p *Provider) waitTask(ctx context.Context, log logr.Logger, upid string, b
|
||||
}
|
||||
|
||||
// waitUnlocked polls a VM's config until no task holds its lock and returns
|
||||
// that config, for Create's ownership check (spec entry 2). A VM that
|
||||
// that config, for Create's ownership check (spec entry 2). A suspended lock
|
||||
// is returned at once: it marks a hibernated VM, not a task, and never
|
||||
// clears on its own, so waiting would only burn the clone budget. A VM that
|
||||
// disappears meanwhile, as a failed clone does, returns the does-not-exist
|
||||
// error for the caller to judge.
|
||||
func (p *Provider) waitUnlocked(ctx context.Context, node string, vmid int, budget time.Duration) (vmConfig, error) {
|
||||
@@ -100,10 +102,27 @@ func (p *Provider) waitUnlocked(ctx context.Context, node string, vmid int, budg
|
||||
return false, fmt.Errorf("reading config of VM %d: %w", vmid, err)
|
||||
}
|
||||
cfg = c
|
||||
return c["lock"] == "", nil
|
||||
return c["lock"] == "" || c["lock"] == lockSuspended, nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// waitRunning polls status/current after the start task until QEMU runs
|
||||
// the guest (spec entry 10, step 9). A VM that is stopped again is
|
||||
// Transient at once: it will not come up on its own, and the next attempt
|
||||
// adopts and starts it.
|
||||
func (p *Provider) waitRunning(ctx context.Context, node string, vmid int) error {
|
||||
return pollUntil(ctx, p.budgets.poll, p.budgets.running, fmt.Sprintf("VM %d to run", vmid), func() (bool, error) {
|
||||
st, err := p.api.VMStatus(ctx, node, vmid)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("reading status of VM %d: %w", vmid, err)
|
||||
}
|
||||
if st.Status == "stopped" {
|
||||
return false, transientf("VM %d stopped right after its start", vmid)
|
||||
}
|
||||
return st.Status == "running" && st.QMPStatus == "running", nil
|
||||
})
|
||||
}
|
||||
|
||||
@@ -160,8 +160,10 @@ func TestWaitUnlocked_table(t *testing.T) {
|
||||
cancelled bool
|
||||
wantErr func(error) bool
|
||||
wantCalls int
|
||||
wantLock string
|
||||
}{
|
||||
{name: "unlocked answers at once", wantCalls: 1},
|
||||
{name: "suspended lock is not waited on", lock: lockSuspended, budget: time.Hour, wantCalls: 1, wantLock: lockSuspended},
|
||||
{
|
||||
name: "waits until the lock clears", lock: "clone", wantCalls: 3,
|
||||
onCall: func(f *fakeAPI, n int) {
|
||||
@@ -219,8 +221,8 @@ func TestWaitUnlocked_table(t *testing.T) {
|
||||
}
|
||||
cfg, err := fx.p.waitUnlocked(ctx, fakeNode, testVMID, budget)
|
||||
if tc.wantErr == nil {
|
||||
if err != nil || cfg["name"] != testName || cfg["lock"] != "" {
|
||||
t.Errorf("waitUnlocked = %v, %v; want the unlocked config", cfg, err)
|
||||
if err != nil || cfg["name"] != testName || cfg["lock"] != tc.wantLock {
|
||||
t.Errorf("waitUnlocked = %v, %v; want the config with lock %q", cfg, err, tc.wantLock)
|
||||
}
|
||||
} else if !tc.wantErr(err) || cfg != nil {
|
||||
t.Errorf("waitUnlocked = %v, %v; want no config and the expected error", cfg, err)
|
||||
|
||||
Reference in New Issue
Block a user