proxmox: a Proxmox VE provider built from the provider contract #9

Merged
kacerr merged 13 commits from feat/proxmox-provider into main 2026-09-24 13:12:45 +02:00
9 changed files with 1900 additions and 28 deletions
Showing only changes of commit 34f132d421 - Show all commits
@@ -10,7 +10,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18-
- [x] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 - [x] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2
- [x] Phase 4 — Seam, HTTP client, decorator, fake - [x] Phase 4 — Seam, HTTP client, decorator, fake
- [x] Phase 5 — Get, ListByTag, task helpers - [x] Phase 5 — Get, ListByTag, task helpers
- [ ] Phase 6 — Create - [x] Phase 6 — Create
- [ ] Phase 7 — Delete, Checker, CRD comment - [ ] Phase 7 — Delete, Checker, CRD comment
- [ ] Phase 8 — Contract and observability sweep, integration test, testing.md - [ ] Phase 8 — Contract and observability sweep, integration test, testing.md
- [ ] Phase 9 — Deployment and documentation - [ ] Phase 9 — Deployment and documentation
@@ -657,3 +657,135 @@ Worth noting:
- Nothing in this phase talked to the host. Every Proxmox message a test - Nothing in this phase talked to the host. Every Proxmox message a test
relies on comes from `testdata/` or from fake errors that relies on comes from `testdata/` or from fake errors that
`TestFakeAPI_errorsMatchCaptures` ties to the captures. `TestFakeAPI_errorsMatchCaptures` ties to the captures.
## Phase 6 — Create
`create.go` holds `Create` and its steps: placement, the by-name lookup,
the template check, the VMID probe with its ownership check (`inspect`),
the clone, and `provision` (seed, config, resize, start). `configParams`,
`rewriteNet0`, `resizeTarget`, `bootDisk` and `diskBytes` are pure helpers
with their own table tests. `seed.go` gains `uploadSeed`, which uploads
over any existing file with no delete first (F12). `task.go` gains
`waitRunning`, built on `pollUntil` with `budgets.running`. It finishes
only at `running`/`running`, and a `stopped` status after the start is
Transient at once. The Create stub is gone, and `TestStubs_answerTransient`
now covers only Delete.
The spec won over the plan wherever the Phase 6 notes said it would. The
plan's allocation loop is replaced by entry 2: a taken VMID and a clone
answering F4 both go through the ownership check before any probing, and a
locked or nameless entry is waited on with `waitUnlocked` and the clone
budget. The seed is uploaded without a delete. The resize is a task. `net0`
changes only when `bridge` or `vlanTag` is set, and `tag=` is replaced or
appended in place.
Judgment calls, with spec entries 2, 4 and 10 updated to match:
- **The Phase 5 open question: a `suspended` lock is not waited on.**
`waitUnlocked` now returns at once when it sees `suspended`, because a
hibernated VM is not a task and would otherwise burn the 10-minute clone
budget on every attempt. The ownership check then judges the VM the way
`Get` does. A foreign hibernated VM is probed past. Ours is Permanent
("resume or destroy it"), following kvm2's precedent for a paused or
pmsuspended domain in Create. Transient was the alternative: it would
self-heal once an admin acts, but it would retry indefinitely on
something only an admin can fix. `TestWaitUnlocked_table` has a new row,
and `TestCreate_hibernatedVMIsNotWaitedOn` covers both cases.
- **The template is checked only when a clone is needed.** Spec entry 10
read the template's config in step 2, before the by-name lookup. A retry
for a VM that already exists would then fail Permanent if the template
had been deleted since, which entry 4 explicitly allows. The template
check now runs after the by-name lookup misses, so a repeat Create costs
three calls: resources, config and status
(`TestCreate_adoptionDoesNotNeedTheTemplate`).
- **The template's shape is checked before the clone.** No `net0` while
`bridge` or `vlanTag` is set, or no boot disk (or none with a readable
`size=`) while `diskGiB` is set, is Permanent. Found after the clone, the
VM would already carry the bad config, so fixing the template could not
help that Proxy's next attempt. `template()` runs the same
`configParams` and `resizeTarget` over the template's config that
`provision` later runs over the clone's.
- **An empty `req.Name` is Permanent at placement.** A nameless request
would match nameless mid-clone entries in the by-name lookup.
- **Adopted VMs get one `status/current` read** rather than trusting the
listing's status, which may be stale after a lock wait. A running VM,
whatever its `qmpstatus`, returns its ID at once, as entry 10 says. `Get`
then reports prelaunch as Provisioning and paused as Stopped.
- **After the clone task, Create reads the config through
`waitUnlocked`.** That read supplies the `net0` and boot disk the later
steps need, and it turns a clone that ended but kept its lock (F15) into
a Transient lock wait rather than a refused config write.
- **An F4 whose VMID then resolves to no QEMU VM probes on.** Either the
racer's clone failed and removed it, or a container took it.
- **Errors carry no ID** (`wrapErr("create", "", …)`), since the VMID is
known only after allocation. The messages name the VMID wherever one
exists.
Test names: every plan and entry 21 name exists except
`TestCreate_resizeOnlyWhenLarger`. It is covered by the spec's pair:
`TestCreate_resizeWaitsForTask` for the larger case and
`TestCreate_resizeSkippedWhenNotLarger` for equal, smaller and unset. Tests
beyond both lists: `TestCreate_vmThatStopsAfterStartIsTransientAtOnce`,
`TestCreate_hibernatedVMIsNotWaitedOn`,
`TestCreate_templateShapeRefusedBeforeClone`,
`TestCreate_adoptionDoesNotNeedTheTemplate`,
`TestCreate_cloneTargetAndStorage` (read from the decorator's V(2) clone
line), `TestCreate_adoptedVMStaysOnItsNode`, `TestCreate_outcomeLine`,
`TestCreate_quietAtV0`, `TestRewriteNet0_table`, `TestBootDisk_capturedClone`
and `TestDiskBytes_table`. The only new test helpers are the small ones in
`create_test.go` (`slot`, `newCreateFixture`, `req`/`create`/`createFails`,
`assertRunning`, `callsBefore`, `clearLockOnVMConfig`). Everything else
reuses the Phase 5 fixture and fake. Two tests check config-write contents
without a recording wrapper. `TestCreate_markWrittenInCloneCall` reads the
VM at the start of `UploadISO`, which is after the clone and before the
config write. `TestCreate_noBridgeOverrideLeavesNet0` changes `net0` at the
start of `SetVMConfig`, so a write that carried `net0` would put the old
value back.
A mutation check applied 18 mutations, and each failed at least one test.
They were: no lock wait in the ownership check; taken VMIDs never
inspected; the F4 race skipping the ownership check; no cloud-init drive
check; upload or resize task not waited on; resize at equal size; a
CD-ROM counted as a boot disk; prelaunch counted as running; the suspended
lock waited on; `net0` always written; `bridge=` appended instead of
replaced; duplicate names ignored; a running VM re-provisioned; `target`
always set; a foreign UID adopted; the tag dropped; probes not capped by
the range. The last one was first **missed**. A 4-VMID range with 32
probes wraps, re-inspects each VM eight times and still ends Transient, so
`TestCreate_vmidRangeExhaustedIsTransient` now pins five config reads.
Five mutations did not compile on the first try and were redone. Every
mutation was reverted, and the files were compared identical to backups.
```bash
go test -race -count=2 ./internal/provider/proxmox/ # ok
grep -n 'CloudInit' internal/provider/proxmox/create.go internal/provider/proxmox/seed.go
# seed.go: the pass-through into nocloud.BuildSeedISO; create.go: len(req.CloudInit)
grep -n 'not implemented' internal/provider/proxmox/*.go | grep -v _test # Delete only
grep -n 'context.Background()' internal/provider/proxmox/*.go | grep -v _test.go # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -short -count=1 ./... # all ok but the parked kvm2 flake, below
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 393 before, 519 after
bin/golangci-lint run --disable staticcheck ./internal/provider/proxmox/...
```
Worth noting:
- `go test -short ./...` failed once on the parked kvm2 flake
`TestCheck_hostsCheckedConcurrently` (measured 199.98 ms against a
200 ms floor). Three reruns passed. This phase does not touch kvm2.
- The new lint findings follow the baseline's shape. There are logcheck
hits on `create`, `allocate` and `clone` (ctx plus logger, as in kvm2),
and goconst on test strings. One more unparam than the notes list:
`(*fakeAPI).iso`'s `storage` always receives `fakeISOStorage`, because
this phase is its first caller. `addTemplate`'s hit changed from a
constant VMID to an unused result.
- The config write sets `agent=1` and `tags=proxy_operator` as values. A
template's agent options (such as `fstrim_cloned_disks=1`) and its own
tags are replaced, not merged. That is what entry 10 specifies, and the
guide should say so.
- In the fake, a cross-node clone puts the config on the target at once.
Real Proxmox may move it at the end of the task, in which case a lock
wait on the target node would see `does not exist` and the next attempt
would find the VM. This stays untested, like every cross-node path
(entry 24).
- Nothing in this phase talked to the host.
+21 -7
View File
@@ -72,7 +72,12 @@ lands on the same number: `vmidFor(uid, attempt)` is
seed step (entry 10). It is **foreign** when the name differs: move to seed step (entry 10). It is **foreign** when the name differs: move to
the next attempt. The same name with no mark, or with a different the next attempt. The same name with no mark, or with a different
non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the
operator never takes over a VM it did not create. operator never takes over a VM it did not create. A `suspended` lock (a
VM hibernated to disk) is not waited on: it is not a task and never
clears by itself. Ownership is judged at once, as `Get` does (entry 9):
a foreign hibernated VM is probed past, and ours is `ErrPermanent`
(resume or destroy it), like kvm2's paused domain. Container and
template entries at a VMID are foreign without a config read.
The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after
a crash mid-clone, the retry's by-name lookup misses the nameless VM, but a crash mid-clone, the retry's by-name lookup misses the nameless VM, but
@@ -113,7 +118,12 @@ template may be edited, replaced or deleted without affecting running VMs.
Before cloning, Create reads the template's config and refuses it with Before cloning, Create reads the template's config and refuses it with
`ErrPermanent` when it is not a template (`template` is not 1) or when any `ErrPermanent` when it is not a template (`template` is not 1) or when any
disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`), disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`),
which would compete with the provider's seed (entry 5). The template must which would compete with the provider's seed (entry 5). Because the clone
copies the template's config, what the config or resize step would refuse
on the clone is refused on the template too, before any VM exists: no
`net0` while `bridge` or `vlanTag` is set, and no boot disk, or one
without a readable `size=`, while `diskGiB` is set. A Permanent error
after the clone would leave a VM that fixing the template cannot help. The template must
boot from its first disk, run cloud-init with the NoCloud datasource, and boot from its first disk, run cloud-init with the NoCloud datasource, and
either ship `qemu-guest-agent` or be able to install it (entry 7). either ship `qemu-guest-agent` or be able to install it (entry 7).
@@ -320,11 +330,15 @@ waits for the clone task first.
Steps, each idempotent, in order, under the per-name lock (entry 19): Steps, each idempotent, in order, under the per-name lock (entry 19):
1. Validate placement (entry 3). No backend call yet. 1. Validate placement (entry 3). No backend call yet.
2. `GET /cluster/resources`. Find the template's node; a missing template 2. `GET /cluster/resources` once, then look the name up (entry 2).
is `ErrPermanent`. Read the template's config and refuse it as entry 4 3. When the name finds nothing: find the template's node (a missing
says. template is `ErrPermanent`), read the template's config and refuse it as
3. Find or allocate the VMID (entry 2). An adopted VM that is running entry 4 says, then allocate the VMID (entry 2). The template is checked
returns its ID at once. An adopted VM that is stopped continues at step 5. only here, because a VM that already exists no longer depends on it
(entry 4): a template deleted after its clone must not fail that
clone's adoption. An adopted VM, found by name or by the probe, gets one
`status/current` read. Running, it returns its ID at once. Stopped, it
continues at step 5.
4. Clone and wait (clone budget). 4. Clone and wait (clone budget).
5. Build the seed and upload it to the VM's node, overwriting; wait 5. Build the seed and upload it to the VM's node, overwriting; wait
(upload budget). (upload budget).
+425
View File
@@ -0,0 +1,425 @@
package proxmox
import (
"context"
"fmt"
"maps"
"regexp"
"slices"
"strconv"
"strings"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// maxProbes bounds the VMID probe of spec entry 2.
const maxProbes = 32
// diskKey is a Proxmox disk slot: the keys a boot disk, a CD-ROM or a
// cloud-init drive can sit on.
var diskKey = regexp.MustCompile(`^(ide|sata|scsi|virtio)[0-9]+$`)
// ownedVM is a VM Create has cloned or adopted: where it lives and its
// config as last read, which the config and resize steps start from.
type ownedVM struct {
node string
vmid int
cfg vmConfig
}
// Create is spec entry 10: validate placement, then, under the per-name
// lock, adopt the VM a previous attempt left or clone a new one, and carry
// it through seed, config, resize and start until it runs.
func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (string, error) {
node, mt, tmpl, err := p.placement(req)
if err != nil {
return "", p.wrapErr("create", "", err)
}
log := p.logger(ctx).WithValues("name", req.Name)
log.V(2).Info("proxmox create resolved",
"zone", node, "machineType", req.Placement.MachineType, "cores", mt.Cores, "memoryMiB", mt.MemoryMiB,
"diskGiB", mt.DiskGiB, "image", req.Placement.Image, "templateVMID", tmpl.VMID, "storage", p.cfg.Storage,
"isoStorage", p.cfg.ISOStorage, "bridge", p.cfg.Bridge, "vlanTag", p.cfg.VLANTag, "cloudInitBytes", len(req.CloudInit))
unlock, err := p.lockName(ctx, req.Name)
if err != nil {
return "", p.wrapErr("create", "", err)
}
defer unlock()
vm, err := p.create(ctx, log, node, mt, tmpl, req)
if err != nil {
return "", p.wrapErr("create", "", err)
}
log.V(1).Info("proxmox vm ensured running", "vmid", vm.vmid, "node", vm.node)
return formatProviderID(vm.vmid, req.Name), nil
}
// placement is step 1: every catalogue key is checked before any backend
// call, and an unknown one is Permanent naming the field and the value
// (spec entry 3). Region is ignored.
func (p *Provider) placement(req provider.CreateRequest) (string, provider.ProxmoxMachineType, provider.ProxmoxTemplate, error) {
var (
mt provider.ProxmoxMachineType
tmpl provider.ProxmoxTemplate
)
if req.Name == "" {
return "", mt, tmpl, permanentf("the request has no instance name")
}
node, err := p.node(req.Placement.Zone)
if err != nil {
return "", mt, tmpl, err
}
mt, ok := p.cfg.MachineTypes[req.Placement.MachineType]
if !ok {
return "", mt, tmpl, permanentf("placement.machineType %q is not in proxmox.machineTypes", req.Placement.MachineType)
}
tmpl, ok = p.cfg.Templates[req.Placement.Image]
if !ok {
return "", mt, tmpl, permanentf("placement.image %q is not in proxmox.templates", req.Placement.Image)
}
return node, mt, tmpl, nil
}
// create is steps 2 to 9. The template is checked only when a clone is
// needed: a VM a previous attempt created no longer depends on it (spec
// entry 4), so a template removed since must not fail its adoption.
func (p *Provider) create(ctx context.Context, log logr.Logger, node string, mt provider.ProxmoxMachineType, tmpl provider.ProxmoxTemplate, req provider.CreateRequest) (ownedVM, error) {
rs, err := p.api.ClusterResources(ctx)
if err != nil {
return ownedVM{}, fmt.Errorf("listing cluster resources: %w", err)
}
vm, adopted, err := p.adoptByName(ctx, rs, req)
if err != nil {
return ownedVM{}, err
}
if !adopted {
src, err := p.template(ctx, rs, req.Placement.Image, tmpl.VMID, mt)
if err != nil {
return ownedVM{}, err
}
if vm, adopted, err = p.allocate(ctx, log, rs, node, src, req); err != nil {
return ownedVM{}, err
}
}
log = log.WithValues("vmid", vm.vmid, "node", vm.node)
if adopted {
st, err := p.api.VMStatus(ctx, vm.node, vm.vmid)
if err != nil {
return ownedVM{}, fmt.Errorf("reading status of VM %d: %w", vm.vmid, err)
}
log.V(1).Info("proxmox vm adopted", "status", st.Status, "qmpstatus", st.QMPStatus)
if st.Status == "running" {
return vm, nil
}
}
return vm, p.provision(ctx, log, vm, mt, req)
}
// adoptByName is the by-name half of spec entry 2: two VMs with the name
// are for an admin to sort out, one goes through the ownership check.
func (p *Provider) adoptByName(ctx context.Context, rs []vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
var found []vmResource
for _, r := range rs {
if r.Type == "qemu" && r.Template == 0 && r.Name == req.Name {
found = append(found, r)
}
}
switch len(found) {
case 0:
return ownedVM{}, false, nil
case 1:
return p.inspect(ctx, found[0], req)
}
vmids := make([]string, len(found))
for i, r := range found {
vmids[i] = strconv.Itoa(r.VMID)
}
return ownedVM{}, false, permanentf("VMs %s are all named %s; remove all but one", strings.Join(vmids, ", "), req.Name)
}
// inspect is the ownership check of spec entry 2 on a VMID that is taken.
// It waits out a task's lock (a clone in progress is also nameless), then
// reads the name and the mark: another name is someone else's VM, and our
// name without our mark is never taken over.
func (p *Provider) inspect(ctx context.Context, r vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
cfg, err := p.waitUnlocked(ctx, r.Node, r.VMID, p.budgets.clone)
if err != nil {
return ownedVM{}, false, err
}
if cfg["name"] != req.Name {
return ownedVM{}, false, nil
}
mark := parseDescription(cfg["description"])
switch {
case !mark.Managed:
return ownedVM{}, false, permanentf("VM %d is named %s but carries no proxy-operator mark; it was not created by this operator", r.VMID, req.Name)
case mark.UID != "" && mark.UID != req.UID:
return ownedVM{}, false, permanentf("VM %d named %s belongs to proxy uid %s, not %s", r.VMID, req.Name, mark.UID, req.UID)
case cfg["lock"] == lockSuspended:
return ownedVM{}, false, permanentf("VM %d named %s is hibernated (lock suspended); resume or destroy it in Proxmox", r.VMID, req.Name)
}
return ownedVM{node: r.Node, vmid: r.VMID, cfg: cfg}, true, nil
}
// template finds the clone source and refuses one that is not a template
// or brings a Proxmox cloud-init drive, which would compete with the seed
// (spec entry 4). The clone copies the template's config, so what the
// config or resize step would refuse on the clone is refused here, while no
// VM exists that a fixed template could no longer help.
func (p *Provider) template(ctx context.Context, rs []vmResource, image string, vmid int, mt provider.ProxmoxMachineType) (vmResource, error) {
i := slices.IndexFunc(rs, func(r vmResource) bool { return r.VMID == vmid && r.Type == "qemu" })
if i < 0 {
return vmResource{}, permanentf("placement.image %q: template VM %d does not exist", image, vmid)
}
src := rs[i]
cfg, err := p.api.VMConfig(ctx, src.Node, vmid)
if err != nil {
return vmResource{}, fmt.Errorf("reading config of template %d: %w", vmid, err)
}
if cfg["template"] != "1" {
return vmResource{}, permanentf("placement.image %q: VM %d is not a template", image, vmid)
}
for _, k := range slices.Sorted(maps.Keys(cfg)) {
if diskKey.MatchString(k) && strings.Contains(cfg[k], "cloudinit") {
return vmResource{}, permanentf("placement.image %q: template %d has a Proxmox cloud-init drive on %s; remove it, the provider attaches its own seed", image, vmid, k)
}
}
shape := ownedVM{node: src.Node, vmid: vmid, cfg: cfg}
if _, err := p.configParams(shape, mt, ""); err != nil {
return vmResource{}, err
}
if _, _, _, err := resizeTarget(shape, mt.DiskGiB); err != nil {
return vmResource{}, err
}
return src, nil
}
// allocate is the probe of spec entry 2: the attempt-th VMID for the UID
// is cloned onto when the listing shows it free and put through the
// ownership check when it is taken, or when the clone finds it taken after
// all (F4). The boolean reports an adoption rather than a fresh clone.
func (p *Provider) allocate(ctx context.Context, log logr.Logger, rs []vmResource, node string, src vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
taken := make(map[int]vmResource, len(rs))
for _, r := range rs {
taken[r.VMID] = r
}
rng := p.cfg.VMIDRange
probes := min(maxProbes, rng.Max-rng.Min+1)
for attempt := range probes {
vmid := vmidFor(req.UID, rng.Min, rng.Max, attempt)
r, isTaken := taken[vmid]
if !isTaken {
vm, err := p.clone(ctx, log, node, src, vmid, req)
if !isAlreadyExists(err) {
return vm, false, err
}
var found bool
if r, found, err = p.resolve(ctx, vmid); err != nil {
return ownedVM{}, false, err
}
if !found {
log.V(1).Info("proxmox vmid taken by something other than a VM, probing on", "vmid", vmid, "attempt", attempt)
continue
}
}
if r.Type == "qemu" && r.Template == 0 {
vm, ours, err := p.inspect(ctx, r, req)
if err != nil || ours {
return vm, ours, err
}
}
log.V(1).Info("proxmox vmid taken by another vm, probing on", "vmid", vmid, "attempt", attempt)
}
return ownedVM{}, false, transientf("no free VMID for %s after %d probes in %d..%d", req.Name, probes, rng.Min, rng.Max)
}
// clone is step 4: a full clone carrying the mark, so the VM is never
// unmarked (spec entry 8), then its unlocked config. A taken VMID comes
// back as the API's error for allocate to judge.
func (p *Provider) clone(ctx context.Context, log logr.Logger, node string, src vmResource, vmid int, req provider.CreateRequest) (ownedVM, error) {
params := cloneParams{
NewID: vmid,
Name: req.Name,
Description: formatDescription(req.UID, p.now()),
Storage: p.cfg.Storage,
}
if node != src.Node {
params.Target = node
}
upid, err := p.api.Clone(ctx, src.Node, src.VMID, params)
if err != nil {
return ownedVM{}, fmt.Errorf("cloning template %d to VM %d: %w", src.VMID, vmid, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.clone); err != nil {
return ownedVM{}, fmt.Errorf("cloning template %d to VM %d: %w", src.VMID, vmid, err)
}
cfg, err := p.waitUnlocked(ctx, node, vmid, p.budgets.clone)
if err != nil {
return ownedVM{}, err
}
return ownedVM{node: node, vmid: vmid, cfg: cfg}, nil
}
// provision is steps 5 to 9, each safe to repeat on a VM a previous attempt
// left stopped: the upload overwrites, the config write replaces, and the
// resize is skipped once the disk is large enough. Everything that can be
// refused from the config alone is checked before the first write.
func (p *Provider) provision(ctx context.Context, log logr.Logger, vm ownedVM, mt provider.ProxmoxMachineType, req provider.CreateRequest) error {
params, err := p.configParams(vm, mt, req.Name)
if err != nil {
return err
}
disk, size, grow, err := resizeTarget(vm, mt.DiskGiB)
if err != nil {
return err
}
if err := p.uploadSeed(ctx, log, vm.node, req); err != nil {
return err
}
if err := p.api.SetVMConfig(ctx, vm.node, vm.vmid, params); err != nil {
return fmt.Errorf("configuring VM %d: %w", vm.vmid, err)
}
if grow {
upid, err := p.api.Resize(ctx, vm.node, vm.vmid, disk, size)
if err != nil {
return fmt.Errorf("resizing %s of VM %d to %s: %w", disk, vm.vmid, size, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.resize); err != nil {
return fmt.Errorf("resizing %s of VM %d to %s: %w", disk, vm.vmid, size, err)
}
}
upid, err := p.api.Start(ctx, vm.node, vm.vmid)
if err != nil {
return fmt.Errorf("starting VM %d: %w", vm.vmid, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.start); err != nil {
return fmt.Errorf("starting VM %d: %w", vm.vmid, err)
}
return p.waitRunning(ctx, vm.node, vm.vmid)
}
// configParams is step 6's write: size, agent, tag and seed, and net0 only
// when the config sets a bridge or a VLAN tag (spec entry 7).
func (p *Provider) configParams(vm ownedVM, mt provider.ProxmoxMachineType, name string) (map[string]string, error) {
params := map[string]string{
"cores": strconv.Itoa(int(mt.Cores)),
"memory": strconv.FormatInt(mt.MemoryMiB, 10),
"agent": "1",
"tags": proxyTag,
"ide2": seedVolID(p.cfg.ISOStorage, name) + ",media=cdrom",
}
if p.cfg.Bridge == "" && p.cfg.VLANTag == 0 {
return params, nil
}
net0 := vm.cfg["net0"]
if net0 == "" {
return nil, permanentf("VM %d has no net0 to apply proxmox.bridge or proxmox.vlanTag to; the template needs a network device", vm.vmid)
}
params["net0"] = rewriteNet0(net0, p.cfg.Bridge, p.cfg.VLANTag)
return params, nil
}
// rewriteNet0 replaces or appends bridge= and tag= in a net0 value, keeping
// every other option, the model and the MAC in place. An empty bridge or a
// zero tag leaves that option alone.
func rewriteNet0(net0, bridge string, tag int) string {
want := map[string]string{}
if bridge != "" {
want["bridge"] = bridge
}
if tag != 0 {
want["tag"] = strconv.Itoa(tag)
}
opts := strings.Split(net0, ",")
for i, opt := range opts {
key, _, _ := strings.Cut(opt, "=")
if v, ok := want[key]; ok {
opts[i] = key + "=" + v
delete(want, key)
}
}
for _, key := range []string{"bridge", "tag"} {
if v, ok := want[key]; ok {
opts = append(opts, key+"="+v)
}
}
return strings.Join(opts, ",")
}
// resizeTarget decides step 7: grow the boot disk to diskGiB when that is
// larger than its size= (spec entry 6). A disk is never shrunk.
func resizeTarget(vm ownedVM, diskGiB int64) (disk, size string, grow bool, err error) {
if diskGiB == 0 {
return "", "", false, nil
}
disk, ok := bootDisk(vm.cfg)
if !ok {
return "", "", false, permanentf("VM %d has no boot disk to grow to %dG", vm.vmid, diskGiB)
}
cur, ok := diskBytes(vm.cfg[disk])
if !ok {
return "", "", false, permanentf("boot disk %s of VM %d has no readable size in %q", disk, vm.vmid, vm.cfg[disk])
}
if cur >= diskGiB<<30 {
return "", "", false, nil
}
return disk, strconv.FormatInt(diskGiB, 10) + "G", true, nil
}
// bootDisk is the first disk of boot: order=… that is not a CD-ROM, else
// the legacy bootdisk key, else the first present of scsi0, virtio0,
// sata0 and ide0. Proxmox appends the seed's ide2 to the order (F19), so
// CD-ROMs are skipped.
func bootDisk(cfg vmConfig) (string, bool) {
isDisk := func(k string) bool {
v, ok := cfg[k]
return ok && diskKey.MatchString(k) && diskOption(v, "media") != "cdrom"
}
for dev := range strings.SplitSeq(diskOption(cfg["boot"], "order"), ";") {
if isDisk(dev) {
return dev, true
}
}
if k := cfg["bootdisk"]; isDisk(k) {
return k, true
}
for _, k := range []string{"scsi0", "virtio0", "sata0", "ide0"} {
if isDisk(k) {
return k, true
}
}
return "", false
}
// diskOption returns the value of key= among a config value's
// comma-separated options.
func diskOption(value, key string) string {
for opt := range strings.SplitSeq(value, ",") {
if v, ok := strings.CutPrefix(opt, key+"="); ok {
return v
}
}
return ""
}
// diskBytes reads a disk value's size=, in Proxmox's <n>[KMGT] notation
// (bytes without a suffix).
func diskBytes(disk string) (int64, bool) {
s := diskOption(disk, "size")
if s == "" {
return 0, false
}
shift := map[byte]uint{'K': 10, 'M': 20, 'G': 30, 'T': 40}
unit, ok := shift[s[len(s)-1]]
if ok {
s = s[:len(s)-1]
}
n, err := strconv.ParseFloat(s, 64)
if err != nil || n < 0 {
return 0, false
}
return int64(n * float64(int64(1)<<unit)), true
}
File diff suppressed because it is too large Load Diff
+2 -8
View File
@@ -173,14 +173,8 @@ func (p *Provider) lockName(ctx context.Context, name string) (func(), error) {
return release, nil return release, nil
} }
// Create and Delete are stubs until the phases that implement them (Create // Delete is a stub until Phase 7. It answers Transient, so a Proxy on this
// in 6, Delete in 7). Each answers Transient, so a Proxy on this provider // provider retries rather than latching.
// retries rather than latching.
func (p *Provider) Create(_ context.Context, req provider.CreateRequest) (string, error) {
return "", p.wrapErr("create", "", transientf("proxmox Create is not implemented yet (request %s)", req.Name))
}
func (p *Provider) Delete(_ context.Context, id string) error { func (p *Provider) Delete(_ context.Context, id string) error {
return p.wrapErr("delete", id, transientf("proxmox Delete is not implemented yet")) return p.wrapErr("delete", id, transientf("proxmox Delete is not implemented yet"))
} }
+3 -8
View File
@@ -216,17 +216,12 @@ func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) {
} }
// TestStubs_answerTransient keeps a Proxy on this provider retrying, not // TestStubs_answerTransient keeps a Proxy on this provider retrying, not
// latched, until Create and Delete are implemented in later phases. // latched, until Delete is implemented in Phase 7.
func TestStubs_answerTransient(t *testing.T) { func TestStubs_answerTransient(t *testing.T) {
t.Parallel() t.Parallel()
ctx := context.Background()
p := newProvider("proxmox-home", testConfig(), nil) p := newProvider("proxmox-home", testConfig(), nil)
_, createErr := p.Create(ctx, provider.CreateRequest{Name: "proxy-x"}) if err := p.Delete(context.Background(), "vms/150042/proxy-x"); !errors.Is(err, provider.ErrTransient) {
deleteErr := p.Delete(ctx, "vms/150042/proxy-x") t.Errorf("delete error = %v, want ErrTransient", err)
for op, err := range map[string]error{"create": createErr, "delete": deleteErr} {
if !errors.Is(err, provider.ErrTransient) {
t.Errorf("%s error = %v, want ErrTransient", op, err)
}
} }
} }
+22
View File
@@ -2,9 +2,12 @@ package proxmox
import ( import (
"bytes" "bytes"
"context"
"errors" "errors"
"fmt" "fmt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud"
) )
@@ -41,3 +44,22 @@ func (p *Provider) buildSeed(req provider.CreateRequest) (*bytes.Buffer, error)
} }
return &iso, nil return &iso, nil
} }
// uploadSeed puts a freshly built seed on the VM's node and waits for the
// copy (spec entry 5). An upload overwrites a file of the same name (F12),
// so what a cut-off attempt left is replaced without a delete first.
func (p *Provider) uploadSeed(ctx context.Context, log logr.Logger, node string, req provider.CreateRequest) error {
iso, err := p.buildSeed(req)
if err != nil {
return err
}
volid := seedVolID(p.cfg.ISOStorage, req.Name)
upid, err := p.api.UploadISO(ctx, node, p.cfg.ISOStorage, seedISOName(req.Name), iso, int64(iso.Len()))
if err != nil {
return fmt.Errorf("uploading seed %s: %w", volid, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.upload); err != nil {
return fmt.Errorf("uploading seed %s: %w", volid, err)
}
return nil
}
+21 -2
View File
@@ -89,7 +89,9 @@ func (p *Provider) waitTask(ctx context.Context, log logr.Logger, upid string, b
} }
// waitUnlocked polls a VM's config until no task holds its lock and returns // waitUnlocked polls a VM's config until no task holds its lock and returns
// that config, for Create's ownership check (spec entry 2). A VM that // that config, for Create's ownership check (spec entry 2). A suspended lock
// is returned at once: it marks a hibernated VM, not a task, and never
// clears on its own, so waiting would only burn the clone budget. A VM that
// disappears meanwhile, as a failed clone does, returns the does-not-exist // disappears meanwhile, as a failed clone does, returns the does-not-exist
// error for the caller to judge. // error for the caller to judge.
func (p *Provider) waitUnlocked(ctx context.Context, node string, vmid int, budget time.Duration) (vmConfig, error) { func (p *Provider) waitUnlocked(ctx context.Context, node string, vmid int, budget time.Duration) (vmConfig, error) {
@@ -100,10 +102,27 @@ func (p *Provider) waitUnlocked(ctx context.Context, node string, vmid int, budg
return false, fmt.Errorf("reading config of VM %d: %w", vmid, err) return false, fmt.Errorf("reading config of VM %d: %w", vmid, err)
} }
cfg = c cfg = c
return c["lock"] == "", nil return c["lock"] == "" || c["lock"] == lockSuspended, nil
}) })
if err != nil { if err != nil {
return nil, err return nil, err
} }
return cfg, nil return cfg, nil
} }
// waitRunning polls status/current after the start task until QEMU runs
// the guest (spec entry 10, step 9). A VM that is stopped again is
// Transient at once: it will not come up on its own, and the next attempt
// adopts and starts it.
func (p *Provider) waitRunning(ctx context.Context, node string, vmid int) error {
return pollUntil(ctx, p.budgets.poll, p.budgets.running, fmt.Sprintf("VM %d to run", vmid), func() (bool, error) {
st, err := p.api.VMStatus(ctx, node, vmid)
if err != nil {
return false, fmt.Errorf("reading status of VM %d: %w", vmid, err)
}
if st.Status == "stopped" {
return false, transientf("VM %d stopped right after its start", vmid)
}
return st.Status == "running" && st.QMPStatus == "running", nil
})
}
+4 -2
View File
@@ -160,8 +160,10 @@ func TestWaitUnlocked_table(t *testing.T) {
cancelled bool cancelled bool
wantErr func(error) bool wantErr func(error) bool
wantCalls int wantCalls int
wantLock string
}{ }{
{name: "unlocked answers at once", wantCalls: 1}, {name: "unlocked answers at once", wantCalls: 1},
{name: "suspended lock is not waited on", lock: lockSuspended, budget: time.Hour, wantCalls: 1, wantLock: lockSuspended},
{ {
name: "waits until the lock clears", lock: "clone", wantCalls: 3, name: "waits until the lock clears", lock: "clone", wantCalls: 3,
onCall: func(f *fakeAPI, n int) { onCall: func(f *fakeAPI, n int) {
@@ -219,8 +221,8 @@ func TestWaitUnlocked_table(t *testing.T) {
} }
cfg, err := fx.p.waitUnlocked(ctx, fakeNode, testVMID, budget) cfg, err := fx.p.waitUnlocked(ctx, fakeNode, testVMID, budget)
if tc.wantErr == nil { if tc.wantErr == nil {
if err != nil || cfg["name"] != testName || cfg["lock"] != "" { if err != nil || cfg["name"] != testName || cfg["lock"] != tc.wantLock {
t.Errorf("waitUnlocked = %v, %v; want the unlocked config", cfg, err) t.Errorf("waitUnlocked = %v, %v; want the config with lock %q", cfg, err, tc.wantLock)
} }
} else if !tc.wantErr(err) || cfg != nil { } else if !tc.wantErr(err) || cfg != nil {
t.Errorf("waitUnlocked = %v, %v; want no config and the expected error", cfg, err) t.Errorf("waitUnlocked = %v, %v; want no config and the expected error", cfg, err)