kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

proxmox: deployment files, sample, user guide and doc rows

Browse Source 
The optional proxmox-token Secret (token, ca.pem) mounted at
/etc/proxy-operator/proxmox, make proxmox-token-secret with content
checks that never print the token, a commented proxmox-home entry in
both providers configs, and config/samples/proxy_proxmox.yaml.
TestLoadConfig_proxmoxSampleEntry loads the commented entry from both
files, and a second test ties the sample Proxy's placement to it.

docs/proxmox-provider.md is the user guide: prerequisites and privileges,
cluster setup, config reference, placement, what lands on Proxmox,
logging, tracing, timeouts, health check, error classes, known gaps and
testing. providers-existing.md gains the proxmox column and section;
architecture.md, the README, providers.md and CLAUDE.md gain their rows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-23 21:14:18 +00:00
co-authored by Claude Opus 5.5
parent 6e9ef26dcb
commit 7f2073f94b
14 changed files with 1407 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/providers-existing.md
+256 -64
View File
@@ -1,17 +1,20 @@
# The existing providers
This is the appendix to [provider-contract.md](provider-contract.md): how the
four providers that exist today (kubernetes, gcp, kvm, kvm2) meet the
contract, which profile each one chose on every axis, and what is still
five providers that exist today (kubernetes, gcp, kvm, kvm2, proxmox) meet
the contract, which profile each one chose on every axis, and what is still
missing. It is reference material for whoever writes the next provider, not
a user guide; the user guides are [kvm-provider.md](kvm-provider.md),
[kvm2-provider.md](kvm2-provider.md),
[proxmox-provider.md](proxmox-provider.md),
[gcp-vm-validation.md](gcp-vm-validation.md) and
[gcp-in-specific-project.md](gcp-in-specific-project.md). The README's
"Implemented providers" table stays the short public summary. kvm2 is the
one provider built from the contract itself (clean-room, from
[kvm2-provider-spec.md](kvm2-provider-spec.md), without reading kvm's code),
so its column is also a record of what the contract produced.
"Implemented providers" table stays the short public summary. kvm2 and
proxmox are the two providers built from the contract itself (kvm2
clean-room from [kvm2-provider-spec.md](kvm2-provider-spec.md) without
reading kvm's code, proxmox from
[proxmox-provider-spec.md](proxmox-provider-spec.md)), so their columns are
also a record of what the contract produced.
The per-provider call flows are drawn in
[architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
@@ -27,59 +30,59 @@ Code references are a file path plus a function or symbol name.
One row per axis of contract §4, in the same order, with the profile name
and the symbol that shows it.
| Axis | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) |
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) |
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) |
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) |
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) |
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) |
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) |
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) |
| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) |
| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none |
| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) |
| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) |
| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection |
| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier |
| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.<Method>` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) |
| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` |
| Axis | kubernetes | gcp | kvm | kvm2 | proxmox |
|---|---|---|---|---|---|
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) | provisioning-window: boot plus the guest agent's install at first boot, about 50 s to an address on the test host (`agentAddress`) |
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) | **blocking**: waits through the clone, upload, resize and start tasks and a 60-second poll for running (`provision`, `waitRunning`) |
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: the stop, destroy and seed delete tasks complete before return (`destroy`, `deleteSeed`) |
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) | discovered: the guest agent only, first routable IPv4 in the agent's order (`agentAddress`, `firstIPv4`) |
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) | written-metadata: three `key=value` lines in the VM's description, passed in the clone call itself (`formatDescription`, `parseDescription`) |
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) | written-at-create: `proxy-operator-created-at` in the description, parse error gives zero (`parseDescription`) |
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) | fail-whole for the one cluster-wide `/cluster/resources` call, one unreadable VM skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the description only, no status or agent call (`ListByTag`) |
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) | absent: thin storage overprovisions, so no out-of-space answer exists to match; free space goes to the Checker's details (`classify`) |
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) | error-code: the 500 `Configuration file '…' does not exist`, plus list-and-miss on `/cluster/resources`; Transient inside Create (`isNotExist`, `classifyAPI`) |
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) | **allocated-id**: `vms/<vmid>/<name>`, the VMID from an FNV-1a hash of the UID with forward probing, each taken VMID put through an ownership check with a UID guard (`vmidFor`, `allocate`, `inspect`) |
| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) | create-blocks: cloned, configured, started and polled to running in one `Create`; `qmpstatus: prelaunch` maps to Provisioning (`mapState`) |
| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none | task-handle and backend-lock: every clone, upload, resize, start, stop, destroy and volume delete returns a UPID polled to its end within the call (`waitTask`); a VM's lock is waited on in Create and Transient in Get and Delete (`waitUnlocked`) |
| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: the seed ISO deleted by name (`deleteSeed`, `deleteSeedEverywhere`); the disks go with the destroy's `purge=1&destroy-unreferenced-disks=1` |
| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) | mounted-files: the token from the optional `proxmox-token` Secret, read on every request (`readToken`); the CA from the same Secret, loaded once and cached (`httpClient`) |
| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) | thin-client: `net/http` and `encoding/json` behind the ctx-first `api` seam, one method per endpoint (`api.go`, `httpAPI`) |
| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection | per-call: dial and TLS handshake 10 s, each request `requestTimeoutSeconds`, a budget per task wait (`defaultBudgets`); a cancelled request ends only itself |
| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `proxmox api call` line per request (`observe.go`), V(2) parameters, config summaries and addresses; no wire tier |
| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.<Method>` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) | transport-wrapper: `tracing.RestConfigWrapper()` on the client's transport, parent-gated, one `HTTP <METHOD>` span per request (`newHTTPAPI`) |
| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` | integration-tag: `proxmox_integration_test.go`, `make test-integration-proxmox`; compiled, not yet run against a host |
## 2. Other differences
Rows that are not profile axes but that a new provider author compares
against.
| Feature | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) |
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image |
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) |
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no |
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped |
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined |
| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) |
| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection |
| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got |
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) |
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) |
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter |
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) |
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN |
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself |
| Feature | kubernetes | gcp | kvm | kvm2 | proxmox |
|---|---|---|---|---|---|
| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) | **yes**: a VM with the proxy's name and no mark, or with a different non-empty UID, is Permanent; a marked one with a matching or empty UID is adopted (`inspect`) |
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image | Zone (node), MachineType, Image (a template key) |
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) and uploaded through the API (`uploadSeed`) |
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no | no |
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped | Provisioning, Running, Stopped |
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined | n/a: Delete waits, and a concurrent Get sees the VM (Stopped once stopped) until the destroy removes its config |
| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) | yes (`wrapErr`) |
| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection | each node, concurrently, on the shared HTTP client |
| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got | `isoStorage`, `tokenID`, `pveVersion`, `kernel`, `isoStorageAvailGiB`, `templates`, filled in as far as the check got |
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) | the token yes, read on every request (`readToken`); the CA no, cached after first use |
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) | one `net/http` client, 4 idle connections per host, HTTP/1.1 only; a cancelled request ends only itself |
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter | `fakeAPI` behind `api` (`fake_test.go`), plus `httptest.NewTLSServer` replaying the Phase 0 captures for the real client (`client_test.go`) |
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) |
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN | address on the proxy network (bridge or VLAN from the config); the operator needs a route to it |
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself | whatever the proxy network routes through; not per VM by itself |
## 3. Observability status
| Layer | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators |
| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.<Method>` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx |
| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span |
| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) |
| Layer | kubernetes | gcp | kvm | kvm2 | proxmox |
|---|---|---|---|---|---|
| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators | yes: decorators |
| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.<Method>` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx | yes: `tracing.RestConfigWrapper()` on the transport, parent-gated, one `HTTP <METHOD>` per request |
| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span | yes: every `proxmox api call` line at V(1), with the operation span's `spanID` |
| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) | no; `TestTracing_httpSpansUnderOperationSpan` asserts the parent-child topology in an in-memory recorder |
## 4. kubernetes (`internal/provider/kubernetes`)
@@ -528,20 +531,209 @@ same libvirt.
its backing image or size.
- The sample cloud-init installs Squid with `http_access allow all`.
## 8. Gaps shared by the existing providers
## 8. proxmox (`internal/provider/proxmox`)
- **Adoption is not verified in three of the four.** kubernetes, gcp and
The full user guide is [proxmox-provider.md](proxmox-provider.md) and the
design it was built from is
[proxmox-provider-spec.md](proxmox-provider-spec.md); this section is the
author's view. It is the second provider built from the contract, and the
first on a REST API of the hypervisor's own: no SSH, no libvirt. Every
error text it matches was captured from a PVE 9.2.20 host before the code
was written.
- **Package and seam.** `proxmox.go` (`Provider`, `New`, `withDefaults`,
`node`, `wrapErr`, the per-name lock, the budgets), `create.go`,
`get.go`, `delete.go`, `list.go` (the four methods), `api.go` (the
seam), `client.go` (the production adapter `httpAPI`, the token and CA
loading), `observe.go` (the logging decorator), `task.go` (task, lock
and running waits), `seed.go`, `mark.go`, `id.go`, `state.go`,
`errors.go`, `check.go`; the seed ISO comes from the shared
`internal/provider/nocloud` and the name lock from
`internal/provider/keyedlock`. The seam is `api`: one method per
endpoint, sixteen in all, each with `ctx` first. `New` wraps `httpAPI` in
`observe` once, so provider code never holds an undecorated seam;
`newProvider` is the test constructor.
- **Config.** `ProviderConfig.Proxmox`, the `ProxmoxConfig` type in
`internal/provider/config_proxmox.go`, validated by `validateProxmox`
(the URL shape, node names and storage ids by regexp because they end
up in request paths, `vlanTag`, `vmidRange`, the two catalogues).
`isoStorage` has no default on purpose: the token needs
`Datastore.Allocate` there. Defaults (`vmidRange` 100000 to 199999,
`requestTimeoutSeconds` 30, `installGuestAgent` true) are filled by
`withDefaults`. `New` never makes a request; it warns when the token
file does not exist, or when `insecureSkipTLSVerify` is on.
- **ID.** `formatProviderID(vmid, name)` gives `vms/<vmid>/<name>`, with no
node, because VMs migrate; the node is resolved from
`/cluster/resources` on every call (`resolve`). `parseProviderID`
accepts exactly three segments, the literal `vms`, a decimal VMID
without leading zeros between 100 and 999999999 and a non-empty name,
else `ErrPermanent`. The name is there so `Get` and `Delete` can tell
our VM from another that later got the same VMID.
- **Create.** `placement` checks `Zone` (`node`; optional with one node),
`MachineType` and `Image` (both catalogue keys) before any request, and
logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
the per-name lock, `create`: one `ClusterResources` read;
`adoptByName` (two VMs with the name are Permanent, one goes to
`inspect`); with none, `template` reads the template's config and
refuses a non-template, a cloud-init drive, a missing `net0` when the
config rewrites it, and an unsized boot disk when `diskGiB` is set, all
before any clone; `allocate` probes `vmidFor(uid, attempt)` up to 32
times, cloning onto a free VMID and running `inspect` on a taken one or
on the clone's `config file already exists` answer. `inspect` waits out
a lock (`waitUnlocked`; a clone in progress is also nameless), then
adopts our name with our mark and a matching or empty UID, probes past
another name, and is Permanent for our name without the mark, with
another UID, or hibernated. An adopted VM gets one `VMStatus`: running
returns at once. Otherwise `provision`: `uploadSeed` (rebuilt and
overwritten every attempt), `SetVMConfig` (`configParams`: cores,
memory, `agent=1`, the tag, `ide2`, and `net0` through `rewriteNet0`
when a bridge or VLAN is configured), the resize task when
`resizeTarget` says the boot disk is smaller than `diskGiB`, the start
task, and `waitRunning` (every second up to 60 s, Transient on expiry
or on a stop). `Create` returns only when the VM runs.
- **Get and state.** `get` checks in order: absent from the resources is
NotFound; a lock other than `suspended` is Transient (checked before the
name, because a mid-clone entry has none); `status: unknown` is
Transient; the config's `does not exist` is NotFound, a config lock
Transient; another name or no mark is NotFound; hibernated is Stopped
without further calls. Then `VMStatus` and `mapState(status,
qmpstatus, ip)`:
| `status` | `qmpstatus` | agent | Provider state |
|---|---|---|---|
| running | running | an IPv4 | Running |
| running | running | 200 without an IPv4, or `QEMU guest agent is not running` | Provisioning |
| running | running | `No QEMU guest agent configured` | Stopped |
| running | prelaunch | not asked | Provisioning |
| running | paused, suspended, io-error, anything else | not asked | Stopped |
| stopped, anything else | any | not asked | Stopped |
Any other agent answer, including `VM <vmid> is not running`, is
Transient. `UID` and `CreatedAt` are read from the description on every
call. A running `Get` is four calls: resources, config, status, agent.
- **Delete.** Under the per-name lock, `destroy`: `resolve`; a lock other
than `suspended` or `status: unknown` is Transient; the config's `does
not exist`, another name or no mark leave the VM alone; our VM
hibernated is Permanent, because the destroy refuses it and the lock
never clears. Otherwise stop (unless stopped) and destroy with
`purge=1&destroy-unreferenced-disks=1`, each a task waited on through
`runTask`, where `does not exist` means gone. Then `deleteSeed` on the
VM's node, where a failure fails the call (a 403 for
`Datastore.Allocate` is Permanent), or, with no VM of ours destroyed,
`deleteSeedEverywhere` over every configured node, where a node that
fails is logged and skipped unless ctx ended. A missing volume still
ends in a successful task, so no message is matched. Nothing is left
when nil is returned.
- **ListByTag.** One `ClusterResources` covers every node, unreachable
ones included (their guests are listed with `status: unknown`).
`isCandidate` keeps QEMU VMs that are not templates and carry the tag
`proxy_operator` or a name starting `proxy-`; a nameless entry (a clone
in progress) is not a candidate. One `VMConfig` per candidate; the mark
in the description decides, and the `Instance` is `ID`, `UID` and
`CreatedAt` only. A candidate whose config cannot be read is skipped at
V(1), unless ctx ended, which fails the call; the resources call failing
fails the whole call.
- **Errors.** `classify(op, err)`: an error already carrying a class keeps
it; then `apiError`s by `classifyAPI` (every 4xx Permanent; the 500
`does not exist` NotFound, Transient when `op == "create"`; every other
status Transient, including the lock, the taken VMID and pveproxy's 595
and 596); TLS verification failures (`tls.CertificateVerificationError`
and the three `x509` types) Permanent; everything else Transient. The
500 matchers (`isNotExist`, `isLocked`, `isAlreadyExists`, the three
agent ones) are anchored to the 500 status and each is backed by a
captured response. `permanentf` and `transientf` classify at the
source: placement, the template checks, the ownership check, a
malformed token file or CA file (Permanent); a missing token or CA file,
an expired wait, a failed task (Transient). Nothing is Quota.
- **Check.** `Check` runs `checkNode` per node concurrently on the shared
client. `probeNode` reads the token file for `tokenID` (no request when
it is unusable), then `Version`, `NodeStatus`, `checkStorage` (a 500
whatever its text, inactive, or no `iso` content is `StorageNotFound`)
and `checkTemplates` (each catalogue VMID exists and is a template,
else `TemplateNotFound`). `checkNode` answers `Timeout` whenever the
round's ctx ended; otherwise `checkReason`: the two failures above,
`errCredentials` `CredentialsUnavailable`, `errCAFile` or a TLS
verification failure `CertificateInvalid`, 401 `Unauthorized`, 403
`Forbidden`, 595 and 596 `Unreachable`, any other status `APIError`, a
`net.OpError` `Unreachable` (before the deadline, so a dial out of its
own budget is a node that did not answer), a deadline or cancellation
`Timeout`, a cut connection `Unreachable`.
- **Connections.** One `http.Client`, built on first use once the CA
loads (`httpClient`), shared by every call and the Checker: dial and
TLS handshake 10 s each, `Client.Timeout` `requestTimeoutSeconds`, 4
idle connections per host kept 90 s, HTTP/2 disabled. A cancelled
request ends only itself. The token file is read by `readToken` on
every request.
- **Logging.** Logger from the context, named `proxmox`, with `provider`,
and `vmid`, `name` and `node` on lines about an instance. V(1): one line
per operation outcome (`proxmox vm ensured running`, `proxmox vm
fetched`, `proxmox vm deleted` with `destroyed`, `proxmox vms listed`,
`proxmox node checked`), one `proxmox api call` line per seam call from
`observe` (call, identifiers, UPID, duration, `httpStatus` and `error`),
`proxmox task failed`, and the no-op lines. V(2): request parameters,
config summaries, agent addresses, the resolved Create parameters.
Cloud-init, the token, the CA and the upload body are never logged.
- **Tracing.** No span of its own: `newHTTPAPI` wraps the transport in
`tracing.RestConfigWrapper(opts...)`, so each request is an
`HTTP <METHOD>` client span under the caller's span, and none without
one. `observe` only logs, so nothing is counted twice.
- **Tests.** Pure pieces (`id_test.go`, `mark_test.go`, `state_test.go`,
`seed_test.go`, `errors_test.go` with the classify table over the
captures, `api_test.go`); `client_test.go` (the real client over
`httptest.NewTLSServer`: the token never in errors or logs, x509
failures Permanent, cancellation, the child span); `create_test.go`,
`get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`,
`proxmox_test.go`, `check_test.go`, `observe_test.go` (every seam
method logs exactly one line, by reflection) and `provider_test.go`
(the contract §11 sweep, named test by test in the spec's entry 21).
The double is `fakeAPI` in `fake_test.go`: VMs, tasks and ISOs in
memory, a clone that leaves a nameless locked entry until its task
ends, per-method errors and blocking, a call log; its error texts are
checked against the captures. Real backend:
`proxmox_integration_test.go` behind `//go:build integration`, driven
by `PROXMOX_TEST_*` variables and `make test-integration-proxmox`,
described in
[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host);
it compiles but has not run against a host yet.
- **Deployment.** The optional `proxmox-token` Secret (keys `token` and
`ca.pem`) mounted at `/etc/proxy-operator/proxmox`
(`config/manager/manager.yaml`), `make proxmox-token-secret`, a
commented `proxmox-home` entry in both providers configs (loaded by
`TestLoadConfig_proxmoxSampleEntry`), `config/samples/proxy_proxmox.yaml`,
the `proxmox` type in the `status.type` comment of
`api/v1alpha1/providerhealth_types.go`. No RBAC, and no
`hack/providers-dev.yaml` entry, since it needs credentials.
- **Known gaps** (see also [proxmox-provider.md § Known gaps](proxmox-provider.md#known-gaps)).
- `Create` blocks until the VM runs.
- A clone lock that never clears makes every Create for that Proxy wait
out the 10-minute clone budget; `qm unlock` by hand.
- GC blind spot: a seed ISO whose VM was destroyed outside the operator
is invisible to GC, and a seed on node-local storage stays behind
when its VM migrated.
- The VMID range is shared with humans; a hand-made VM with a proxy's
name is Permanent.
- Cross-node clones (`target`) need shared storage and are untested.
- A hibernated VM of ours latches its Proxy in Create and in Delete.
- The CA is cached after first use; the token is not.
- Never run against a real host yet; three answers were never captured
(the storage status for a missing storage, a destroy on a hibernated
VM, a stop on a paused VM).
## 9. Gaps shared by the existing providers
- **Adoption is not verified in three of the five.** kubernetes, gcp and
kvm adopt any existing instance with the right name without checking
`LabelUID`; kvm2 returns `ErrPermanent` when the UID differs.
- **The Create comment does not match kvm and kvm2.** The interface comment
in `internal/provider/provider.go` says `Create` returns as soon as the
request is submitted; both libvirt providers block until the domain has
started.
`LabelUID`; kvm2 and proxmox return `ErrPermanent` when the UID differs.
- **The Create comment does not match kvm, kvm2 and proxmox.** The
interface comment in `internal/provider/provider.go` says `Create`
returns as soon as the request is submitted; the three VM providers on
your own hypervisors block until the VM has started.
- **`Stopped` and `Terminated` are produced inconsistently.** kubernetes
never produces Stopped, kvm and kvm2 never produce Terminated, and the
reconciler treats them the same.
never produces Stopped, kvm, kvm2 and proxmox never produce Terminated,
and the reconciler treats them the same.
- **Different `ListByTag` partial-failure policies.** gcp leaves failed
parts out, kvm and kvm2 fail the whole list. Both are safe.
parts out, kvm and kvm2 fail the whole list, and proxmox fails it when
its one cluster-wide listing fails. Both are safe.
- **Stale `mock` comments** in `internal/provider/provider.go`,
`internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
(and so the generated CRD's `spec.provider` description).
@@ -555,10 +747,10 @@ same libvirt.
log lines. kvm2 has them, but only an in-memory recorder checks their
topology. Only kubernetes is tested against a real collector, and that
test checks only the operation spans; nothing checks that gcp's SDK spans,
the kubernetes API spans or kvm2's spans nest under `provider.*` in a
real collector.
the kubernetes API spans, kvm2's spans or proxmox's HTTP spans nest under
`provider.*` in a real collector.
## 9. Proposed features
## 10. Proposed features
Listed only. Nothing below is designed or implemented, except where a
bullet says kvm2 already does it.
@@ -576,7 +768,7 @@ bullet says kvm2 already does it.
backend, so a new provider gets the contract tests for free.
- **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent`
on a mismatch instead of adopting an instance that belongs to someone else.
kvm2 does this; the other three do not.
kvm2 and proxmox do this; the other three do not.
- **Report asynchronous create failures:** gcp keeps the operation name, and
`Get` reports the failure instead of NotFound.
- **Capacity or quota signals** for kvm, such as pool free space and host