kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add the appendix on the existing providers

Browse Source 
Profile table keyed on the contract's axis codes, per-provider reference
sections for kubernetes, gcp and kvm, shared gaps and proposed features.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-16 11:36:03 +00:00
co-authored by Claude Fable 5.1
parent d61868fe2d
commit 2d959af313
2 changed files with 427 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/providers-existing.md
+392
View File
@@ -0,0 +1,392 @@
# The existing providers
This is the appendix to [provider-contract.md](provider-contract.md): how the
three providers that exist today (kubernetes, gcp, kvm) meet the contract,
which profile each one chose on every axis, and what is still missing. It is
reference material for whoever writes the next provider, not a user guide;
the user guides are [kvm-provider.md](kvm-provider.md),
[gcp-vm-validation.md](gcp-vm-validation.md) and
[gcp-in-specific-project.md](gcp-in-specific-project.md). The README's
"Implemented providers" table stays the short public summary.
The per-provider call flows are drawn in
[architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
the health checks are tabulated in
[§11](architecture.md#11-provider-health-checks-internalproviderhealth), and the
judgment calls behind several rows below are recorded under its "Decisions"
list. They are linked, not repeated.
Code references are a file path plus a function or symbol name.
## 1. Profile table
One row per axis of contract §4, in the same order, with the profile name
and the symbol that shows it.
| Axis | kubernetes | gcp | kvm |
|---|---|---|---|
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) |
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) |
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) |
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) |
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) |
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) |
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) |
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) |
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` |
| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning |
| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none |
| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) |
| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) |
| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection |
| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier |
| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans |
| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` |
## 2. Other differences
Rows that are not profile axes but that a new provider author compares
against.
| Feature | kubernetes | gcp | kvm |
|---|---|---|---|
| Create verifies the adopted instance's UID | no | no | no |
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image |
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no |
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped |
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a |
| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes |
| Checker target | `apiserver` | the project | each host, concurrently |
| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` |
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load |
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) |
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` |
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN |
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself |
## 3. Observability status
| Layer | kubernetes | gcp | kvm |
|---|---|---|---|
| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators |
| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** |
| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) |
| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no |
## 4. kubernetes (`internal/provider/kubernetes`)
- **Package and seam.** `kubernetes.go`, `pod.go`, `check.go`. There is no
seam interface: `newWithClient` injects a controller-runtime
`client.Client`, and tests use the fake client. `New` calls
`NewWithTransportWrapper`, which builds its own uncached client from
`ctrl.GetConfig()` with an optional `transport.WrapperFunc` for tracing.
- **Config.** `KubernetesConfig.Image` only, defaulting to the Squid image
constant in `kubernetes.go`.
- **ID.** `providerID(namespace, name)` gives `namespace/name`;
`parseProviderID` uses `cache.SplitMetaNamespaceKey` and rejects an empty
namespace with `ErrPermanent`. An empty name part is not rejected.
- **Create.** `buildPod` (pure) then `client.Create`; `IsAlreadyExists` is
success. Only `Name`, `Namespace`, `UID` and `Port` are used. The Squid
config is generated by `squidConf` from the port and passed through the
`SQUID_CONF` environment variable, so there is exactly one object per
proxy.
- **Get and state.** `stateFromPod`:
| Pod phase | PodIP | Provider state |
|---|---|---|
| Pending | any | Provisioning |
| Running | empty | Provisioning |
| Running | set | Running |
| Succeeded, Failed, Unknown | any | Terminated |
`instanceFromPod` sets `IP` only when the state is Running. A Pod with a
deletion timestamp still reports its phase. `UID` from the `LabelUID`
label, `CreatedAt` from `CreationTimestamp`.
- **Delete.** `client.Delete`, `IsNotFound` ignored.
- **ListByTag.** One `client.List` across all namespaces with
`MatchingLabels{LabelManaged: LabelManagedYes}`; this is what the
cluster-wide Pod RBAC is for.
- **Errors.** `classify`: `IsNotFound` NotFound; `IsTooManyRequests`,
`IsServerTimeout`, `IsTimeout` Transient; `IsForbidden`, `IsInvalid`,
`IsBadRequest`, `IsUnauthorized` Permanent; default Transient. Nothing is
Quota, because a quota 403 and an RBAC 403 cannot be told apart.
- **Check.** Lists at most one managed Pod cluster-wide (`client.Limit(1)`),
proving read and list access, not create. Target `apiserver`, address the
rest config host. `checkReason`: context errors and timeouts
`ReasonTimeout`; `IsUnauthorized`; `IsForbidden`; any `APIStatus`
`ReasonAPIError`; else `ReasonUnreachable`.
- **Logging and tracing.** No log lines in the package. Child spans come
from the parent-gated transport wrapper.
- **Tests.** `kubernetes_test.go`, `pod_test.go`, `check_test.go` over the
fake client; the kind e2e (`test/e2e`) is the real-backend run.
- **Deployment.** RBAC markers for Pods live next to the reconciler's in
`internal/controller/proxy_controller.go`. `hack/providers-dev.yaml` holds
only this provider, for `make run-dev`.
- **Known gaps.**
- `Error.Provider` is hardcoded to `"kubernetes"`, so errors and logs show
the wrong name when the configured name differs.
- `CloudInit` and all of `Placement` are ignored.
- No quota class: a quota 403 is Permanent.
- A Pod being deleted still reports its phase, so it can read as Running.
- No securityContext, no resources, and `http_access allow all` in
`pod.go`.
- No logging in the package, and no Checker `Details`.
- No `OwnerReference` on Pods: a bypassed finalizer leaves the Pod behind.
- The Pod RBAC is always installed, even when no kubernetes provider is
configured.
## 5. gcp (`internal/provider/gcp`)
- **Package and seam.** `gcp.go`, `insert.go`, `errors.go`, `check.go`,
`wirelog.go`. The seam is `instancesAPI` (`Insert`, `Get`, `Delete`,
`AggregatedList`, `Probe`); `realInstances` adapts the SDK client and
`newWithAPI` is the test constructor. `New` calls `NewWithWireOptions`,
which takes the `--gcp-wire-log-full-payloads` flag from `cmd/main.go`.
- **Config.** `GCPConfig`: `project` required; `network`, `networkTag`,
`diskSizeGb` defaulted in `withDefaults`.
- **ID.** `formatProviderID(zone, name)` gives
`zones/<zone>/instances/<name>`; `parseProviderID` rejects anything else
with `ErrPermanent`. The project comes from the config.
- **Create.** Validates `Zone`, `MachineType` and `Image` up front and
returns `ErrPermanent` naming the missing field. `buildInsertRequest`
(pure) builds one NIC with an ephemeral external IP (`ONE_TO_ONE_NAT`) and
a boot disk with `AutoDelete`, labelled with both marks. `Insert` is
submitted and never awaited; 409 (`isAlreadyExists`) is success.
- **Get and state.** `mapState(status, ip)`:
| Instance status | NAT IP | Provider state |
|---|---|---|
| PROVISIONING, STAGING, REPAIRING | any | Provisioning |
| RUNNING | empty | Provisioning |
| RUNNING | set | Running |
| STOPPING, STOPPED, SUSPENDING, SUSPENDED | any | Stopped |
| TERMINATED | any | Terminated |
| anything else | any | Stopped |
`toInstance` reads the NAT IP of the first access config of the first
NIC, `UID` from the label, and `CreatedAt` from `CreationTimestamp` with
the parse error dropped (zero time on failure).
- **Delete.** Submits a delete operation and returns; 404 (`isNotFound`) is
success. Every deletion poll submits a new operation.
- **ListByTag.** `AggregatedList` with `managedFilter` on the managed label
only and `ReturnPartialSuccess`, so an unreachable zone is left out
silently; the zone is recovered from the URL by `lastPathSegment`.
- **Errors.** `classify`: non-`googleapi` errors Transient; 404 NotFound;
429 Quota; 403 with reason `quotaExceeded` or `rateLimitExceeded`
(`hasReason`) Quota; 400, 401 and other 403s Permanent; other codes
Transient. `logAPIError` records the HTTP status and reasons at V(1)
because `classify` collapses them.
- **Check.** `Probe` makes one `AggregatedList` page with `maxResults=1`
and the managed filter: credentials, the Compute API and
`compute.instances.list` in one call. Target is the project, address
`compute.googleapis.com`. `checkReason`: context errors `ReasonTimeout`;
non-`googleapi` errors `ReasonUnreachable`; 401; 403; else
`ReasonAPIError`.
- **Logging.** `logger(ctx)` from the context, named `gcp` with
`provider`. V(2) for the curated insert request (never the user-data),
V(1) per operation outcome, V(2) per listed instance. The wire log in
`wirelog.go` runs at V(5) with payloads elided unless the flag is set.
- **Tracing.** The SDK adds `otelhttp` itself unless telemetry is disabled;
not parent-gated.
- **Tests.** `insert_test.go` (pure builder), `errors_test.go` (code
table), `gcp_test.go` (IDs, 409, state mapping, list filter and partial
success) over `fakeAPI`; `check_test.go` also drives the real SDK client
against `httptest` to prove exactly one request with `maxResults=1`.
Real backend: the manual runbook in [gcp-vm-validation.md](gcp-vm-validation.md).
- **Deployment.** Credentials by ADC; the runbook mounts a service-account
key and points `GOOGLE_APPLICATION_CREDENTIALS` at it. A commented entry
in both providers configs, `config/samples/proxy_gcp.yaml`, and demo
scripts under `docs/demo/`.
- **Known gaps.**
- Asynchronous insert failures are never read: an insert the API accepts
but that fails later (quota, bad image) is seen only as NotFound on the
next `Get`, and the reconciler creates again, so the reason never
reaches status. [gcp-vm-validation.md](gcp-vm-validation.md#gotchas)
still says such errors do reach status.
- External IP only: a VM without an external NAT IP stays Provisioning
forever.
- No firewall management: the rule for the proxy port is a manual
prerequisite, and `req.Port` is unused.
- The SDK client is never closed, and `Placement.Region` is unused.
- Wrong label value in the docs: `gcp-vm-validation.md` filters on
`labels.proxy-operator-managed=yes`; the value is `true`.
- The sample cannot work: `config/samples/proxy_gcp.yaml` uses a
`debian-cloud` image, which has no cloud-init, so its Squid install never
runs ([gcp-in-specific-project.md](gcp-in-specific-project.md) uses
Ubuntu for that reason).
## 6. kvm (`internal/provider/kvm`)
The full user guide is [kvm-provider.md](kvm-provider.md); this section is
the author's view.
- **Package and seam.** `kvm.go` (the four methods), `conn.go` (SSH dial,
`hostPool`, `libvirtAPI`), `image.go` (`imageCache`), `seed.go` (NoCloud
ISO and vendor-data), `domainxml.go`, `volume.go`, `metadata.go`, `id.go`,
`errors.go`, `check.go`, `trace.go`. The seam is `libvirtAPI`, the twenty
go-libvirt calls the provider uses; `fake_test.go` implements it in memory
and `newWithDialer` injects the dialer and the HTTP client.
- **Config.** `KVMConfig` in `internal/provider/config_kvm.go`:
`sshKeyFile`, `knownHostsFile` or `insecureIgnoreHostKey`, `diskSizeGb`,
`vendorData`, `installGuestAgent`, `images` (`url`, `sha256`, `format`,
`firmware`), `machineTypes` (`vcpus`, `memoryMiB`), `hosts` (`name`,
`address`, `user`, `socket`, `pool`, `bridge`). `validateKVM` restricts
image keys and host names with `kvmIdentifier` because they become a volume
name and an ID segment.
- **ID.** `formatProviderID(host, name)` gives `hosts/<host>/domains/<name>`.
- **Create.** `host` resolves `Placement.Zone` (optional with one host),
the machine type must be a catalogue key, `resolveImage` accepts a
catalogue key or an http(s) URL. Then, inside one `hostPool.call`:
`DomainLookupByName`, and if found `ensureRunning`; otherwise
`imageCache.ensure` (download on first use, streamed through the operator
into the pool with a sha256 marker volume), `ensureOverlay` (reused if it
exists), `writeSeed` (always rebuilt: a cut-off upload must not be
trusted), `defineDomain` with the metadata element, `ensureRunning`.
`Create` returns only when the domain is running.
- **Get and state.** `mapState(state, ip)`:
| libvirt state | IP | Provider state |
|---|---|---|
| nostate | any | Provisioning |
| running, blocked | empty | Provisioning |
| running, blocked | set | Running |
| paused, shutdown, shutoff, crashed, pmsuspended | any | Stopped |
`address` asks the ARP table first, then the guest agent, and
`firstIPv4` drops loopback, link-local and unspecified addresses. Metadata
is read back by `metadata` and parsed by `parseInstanceMetadata`; a domain
without the element reports an empty UID.
- **Delete.** `removeDomain` (destroy, tolerating not running or gone; then
`DomainUndefineFlags` with NVRAM, managed save and snapshot metadata) and
`removeVolumes` (overlay and seed by name, a missing pool is a no-op). A
host no longer in the config is a logged no-op success.
- **ListByTag.** `ConnectListAllDomains` on every host; `managedInstance`
keeps domains whose metadata says managed and skips a domain whose
metadata or state cannot be read. One unreachable host fails the whole
list.
- **Errors.** `classify`: `isClassified` errors keep their class;
`ErrNoDomain`, `ErrNoStorageVol` NotFound; XML, unsupported,
invalid-argument, auth and access errors, and a missing pool, Permanent;
everything else Transient, including SSH handshake and auth failures.
Nothing is Quota. `logAPIError` adds `libvirtErrorFields`.
- **Check.** `checkHost` per host concurrently through `hostPool.probe`,
not `call`, so a check that runs out of time abandons its RPCs instead of
closing the connection a reconcile may be using. Two cheap RPCs
(`ConnectGetLibVersion`, `ConnectGetHostname`) then the pool must exist
and be running. Reasons: the shared ones plus `SSHMaterialUnavailable`,
`SSHHandshakeFailed`, `LibvirtConnectFailed`, `LibvirtError`,
`PoolNotFound`, `PoolNotActive`, chosen by `checkReason` from the
`dialError` stage. Details: `user`, `pool`, `hostname`, `libvirtVersion`,
`poolState`, `poolCapacity`, `poolAvailable`.
- **Connections.** `hostPool` keeps one SSH plus libvirt connection per
host, dialled lazily (`lazySSH.dial`, `sshAuth.dial`, `dialSSH`,
`connectLibvirt`), with a 15s `keepalive`. `hostEntry.acquire` is a
one-slot semaphore so waiting can end with the caller's context. A
cancelled `call` closes the shared connection to unblock the RPC;
`probe` abandons it and leaves the connection open. The SSH material is
read once by `lazySSH.load` and cached.
- **Logging.** Logger from the context, named `kvm`, with `provider`,
`host` and `name`. V(1): one line per operation outcome and one
`libvirt call` line per RPC from the `tracingAPI` decorator in `trace.go`
(rpc name, identifiers, duration, error fields). V(2): domain and volume
XML, metadata, addresses. Upload and download bodies are never logged.
- **Tracing.** No child spans; `trace.go` is a log decorator.
- **Tests.** Pure builders (`domainxml_test.go`, `volume_test.go`,
`seed_test.go`, `id_test.go`, `metadata_test.go`); `errors_test.go`,
`conn_test.go` (pool behaviour and a real in-process SSH server);
`image_test.go`, `kvm_test.go` (Create resumes after every partial step,
every Delete already-gone case, ListByTag policies, logging tiers, a
cloud-init sentinel never logged); `trace_test.go`; `check_test.go`. Real
backend: `kvm_integration_test.go` behind `//go:build integration`, driven
by `KVM_TEST_*` variables and `make test-integration-kvm`, described in
[testing.md](testing.md#the-kvm-integration-test-real-hypervisor).
- **Deployment.** The optional `kvm-ssh` Secret mounted at
`/etc/proxy-operator/kvm` (`config/manager/manager.yaml`),
`make kvm-ssh-secret`, a commented entry in both providers configs,
`config/samples/proxy_kvm.yaml`.
- **Known gaps** (see also [kvm-provider.md § Known gaps](kvm-provider.md#known-gaps)).
- No quota or capacity detection.
- `Create` blocks, including the first image download on each host.
- GC blind spots: volumes without a domain are invisible to GC, and so are
VMs on a host removed from the config, which `Delete` treats as done.
- SSH material is cached after the first successful load, so rotating
keys needs a restart.
- Cancelling one operation cuts off the others on that host: a cancelled
`call` closes the shared per-host connection.
- SSH auth failures retry forever: SSH authentication and host-key
failures are Transient, while libvirt auth errors are Permanent.
- `Create` can return `ErrNotFound` when a volume or domain disappears
during the call.
- No reused-overlay check: an existing overlay is reused without checking
its backing image or size.
- Sample mismatch: `config/samples/proxy_kvm.yaml` places on host `nas1`,
while both providers configs define only `beelink`.
- Doc drift in kvm-provider.md: it says ARP is the fallback, but the code
tries ARP first; it says `host` and `name` are on every log line, but
the check and list paths lack them.
## 7. Gaps shared by the existing providers
- **Adoption is not verified.** None of the three checks `LabelUID` when
`Create` finds an existing instance with the right name.
- **The Create comment does not match kvm.** The interface comment in
`internal/provider/provider.go` says `Create` returns as soon as the
request is submitted; kvm blocks until the domain has started.
- **`Stopped` and `Terminated` are produced inconsistently.** kubernetes
never produces Stopped, kvm never produces Terminated, and the reconciler
treats them the same.
- **Different `ListByTag` partial-failure policies.** gcp leaves failed
parts out, kvm fails the whole list. Both are safe.
- **Stale `mock` comments** in `internal/provider/provider.go`,
`internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
(and so the generated CRD's `spec.provider` description).
- **GC does not know about installations.** Two clusters sharing one project
or hypervisor delete each other's instances. This was a merge-blocker in
the 2026-08-10 PR review
([docs/reviews](reviews/2026-08-10-1134-proxy-operator-pr-review.md)) and
is still open.
- **Tracing.** kvm has no child spans, so a kvm trace shows `provider.create`
and nothing under it; correlation works only through `traceID` on its V(1)
log lines. Only kubernetes is tested against a real collector, and that
test checks only the operation spans; nothing checks that gcp's SDK spans
or the kubernetes API spans nest under `provider.*`.
## 8. Proposed features
Listed only. Nothing below is designed or implemented.
- **Child spans in kvm:** extend the `tracingAPI` decorator in
`kvm/trace.go`, which already wraps every libvirt call, to also start a
span per RPC, and add spans for the SSH dial and the image download. The
seam's methods take no `ctx` today (`hostPool.call` hands `fn` only the
API), so the span context has to be plumbed in.
- **Tempo e2e checks for child spans**, and for gcp and kvm through their
real backend runbooks.
- **A shared conformance suite:** for example `internal/provider/providertest`,
which runs the contract §11 tests against any provider plus a fake
backend, so a new provider gets the contract tests for free.
- **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent`
on a mismatch instead of adopting an instance that belongs to someone else.
- **Report asynchronous create failures:** gcp keeps the operation name, and
`Get` reports the failure instead of NotFound.
- **Capacity or quota signals** for kvm, such as pool free space and host
memory, mapped to `ErrQuotaExceeded`.
- **GC for secondary resources:** mark volumes and similar resources, and
let `ListByTag` or a second listing method cover them.
- **Reload credentials and config without a restart.**
- **Checker `Details`** for kubernetes (server version) and gcp (quota
headroom).
- **Internal-IP mode** for gcp, and a firewall rule managed per provider.
- **An installation-identity mark**, set on create and filtered on in
`ListByTag`, so installations sharing a backend do not collect each
other's instances.
- **A permanent-failure latch that sees provider availability**, so adding
a missing provider releases the Proxies waiting on it.
- **Validate provider names at load**, and derive the config type
allow-list from the constructor map.
- **Per-provider RBAC** as an optional kustomize component, and an
`OwnerReference` on kubernetes Pods.