egress-proxies-operator/docs/providers-existing.md

The existing providers

This is the appendix to provider-contract.md: how the five providers that exist today (kubernetes, gcp, kvm, kvm2, proxmox) meet the contract, which profile each one chose on every axis, and what is still missing. It is reference material for whoever writes the next provider, not a user guide; the user guides are kvm-provider.md, kvm2-provider.md, proxmox-provider.md, gcp-vm-validation.md and gcp-in-specific-project.md. The README's "Implemented providers" table stays the short public summary. kvm2 and proxmox are the two providers built from the contract itself (kvm2 clean-room from kvm2-provider-spec.md without reading kvm's code, proxmox from proxmox-provider-spec.md), so their columns are also a record of what the contract produced.

The per-provider call flows are drawn in architecture.md §5, the health checks are tabulated in §11, and the judgment calls behind several rows below are recorded under its "Decisions" list. They are linked, not repeated.

Code references are a file path plus a function or symbol name.

1. Profile table

One row per axis of contract §4, in the same order, with the profile name and the symbol that shows it.

Axis	kubernetes	gcp	kvm	kvm2	proxmox
readiness	immediate: a Pod is scheduled within seconds (stateFromPod)	provisioning-window: minutes of boot (mapState)	provisioning-window: boot plus, on first use, an image download (imageCache.ensure)	provisioning-window: boot plus, on first use, an image download (ensureImage)	provisioning-window: boot plus the guest agent's install at first boot, about 50 s to an address on the test host (agentAddress)
create-return	submit: returns after client.Create	submit: Insert is never awaited, Operation.Wait is not called	blocking: waits through DomainCreate (ensureRunning), and the first image fetch runs inside (imageCache.fetch)	blocking: waits through DomainCreate and a 15-second poll for running (ensureRunning, waitRunning), and the first image fetch runs inside (fetchImage)	blocking: waits through the clone, upload, resize and start tasks and a 60-second poll for running (provision, waitRunning)
delete-return	returns-early: client.Delete, Pod terminates on its own	returns-early: Delete submits an operation each time	waits: destroy, undefine and volume removal complete before return (removeDomain, removeVolumes)	waits: destroy, undefine and volume removal complete before return (removeDomain, removeVolumes)	waits: the stop, seed delete and destroy tasks complete before return (destroy, deleteSeed)
ip-source	assigned-at-create: PodIP (instanceFromPod)	assigned-at-create: the external NAT IP of the first NIC (toInstance)	discovered: ARP table first, then the guest agent, first routable IPv4 (address, firstIPv4)	discovered: the guest agent first, then the ARP table, first routable IPv4 (addressSources, address, firstIPv4)	discovered: the guest agent only, first routable IPv4 in the agent's order (agentAddress, firstIPv4)
gc-mark	native-labels: Pod labels (buildPod)	native-labels: instance labels (buildInsertRequest)	written-metadata: a namespaced <metadata> element written in DomainDefineXML (metadata.go)	written-metadata: an element in kvm2's own namespace written in DomainDefineXML (metadataURI, newInstanceMetadata)	written-metadata: three key=value lines in the VM's description, passed in the clone call itself (formatDescription, parseDescription)
created-at	backend-timestamp: CreationTimestamp	backend-timestamp: CreationTimestamp, parse error gives zero	written-at-create: createdAt in the metadata element (newInstanceMetadata)	written-at-create: createdAt in the metadata element, parse error gives zero (newInstanceMetadata, createdAt)	written-at-create: proxy-operator-created-at in the description, parse error gives zero (parseDescription)
list-partial	n/a: one call, all namespaces	omit-unreachable: ReturnPartialSuccess on AggregatedList	fail-whole per host, one unreadable domain skipped (ListByTag, managedInstance)	fail-whole per host, one unreadable domain skipped unless ctx ended; ID, UID and CreatedAt from the name and the mark only, no state or address RPC (ListByTag)	fail-whole for the one cluster-wide /cluster/resources call, one unreadable VM skipped unless ctx ended; ID, UID and CreatedAt from the description only, no status or agent call (ListByTag)
quota	absent: a quota 403 is Permanent (classify)	detected: 429, and 403 with reason quotaExceeded or rateLimitExceeded (classify, hasReason)	absent (classify)	detected: a pool-space precheck before any volume is created (checkPoolSpace, neededBytes)	absent: thin storage overprovisions, so no out-of-space answer exists to match; free space goes to the Checker's details (classify)
not-found	status-404: apierrors.IsNotFound	status-404: isNotFound	error-code: ErrNoDomain, ErrNoStorageVol (hasCode)	error-code: ErrNoDomain, ErrNoStorageVol, Transient inside Create (classifyLibvirt)	error-code: the 500 Configuration file '…' does not exist, plus list-and-miss on /cluster/resources; Transient inside Create (isNotExist, classifyAPI)
primary-key	name-is-key: namespace/name	name-is-key: zones/<zone>/instances/<name>	name-is-key: hosts/<host>/domains/<name>	name-is-key: hosts/<host>/domains/<name>, with a UID guard on adoption (create)	allocated-id: vms/<vmid>/<name>, the VMID from an FNV-1a hash of the UID with forward probing, each taken VMID put through an ownership check with a UID guard (vmidFor, allocate, inspect)
provisioning-state	native: PodPending	native: PROVISIONING, STAGING, REPAIRING	create-blocks: a domain is defined and started in one Create; nostate maps to Provisioning	create-blocks: defined, started and polled to running in one Create; nostate maps to Provisioning (mapState)	create-blocks: cloned, configured, started and polled to running in one Create; qmpstatus: prelaunch maps to Provisioning (mapState)
async-ops	none	task-handle, but the handle is discarded: an insert that fails after acceptance is never read	none	none	task-handle and backend-lock: every clone, upload, resize, start, stop, destroy and volume delete returns a UPID polled to its end within the call (waitTask); a VM's lock is waited on in Create and Transient in Get and Delete (waitUnlocked)
secondary-resources	none	cascade: AutoDelete on the boot disk (buildInsertRequest)	explicit: overlay and seed removed by name (removeVolumes); base image kept	explicit: overlay and seed removed by name (removeVolumes); base image kept	explicit: the seed ISO deleted by name (deleteSeed, deleteSeedEverywhere); the disks go with the destroy's purge=1&destroy-unreferenced-disks=1
credentials	workload-identity: in-cluster config or kubeconfig (NewWithTransportWrapper)	environment: Application Default Credentials	mounted-files: SSH key and known_hosts from the optional kvm-ssh Secret, read lazily (newLazySSH)	mounted-files: the same kvm-ssh Secret, read on every dial and never cached (loadCredentials)	mounted-files: the token from the optional proxmox-token Secret, read on every request (readToken); the CA from the same Secret, loaded once and cached (httpClient)
client-kind	vendor-sdk: controller-runtime client.Client, no seam interface	vendor-sdk: cloud.google.com/go/compute behind instancesAPI	rpc-over-ssh: go-libvirt over an SSH-forwarded unix socket, behind libvirtAPI (conn.go)	rpc-over-ssh: go-libvirt over an SSH-forwarded unix socket, behind the ctx-first backend seam (backend.go)	thin-client: net/http and encoding/json behind the ctx-first api seam, one method per endpoint (api.go, httpAPI)
timeouts	ctx-only: no rest.Config.Timeout is set	ctx-only: SDK defaults, with SDK retries on Get and List	per-call for the dial only (defaultHandshakeTimeout in dialSSH); RPCs are ctx-only and a cancelled call closes the connection	per-call: a budget around the dial, every RPC, the uploads and the download (defaultBudgets, call); a cancelled or overlong call kills the pooled connection	per-call: dial and TLS handshake 10 s, each request requestTimeoutSeconds, a budget per task wait (defaultBudgets); a cancelled request ends only itself
logging	none	tiered: V(1) outcomes and logAPIError, V(2) detail, V(5) wire log (wirelog.go)	tiered: V(1) outcomes and one libvirt call line per RPC (trace.go), V(2) XML and addresses; no wire tier	tiered: V(1) outcomes and one libvirt call line per RPC (observe.go), V(2) XML and addresses; no wire tier	tiered: V(1) outcomes and one proxmox api call line per request (observe.go), V(2) parameters, config summaries and addresses; no wire tier
child-spans	transport-wrapper: tracing.RestConfigWrapper() on its own rest.Config, parent-gated	sdk-instrumentation: the SDK adds otelhttp itself; not parent-gated, but every call site has a parent	none: trace.go is a log decorator, not spans	span-decorator: kvm2.<Method> per RPC (observe), plus kvm2.dial (libvirtDialer) and kvm2.download (fetchImage)	transport-wrapper: tracing.RestConfigWrapper() on the client's transport, parent-gated, one HTTP <METHOD> span per request (newHTTPAPI)
backend-test	cluster-e2e: kind (test/e2e)	manual-runbook: gcp-vm-validation.md	integration-tag: kvm_integration_test.go, make test-integration-kvm	integration-tag: kvm2_integration_test.go, make test-integration-kvm2	integration-tag: proxmox_integration_test.go, make test-integration-proxmox; passed on a PVE 9.2.20 host

2. Other differences

Rows that are not profile axes but that a new provider author compares against.

Feature	kubernetes	gcp	kvm	kvm2	proxmox
Create verifies the adopted instance's UID	no	no	no	yes: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (create)	yes: a VM with the proxy's name and no mark, or with a different non-empty UID, is Permanent; a marked one with a matching or empty UID is adopted (inspect)
Placement used	none	Zone, MachineType, Image	Zone (host), MachineType, Image	Zone (host), MachineType, Image	Zone (node), MachineType, Image (a template key)
CloudInit delivered	no: ignored	yes: user-data metadata	yes: NoCloud seed ISO built in-process (buildSeedISO)	yes: NoCloud seed ISO built in-process (nocloud.BuildSeedISO)	yes: NoCloud seed ISO built in-process (nocloud.BuildSeedISO) and uploaded through the API (uploadSeed)
Port used	yes: Squid config and containerPort (squidConf)	no: firewall is manual	no	no	no
States produced	Provisioning, Running, Terminated	all four	Provisioning, Running, Stopped	Provisioning, Running, Stopped	Provisioning, Running, Stopped
Deleting instance still reports its state	yes: a Pod being deleted reports its phase	n/a	n/a	n/a: Delete waits, and a concurrent Get sees the domain until it is undefined	n/a: Delete waits, and a concurrent Get sees the VM (Stopped once stopped) until the destroy removes its config
Error.Provider is the configured name	no: hardcoded "kubernetes"	yes	yes	yes (wrapErr)	yes (wrapErr)
Checker target	apiserver	the project	each host, concurrently	each host, concurrently, on a fresh connection	each node, concurrently, on the shared HTTP client
Checker Details	none	none	user, pool, hostname, libvirtVersion, poolState, poolCapacity, poolAvailable	user, pool, hostname, libvirtVersion, poolState, poolCapacityGiB, poolAvailableGiB, filled in as far as the check got	isoStorage, tokenID, pveVersion, kernel, isoStorageAvailGiB, templates, filled in as far as the check got
Credential rotation without restart	n/a	via ADC	no: cached after first successful load	yes: both files read on every dial (loadCredentials)	the token yes, read on every request (readToken); the CA no, cached after first use
Connection handling	client-go	SDK client, never closed	one pooled SSH plus libvirt connection per host, 15s keepalive (hostPool)	one pooled SSH plus libvirt connection per host, 15s keepalive (hostTable); dedicated connections for the image upload and the check (dialFresh)	one net/http client, 4 idle connections per host, HTTP/1.1 only; a cancelled request ends only itself
Unit-test double	controller-runtime fake client	fakeAPI behind instancesAPI, plus httptest for the real SDK client in check_test.go	fakeAPI behind libvirtAPI, plus an in-process x/crypto/ssh server in conn_test.go	fakeBackend behind backend (fake_test.go), plus an in-process x/crypto/ssh server (sshserver_test.go) and go-libvirt's own mock for the adapter	fakeAPI behind api (fake_test.go), plus httptest.NewTLSServer replaying the Phase 0 captures for the real client (client_test.go)
Proxy software	generated Squid config (pod.go)	from cloudInit; the image must run cloud-init	from cloudInit, plus a vendor-data script that installs the guest agent (seed.go)	from cloudInit, plus a vendor-data script that installs the guest agent (nocloud.GuestAgentScript)	from cloudInit, plus a vendor-data script that installs the guest agent (nocloud.GuestAgentScript)
IP reachable from the operator	Pod network, so in-cluster only (not from make run-dev on a laptop)	external IP, once a firewall rule allows the cluster's egress IP	LAN address on the host's bridge; the operator needs a route to that LAN	LAN address on the host's bridge; the operator needs a route to that LAN	address on the proxy network (bridge or VLAN from the config); the operator needs a route to it
Distinct egress IP per proxy	no: the cluster's egress	yes: ephemeral external IP	whatever the host's LAN routes through; not per VM by itself	whatever the host's LAN routes through; not per VM by itself	whatever the proxy network routes through; not per VM by itself

3. Observability status

Layer	kubernetes	gcp	kvm	kvm2	proxmox
1. Operation spans and metrics	yes: decorators	yes: decorators	yes: decorators	yes: decorators	yes: decorators
2. Child spans per backend call	yes: tracing.RestConfigWrapper(), parent-gated (cmd/main.go, NewWithTransportWrapper)	yes: the SDK's own otelhttp; not parent-gated	no	yes: kvm2.<Method> from the observe decorator, kvm2.dial, kvm2.download; always under the caller's span, since every call carries the caller's ctx	yes: tracing.RestConfigWrapper() on the transport, parent-gated, one HTTP <METHOD> per request
3. Logs carry traceID and spanID	n/a: the package does not log	yes, including the V(5) wire log	yes: every libvirt call line at V(1)	yes: every libvirt call line at V(1), written inside its RPC's span	yes: every proxmox api call line at V(1), with the operation span's spanID
Tested against a real collector	yes: the kind e2e checks provider.create and provider.delete in Tempo (test/e2e/tracing_test.go), operation spans only	no	no	no; the integration test asserts the parent-child topology in an in-memory recorder (assertLifecycleSpans)	no; TestTracing_httpSpansUnderOperationSpan asserts the parent-child topology in an in-memory recorder

4. kubernetes (internal/provider/kubernetes)

• Package and seam. kubernetes.go, pod.go, check.go. There is no seam interface: newWithClient injects a controller-runtime client.Client, and tests use the fake client. New calls NewWithTransportWrapper, which builds its own uncached client from ctrl.GetConfig() with an optional transport.WrapperFunc for tracing.

• Config. KubernetesConfig.Image only, defaulting to the Squid image constant in kubernetes.go.

• ID. providerID(namespace, name) gives namespace/name; parseProviderID uses cache.SplitMetaNamespaceKey and rejects an empty namespace with ErrPermanent. An empty name part is not rejected.

• Create. buildPod (pure) then client.Create; IsAlreadyExists is success. Only Name, Namespace, UID and Port are used. The Squid config is generated by squidConf from the port and passed through the SQUID_CONF environment variable, so there is exactly one object per proxy.

• Get and state. stateFromPod:

  Pod phase	PodIP	Provider state
  Pending	any	Provisioning
  Running	empty	Provisioning
  Running	set	Running
  Succeeded, Failed, Unknown	any	Terminated

  instanceFromPod sets IP only when the state is Running. A Pod with a deletion timestamp still reports its phase. UID from the LabelUID label, CreatedAt from CreationTimestamp.

• Delete. client.Delete, IsNotFound ignored.

• ListByTag. One client.List across all namespaces with MatchingLabels{LabelManaged: LabelManagedYes}; this is what the cluster-wide Pod RBAC is for.

• Errors. classify: IsNotFound NotFound; IsTooManyRequests, IsServerTimeout, IsTimeout Transient; IsForbidden, IsInvalid, IsBadRequest, IsUnauthorized Permanent; default Transient. Nothing is Quota, because a quota 403 and an RBAC 403 cannot be told apart.

• Check. Lists at most one managed Pod cluster-wide (client.Limit(1)), proving read and list access, not create. Target apiserver, address the rest config host. checkReason: context errors and timeouts ReasonTimeout; IsUnauthorized; IsForbidden; any APIStatus ReasonAPIError; else ReasonUnreachable.

• Logging and tracing. No log lines in the package. Child spans come from the parent-gated transport wrapper.

• Tests. kubernetes_test.go, pod_test.go, check_test.go over the fake client; the kind e2e (test/e2e) is the real-backend run.

• Deployment. RBAC markers for Pods live next to the reconciler's in internal/controller/proxy_controller.go. hack/providers-dev.yaml holds only this provider, for make run-dev.

• Known gaps.

  • Error.Provider is hardcoded to "kubernetes", so errors and logs show the wrong name when the configured name differs.
  • CloudInit and all of Placement are ignored.
  • No quota class: a quota 403 is Permanent.
  • A Pod being deleted still reports its phase, so it can read as Running.
  • No securityContext, no resources, and http_access allow all in pod.go.
  • No logging in the package, and no Checker Details.
  • No OwnerReference on Pods: a bypassed finalizer leaves the Pod behind.
  • The Pod RBAC is always installed, even when no kubernetes provider is configured.

5. gcp (internal/provider/gcp)

• Package and seam. gcp.go, insert.go, errors.go, check.go, wirelog.go. The seam is instancesAPI (Insert, Get, Delete, AggregatedList, Probe); realInstances adapts the SDK client and newWithAPI is the test constructor. New calls NewWithWireOptions, which takes the --gcp-wire-log-full-payloads flag from cmd/main.go.

• Config. GCPConfig: project required; network, networkTag, diskSizeGb defaulted in withDefaults.

• ID. formatProviderID(zone, name) gives zones/<zone>/instances/<name>; parseProviderID rejects anything else with ErrPermanent. The project comes from the config.

• Create. Validates Zone, MachineType and Image up front and returns ErrPermanent naming the missing field. buildInsertRequest (pure) builds one NIC with an ephemeral external IP (ONE_TO_ONE_NAT) and a boot disk with AutoDelete, labelled with both marks. Insert is submitted and never awaited; 409 (isAlreadyExists) is success.

• Get and state. mapState(status, ip):

  Instance status	NAT IP	Provider state
  PROVISIONING, STAGING, REPAIRING	any	Provisioning
  RUNNING	empty	Provisioning
  RUNNING	set	Running
  STOPPING, STOPPED, SUSPENDING, SUSPENDED	any	Stopped
  TERMINATED	any	Terminated
  anything else	any	Stopped

  toInstance reads the NAT IP of the first access config of the first NIC, UID from the label, and CreatedAt from CreationTimestamp with the parse error dropped (zero time on failure).

• Delete. Submits a delete operation and returns; 404 (isNotFound) is success. Every deletion poll submits a new operation.

• ListByTag. AggregatedList with managedFilter on the managed label only and ReturnPartialSuccess, so an unreachable zone is left out silently; the zone is recovered from the URL by lastPathSegment.

• Errors. classify: non-googleapi errors Transient; 404 NotFound; 429 Quota; 403 with reason quotaExceeded or rateLimitExceeded (hasReason) Quota; 400, 401 and other 403s Permanent; other codes Transient. logAPIError records the HTTP status and reasons at V(1) because classify collapses them.

• Check. Probe makes one AggregatedList page with maxResults=1 and the managed filter: credentials, the Compute API and compute.instances.list in one call. Target is the project, address compute.googleapis.com. checkReason: context errors ReasonTimeout; non-googleapi errors ReasonUnreachable; 401; 403; else ReasonAPIError.

• Logging. logger(ctx) from the context, named gcp with provider. V(2) for the curated insert request (never the user-data), V(1) per operation outcome, V(2) per listed instance. The wire log in wirelog.go runs at V(5) with payloads elided unless the flag is set.

• Tracing. The SDK adds otelhttp itself unless telemetry is disabled; not parent-gated.

• Tests. insert_test.go (pure builder), errors_test.go (code table), gcp_test.go (IDs, 409, state mapping, list filter and partial success) over fakeAPI; check_test.go also drives the real SDK client against httptest to prove exactly one request with maxResults=1. Real backend: the manual runbook in gcp-vm-validation.md.

• Deployment. Credentials by ADC; the runbook mounts a service-account key and points GOOGLE_APPLICATION_CREDENTIALS at it. A commented entry in both providers configs, config/samples/proxy_gcp.yaml, and demo scripts under docs/demo/.

• Known gaps.

  • Asynchronous insert failures are never read: an insert the API accepts but that fails later (quota, bad image) is seen only as NotFound on the next Get, and the reconciler creates again, so the reason never reaches status. gcp-vm-validation.md still says such errors do reach status.
  • External IP only: a VM without an external NAT IP stays Provisioning forever.
  • No firewall management: the rule for the proxy port is a manual prerequisite, and req.Port is unused.
  • The SDK client is never closed, and Placement.Region is unused.
  • Wrong label value in the docs: gcp-vm-validation.md filters on labels.proxy-operator-managed=yes; the value is true.
  • The sample cannot work: config/samples/proxy_gcp.yaml uses a debian-cloud image, which has no cloud-init, so its Squid install never runs (gcp-in-specific-project.md uses Ubuntu for that reason).

6. kvm (internal/provider/kvm)

The full user guide is kvm-provider.md; this section is the author's view.

• Package and seam. kvm.go (the four methods), conn.go (SSH dial, hostPool, libvirtAPI), image.go (imageCache), seed.go (NoCloud ISO and vendor-data), domainxml.go, volume.go, metadata.go, id.go, errors.go, check.go, trace.go. The seam is libvirtAPI, the twenty go-libvirt calls the provider uses; fake_test.go implements it in memory and newWithDialer injects the dialer and the HTTP client.

• Config. KVMConfig in internal/provider/config_kvm.go: sshKeyFile, knownHostsFile or insecureIgnoreHostKey, diskSizeGb, vendorData, installGuestAgent, images (url, sha256, format, firmware), machineTypes (vcpus, memoryMiB), hosts (name, address, user, socket, pool, bridge). validateKVM restricts image keys and host names with kvmIdentifier because they become a volume name and an ID segment.

• ID. formatProviderID(host, name) gives hosts/<host>/domains/<name>.

• Create. host resolves Placement.Zone (optional with one host), the machine type must be a catalogue key, resolveImage accepts a catalogue key or an http(s) URL. Then, inside one hostPool.call: DomainLookupByName, and if found ensureRunning; otherwise imageCache.ensure (download on first use, streamed through the operator into the pool with a sha256 marker volume), ensureOverlay (reused if it exists), writeSeed (always rebuilt: a cut-off upload must not be trusted), defineDomain with the metadata element, ensureRunning. Create returns only when the domain is running.

• Get and state. mapState(state, ip):

  libvirt state	IP	Provider state
  nostate	any	Provisioning
  running, blocked	empty	Provisioning
  running, blocked	set	Running
  paused, shutdown, shutoff, crashed, pmsuspended	any	Stopped

  address asks the ARP table first, then the guest agent, and firstIPv4 drops loopback, link-local and unspecified addresses. Metadata is read back by metadata and parsed by parseInstanceMetadata; a domain without the element reports an empty UID.

• Delete. removeDomain (destroy, tolerating not running or gone; then DomainUndefineFlags with NVRAM, managed save and snapshot metadata) and removeVolumes (overlay and seed by name, a missing pool is a no-op). A host no longer in the config is a logged no-op success.

• ListByTag. ConnectListAllDomains on every host; managedInstance keeps domains whose metadata says managed and skips a domain whose metadata or state cannot be read. One unreachable host fails the whole list.

• Errors. classify: isClassified errors keep their class; ErrNoDomain, ErrNoStorageVol NotFound; XML, unsupported, invalid-argument, auth and access errors, and a missing pool, Permanent; everything else Transient, including SSH handshake and auth failures. Nothing is Quota. logAPIError adds libvirtErrorFields.

• Check. checkHost per host concurrently through hostPool.probe, not call, so a check that runs out of time abandons its RPCs instead of closing the connection a reconcile may be using. Two cheap RPCs (ConnectGetLibVersion, ConnectGetHostname) then the pool must exist and be running. Reasons: the shared ones plus SSHMaterialUnavailable, SSHHandshakeFailed, LibvirtConnectFailed, LibvirtError, PoolNotFound, PoolNotActive, chosen by checkReason from the dialError stage. Details: user, pool, hostname, libvirtVersion, poolState, poolCapacity, poolAvailable.

• Connections. hostPool keeps one SSH plus libvirt connection per host, dialled lazily (lazySSH.dial, sshAuth.dial, dialSSH, connectLibvirt), with a 15s keepalive. hostEntry.acquire is a one-slot semaphore so waiting can end with the caller's context. A cancelled call closes the shared connection to unblock the RPC; probe abandons it and leaves the connection open. The SSH material is read once by lazySSH.load and cached.

• Logging. Logger from the context, named kvm, with provider, host and name. V(1): one line per operation outcome and one libvirt call line per RPC from the tracingAPI decorator in trace.go (rpc name, identifiers, duration, error fields). V(2): domain and volume XML, metadata, addresses. Upload and download bodies are never logged.

• Tracing. No child spans; trace.go is a log decorator.

• Tests. Pure builders (domainxml_test.go, volume_test.go, seed_test.go, id_test.go, metadata_test.go); errors_test.go, conn_test.go (pool behaviour and a real in-process SSH server); image_test.go, kvm_test.go (Create resumes after every partial step, every Delete already-gone case, ListByTag policies, logging tiers, a cloud-init sentinel never logged); trace_test.go; check_test.go. Real backend: kvm_integration_test.go behind //go:build integration, driven by KVM_TEST_* variables and make test-integration-kvm, described in testing.md.

• Deployment. The optional kvm-ssh Secret mounted at /etc/proxy-operator/kvm (config/manager/manager.yaml), make kvm-ssh-secret, a commented entry in both providers configs, config/samples/proxy_kvm.yaml.

• Known gaps (see also kvm-provider.md § Known gaps).

  • No quota or capacity detection.
  • Create blocks, including the first image download on each host.
  • GC blind spots: volumes without a domain are invisible to GC, and so are VMs on a host removed from the config, which Delete treats as done.
  • SSH material is cached after the first successful load, so rotating keys needs a restart.
  • Cancelling one operation cuts off the others on that host: a cancelled call closes the shared per-host connection.
  • SSH auth failures retry forever: SSH authentication and host-key failures are Transient, while libvirt auth errors are Permanent.
  • Create can return ErrNotFound when a volume or domain disappears during the call.
  • No reused-overlay check: an existing overlay is reused without checking its backing image or size.
  • Sample mismatch: config/samples/proxy_kvm.yaml places on host nas1, while both providers configs define only beelink.
  • Doc drift in kvm-provider.md: it says ARP is the fallback, but the code tries ARP first; it says host and name are on every log line, but the check and list paths lack them.

7. kvm2 (internal/provider/kvm2)

The full user guide is kvm2-provider.md and the design it was built from is kvm2-provider-spec.md; this section is the author's view. Nothing in the package was taken from internal/provider/kvm; where the two agree it is because both follow the same libvirt.

• Package and seam. kvm2.go (Provider, New, withDefaults, host, wrapErr, the per-name lock), create.go, get.go, delete.go, list.go (the four methods), backend.go (the seam, the production adapter and the budgets), ssh.go (credentials, the forwarded socket, keepalive), hosts.go (the pooled table and the production dialer), observe.go (the span and log decorator), image.go (cache, download, quota precheck), xml.go, volume.go, metadata.go, id.go, state.go, errors.go, check.go; the seed ISO comes from the shared internal/provider/nocloud and both lock sets are internal/provider/keyedlock. The seam is backend: the twenty go-libvirt calls the provider uses, each with ctx first (go-libvirt's own methods take none and block), plus Close. libvirtBackend adapts *libvirt.Libvirt and runs every call through call, which applies the budget and the abandon policy. The dialer seam separates the SSH transport: newProvider injects one for tests, newLibvirtDialer is production. The observe decorator is applied once, in libvirtDialer.dial, so provider code never holds an undecorated backend.

• Config. ProviderConfig.KVM2, the same KVMConfig type as kvm, validated by validateKVMBlock with block name kvm2 so messages read kvm2.hosts. Defaults (diskSizeGb 10, socket, pool, image format and firmware) are filled by withDefaults. New never dials; it checks that the credential files exist and warns when one does not, or when insecureIgnoreHostKey is on.

• ID. formatProviderID(host, name) gives hosts/<host>/domains/<name>; parseProviderID accepts exactly four segments with the literal hosts and domains and non-empty host and name, else ErrPermanent.

• Create. Create validates Zone (host; optional with one host), MachineType (a catalogue key) and Image (resolveImage: a catalogue key or an http(s) URL with qcow2 and uefi defaults) before any dial, and logs the resolved parameters with cloudInitBytes at V(2). Then, under the per-name lock (lockName) and on the pooled connection (hostTable.get), create: DomainLookupByName; found means the adoption guard (no kvm2 mark, or a non-empty metadata UID that differs from req.UID, is Permanent) and ensureRunning. Otherwise StoragePoolLookupByName, imageSizeIfUncached (marker lookup, then HEAD, or a closed GET when HEAD is not 2xx) and checkPoolSpace with neededBytes (image if uncached, plus diskSizeGb, plus 16 MiB), then ensureImage (cachedImage on the pooled connection, fetchImage streaming into uploadImage on a dialFresh connection, StoragePoolRefresh, writeMarker last), ensureOverlay (ErrStorageVolExist means reuse), writeSeed (delete, build, upload; always rebuilt), two StorageVolGetPath, buildDomainXML with file-type disks and the metadata element, DomainDefineXML, ensureRunning (DomainGetState, where paused or pmsuspended is Permanent; DomainCreate with ErrOperationInvalid as already running; waitRunning polls every 500 ms for up to 15 seconds, Transient on expiry and at once for any inactive state other than nostate). Create returns only when the domain is running.

• Get and state. mapState(state, ip):

  libvirt state	IP	Provider state
  nostate	any	Provisioning
  running, blocked	empty	Provisioning
  running, blocked	set	Running
  paused, shutdown, shutoff, crashed, pmsuspended, anything else	any	Stopped

  get: DomainLookupByName (only ErrNoDomain reaches NotFound), readMetadata (DomainGetMetadata with metadataURI; ErrNoDomainMetadata means unmanaged with an empty UID; an element that does not parse is Permanent), then DomainGetState and, for an active domain, address. addressSources asks the guest agent first, then ARP; a source that fails is skipped at V(2), only ctx ending is an error, so a cancelled Get is Transient and not a false Provisioning. firstIPv4 drops loopback, link-local and unspecified addresses and an address whose type tag and literal disagree. UID and CreatedAt are read back on every call; an unparsable createdAt gives the zero time.

• Delete. Under the per-name lock: removeDomain (DomainDestroy, tolerating not running and gone; DomainUndefineFlags with NVRAM, managed save and snapshot metadata) and removeVolumes (overlay and seed by name through deleteVolumeIfPresent; a missing pool is a no-op). A host no longer in the config is a logged no-op success. Nothing is left when nil is returned.

• ListByTag. ConnectListAllDomains (active and inactive) on every host in config order; readMetadata and managed keep kvm2's own domains, and the Instance is ID, UID and CreatedAt from the name and the element, with State and IP empty: GC reads nothing else (contract §2.4), so no state or address RPC is made. A domain whose metadata cannot be read is skipped at V(1), unless ctx ended, which fails the call. A host that cannot be dialled or listed fails the whole call.

• Errors. classify(op, err): an error already carrying a class keeps it; libvirt codes through classifyLibvirt (ErrNoDomain and ErrNoStorageVol NotFound, Transient when op == "create"; ErrNoStoragePool, ErrXMLError, ErrNoSupport, ErrInvalidArg, ErrAuthFailed, ErrAccessDenied Permanent; the rest Transient); knownhosts.KeyError and the unable to authenticate text from x/crypto/ssh Permanent; everything else Transient, including a missing credential file. permanentf and transientf classify at the source: placement, XML builders, a parse failure of the key or known_hosts, an image 4xx, a missing or zero Content-Length, a sha256 mismatch, a pool that is not running (Permanent); an image 429 or 5xx, a short body, the running wait expiring (Transient). checkPoolSpace is the only source of ErrQuotaExceeded. logAPIError writes kvm2 backend call failed with libvirtErrorFields.

• Check. Check runs checkHost per host concurrently, each through probeHost on a dialFresh connection that is closed when the probe ends; the pooled table is never consulted. ConnectGetLibVersion, ConnectGetHostname, StoragePoolLookupByName, StoragePoolGetInfo; a pool that is not running is poolNotActiveError. Details are filled as facts arrive (user and pool always; libvirtVersion, hostname, poolState, poolCapacityGiB, poolAvailableGiB). checkHost answers Timeout whenever the round's ctx ended, whatever stage was running and whatever shape the error has; otherwise checkReason: a dialError is judged by its stage (credentials gives CredentialsUnavailable; ssh with a knownhosts.KeyError or the auth text gives Unauthorized; any other stage, including a dial that ran out of its own budget, gives Unreachable); then context.DeadlineExceeded or Canceled Timeout, ErrAccessDenied or ErrAuthFailed Forbidden, ErrNoStoragePool PoolNotFound, any other libvirt code APIError, anything else Unreachable; PoolNotActive from the pool state.

• Connections. hostTable keeps one pooled connection per host, dialled lazily under a per-host one-slot semaphore so waiting ends with the caller's ctx; a dead backend (liveness) is closed and redialled on the next get. dialSSHSocket runs credentials, TCP, the SSH handshake and the direct-streamlocal channel to the socket, each failure a dialError with its stage; libvirtDialer.connect bounds the whole sequence plus the libvirt Connect with the dial budget and starts the keepalive (15 s, keepalive@openssh.com) and a watcher on go-libvirt's Disconnected. call runs each blocking go-libvirt call in a goroutine and waits on the result, ctx and a budget timer; abandoning a call (link.markDead) closes the socket and the SSH client, returns Transient wrapping the ctx or deadline error, and every other call waiting on that link fails once with Transient. loadCredentials runs on every dial.

• Logging. Logger from the context, named kvm2, with provider, and host and name on lines about an instance. V(1): one line per operation outcome (kvm2 domain ensured running, kvm2 domain fetched, kvm2 domain deleted, kvm2 domains listed, kvm2 host checked), one libvirt call line per RPC from observe (rpc name, identifiers, duration, error fields), the connection events libvirt host connected and libvirt host connect failed with stage, and kvm2 backend call failed. V(2): XML, metadata, addresses, listed names, pool sizes, the resolved Create parameters. Upload and download bodies, cloud-init and key material are never logged.

• Tracing. observe.call starts kvm2.<Method> per RPC with kvm2.host and the identifiers (kvm2.domain, kvm2.pool, kvm2.volume), sets kvm2.libvirt_code on failure and writes the libvirt call line inside the span; libvirtDialer.dial starts kvm2.dial (kvm2.address, kvm2.user); fetchImage starts kvm2.download (kvm2.url_host, kvm2.sha256_configured, kvm2.bytes). Both the decorator and the dialer take tracing.Option so tests inject a recorder; New leaves them on the global provider.

• Tests. Pure builders (xml_test.go, volume_test.go, seed_test.go, id_test.go, metadata_test.go, state_test.go); errors_test.go (the classify table with real SSH error strings and a real knownhosts.KeyError); backend_test.go (the adapter over go-libvirt's own mock through a net.Pipe: deadline kills the connection, cancel unblocks, killed by another caller); hosts_test.go; ssh_test.go over sshserver_test.go (every dial stage, host-key policies, key material never in errors or logs); image_test.go (marker rules, streaming asserted by counting bytes, dedicated upload connection); create_test.go, get_test.go, delete_test.go, list_test.go, kvm2_test.go, provider_test.go (the contract §11 list, named test by test in the spec's entry 21); observe_test.go (every seam method spanned and logged exactly once, by reflection); check_test.go (the reason table, never touches the pool). The double is fakeBackend in fake_test.go, whose twin shares state under its own call log so a test can tell the pooled connection from a dedicated one. Real backend: kvm2_integration_test.go behind //go:build integration, driven by KVM2_TEST_* variables and make test-integration-kvm2, described in testing.md.

• Deployment. The kvm-ssh Secret and its mount are shared with kvm (config/manager/manager.yaml says so), make kvm-ssh-secret is reused, a commented kvm2-home entry in both providers configs, config/samples/proxy_kvm2.yaml (zone beelink, matching the shipped config), the kvm2 type in the status.type comment of api/v1alpha1/providerhealth_types.go. No RBAC.

• Known gaps (see also kvm2-provider.md § Known gaps).

  • Create blocks, including the first image download on each host.
  • GC blind spots: volumes without a domain are invisible to GC, and so are VMs on a host removed from the config, which Delete treats as done.
  • No adoption of kvm's domains: a switched entry needs its Proxies deleted first.
  • Cancelling one operation cuts off the others on that host once: a cancelled call kills the pooled connection.
  • Content-Length is required for image downloads.
  • A paused or pmsuspended domain is not started by a repeat Create: ensureRunning returns Permanent, so the Proxy latches until its spec changes; the admin resumes the VM on the host.
  • Credential parse errors carry the parser's own text, which can quote a PEM header type or a malformed known_hosts token (never key bytes).
  • No reused-overlay check: an existing overlay is reused without checking its backing image or size.
  • The sample cloud-init installs Squid with http_access allow all.

8. proxmox (internal/provider/proxmox)

The full user guide is proxmox-provider.md and the design it was built from is proxmox-provider-spec.md; this section is the author's view. It is the second provider built from the contract, and the first on a REST API of the hypervisor's own: no SSH, no libvirt. Every error text it matches was captured from a PVE 9.2.20 host before the code was written.

• Package and seam. proxmox.go (Provider, New, withDefaults, node, wrapErr, the per-name lock, the budgets), create.go, get.go, delete.go, list.go (the four methods), api.go (the seam), client.go (the production adapter httpAPI, the token and CA loading), observe.go (the logging decorator), task.go (task, lock and running waits), seed.go, mark.go, id.go, state.go, errors.go, check.go; the seed ISO comes from the shared internal/provider/nocloud and the name lock from internal/provider/keyedlock. The seam is api: one method per endpoint, sixteen in all, each with ctx first. New wraps httpAPI in observe once, so provider code never holds an undecorated seam; newProvider is the test constructor.

• Config. ProviderConfig.Proxmox, the ProxmoxConfig type in internal/provider/config_proxmox.go, validated by validateProxmox (the URL shape, node names and storage ids by regexp because they end up in request paths, vlanTag, vmidRange, the two catalogues). isoStorage has no default on purpose: the token needs Datastore.Allocate there. Defaults (vmidRange 100000 to 199999, requestTimeoutSeconds 30, installGuestAgent true) are filled by withDefaults. New never makes a request; it warns when the token file does not exist, or when insecureSkipTLSVerify is on.

• ID. formatProviderID(vmid, name) gives vms/<vmid>/<name>, with no node, because VMs migrate; the node is resolved from /cluster/resources on every call (resolve). parseProviderID accepts exactly three segments, the literal vms, a decimal VMID without leading zeros between 100 and 999999999 and a non-empty name, else ErrPermanent. The name is there so Get and Delete can tell our VM from another that later got the same VMID.

• Create. placement checks Zone (node; optional with one node), MachineType and Image (both catalogue keys) before any request, and logs the resolved parameters with cloudInitBytes at V(2). Then, under the per-name lock, create: one ClusterResources read; adoptByName (two VMs with the name are Permanent, one goes to inspect); with none, template reads the template's config and refuses a non-template, a cloud-init drive, a missing net0 when the config rewrites it, and an unsized boot disk when diskGiB is set, all before any clone; allocate probes vmidFor(uid, attempt) up to 32 times, cloning onto a free VMID and running inspect on a taken one or on the clone's config file already exists answer. inspect waits out a lock (waitUnlocked; a clone in progress is also nameless), then adopts our name with our mark and a matching or empty UID, probes past another name, and is Permanent for our name without the mark, with another UID, or hibernated. An adopted VM gets one VMStatus: running returns at once. Otherwise provision: uploadSeed (rebuilt and overwritten every attempt), SetVMConfig (configParams: cores, memory, agent=1, the tag, ide2, and net0 through rewriteNet0 when a bridge or VLAN is configured), the resize task when resizeTarget says the boot disk is smaller than diskGiB, the start task, and waitRunning (every second up to 60 s, Transient on expiry or on a stop). Create returns only when the VM runs.

• Get and state. get checks in order: absent from the resources is NotFound; a lock other than suspended is Transient (checked before the name, because a mid-clone entry has none); status: unknown is Transient; the config's does not exist is NotFound, a config lock Transient; another name or no mark is NotFound; hibernated is Stopped without further calls. Then VMStatus and mapState(status, qmpstatus, ip):

  status	qmpstatus	agent	Provider state
  running	running	an IPv4	Running
  running	running	200 without an IPv4, or QEMU guest agent is not running	Provisioning
  running	running	No QEMU guest agent configured	Stopped
  running	prelaunch	not asked	Provisioning
  running	paused, suspended, io-error, anything else	not asked	Stopped
  stopped, anything else	any	not asked	Stopped

  Any other agent answer, including VM <vmid> is not running, is Transient. UID and CreatedAt are read from the description on every call. A running Get is four calls: resources, config, status, agent.

• Delete. Under the per-name lock, destroy: resolve; a lock other than suspended or status: unknown is Transient; the config's does not exist, another name or no mark leave the VM alone; our VM hibernated is destroyed like a stopped one, since the destroy takes it. Otherwise stop (unless stopped), deleteSeed on the VM's node, where a failure fails the call and leaves the VM for the next Delete (a 403 for Datastore.Allocate is Permanent), then destroy with purge=1&destroy-unreferenced-disks=1. Stop and destroy are tasks waited on through runTask, where does not exist means gone. The seed goes first because a VM already destroyed is NotFound to Get, and nothing would call Delete again. With no VM of ours, deleteSeedEverywhere tries every configured node, where a node that fails is logged and skipped unless ctx ended or the failure is Permanent. A missing volume still ends in a successful task, so no message is matched. Nothing is left when nil is returned.

• ListByTag. One ClusterResources covers every node, unreachable ones included (their guests are listed with status: unknown). isCandidate keeps QEMU VMs that are not templates and carry the tag proxy_operator or a name starting proxy-; a nameless entry (a clone in progress) is not a candidate. One VMConfig per candidate; the mark in the description decides, and the Instance is ID, UID and CreatedAt only. A candidate whose config cannot be read is skipped at V(1), unless ctx ended, which fails the call; the resources call failing fails the whole call.

• Errors. classify(op, err): an error already carrying a class keeps it; then apiErrors by classifyAPI (every 4xx Permanent; the 500 does not exist NotFound, Transient when op == "create"; every other status Transient, including the lock, the taken VMID and pveproxy's 595 and 596); TLS verification failures (tls.CertificateVerificationError and the three x509 types) Permanent; everything else Transient. The 500 matchers (isNotExist, isAlreadyExists, the two agent ones) are anchored to the 500 status and each is backed by a captured response. permanentf and transientf classify at the source: placement, the template checks, the ownership check, a malformed token file or CA file (Permanent); a missing token or CA file, an expired wait, a failed task (Transient). Nothing is Quota.

• Check. Check runs checkNode per node concurrently on the shared client. probeNode reads the token file for tokenID (no request when it is unusable), then Version, NodeStatus, checkStorage (a 500 whatever its text, inactive, or no iso content is StorageNotFound) and checkTemplates (each catalogue VMID exists and is a template, else TemplateNotFound). checkNode answers Timeout whenever the round's ctx ended; otherwise checkReason: the two failures above, errCredentials CredentialsUnavailable, errCAFile or a TLS verification failure CertificateInvalid, 401 Unauthorized, 403 Forbidden, 595 and 596 Unreachable, any other status APIError, a net.OpError Unreachable (before the deadline, so a dial out of its own budget is a node that did not answer), a deadline or cancellation Timeout, a cut connection Unreachable.

• Connections. One http.Client, built on first use once the CA loads (httpClient), shared by every call and the Checker: dial and TLS handshake 10 s each, Client.Timeout requestTimeoutSeconds, 4 idle connections per host kept 90 s, HTTP/2 disabled. A cancelled request ends only itself. The token file is read by readToken on every request.

• Logging. Logger from the context, named proxmox, with provider, and vmid, name and node on lines about an instance. V(1): one line per operation outcome (proxmox vm ensured running, proxmox vm fetched, proxmox vm deleted with destroyed, proxmox vms listed, proxmox node checked), one proxmox api call line per seam call from observe (call, identifiers, UPID, duration, httpStatus and error), proxmox task failed, and the no-op lines. V(2): request parameters, config summaries, agent addresses, the resolved Create parameters. Cloud-init, the token, the CA and the upload body are never logged.

• Tracing. No span of its own: newHTTPAPI wraps the transport in tracing.RestConfigWrapper(opts...), so each request is an HTTP <METHOD> client span under the caller's span, and none without one. observe only logs, so nothing is counted twice.

• Tests. Pure pieces (id_test.go, mark_test.go, state_test.go, seed_test.go, errors_test.go with the classify table over the captures, api_test.go); client_test.go (the real client over httptest.NewTLSServer: the token never in errors or logs, x509 failures Permanent, cancellation, the child span); create_test.go, get_test.go, delete_test.go, list_test.go, task_test.go, proxmox_test.go, check_test.go, observe_test.go (every seam method logs exactly one line, by reflection) and provider_test.go (the contract §11 sweep, named test by test in the spec's entry 21). The double is fakeAPI in fake_test.go: VMs, tasks and ISOs in memory, a clone that leaves a nameless locked entry until its task ends, per-method errors and blocking, a call log; its error texts are checked against the captures. Real backend: proxmox_integration_test.go behind //go:build integration, driven by PROXMOX_TEST_* variables and make test-integration-proxmox, described in testing.md; it passed against PVE 9.2.20 on 2026-09-24.

• Deployment. The optional proxmox-token Secret (keys token and ca.pem) mounted at /etc/proxy-operator/proxmox (config/manager/manager.yaml), make proxmox-token-secret, a commented proxmox-home entry in both providers configs (loaded by TestLoadConfig_proxmoxSampleEntry), config/samples/proxy_proxmox.yaml, the proxmox type in the status.type comment of api/v1alpha1/providerhealth_types.go. No RBAC, and no hack/providers-dev.yaml entry, since it needs credentials.

• Known gaps (see also proxmox-provider.md § Known gaps).

  • Create blocks until the VM runs.
  • A clone lock that never clears makes every Create for that Proxy wait out the 10-minute clone budget; qm unlock by hand.
  • GC blind spot: a seed ISO whose VM was destroyed outside the operator is invisible to GC, and a seed on node-local storage stays behind when its VM migrated.
  • The VMID range is shared with humans; a hand-made VM with a proxy's name is Permanent.
  • Cross-node clones (target) need shared storage and are untested.
  • A hibernated VM of ours latches its Proxy when Create's adoption finds it; a recorded one is replaced.
  • The CA is cached after first use; the token is not.

9. Gaps shared by the existing providers

• Adoption is not verified in three of the five. kubernetes, gcp and kvm adopt any existing instance with the right name without checking LabelUID; kvm2 and proxmox return ErrPermanent when the UID differs.
• The Create comment does not match kvm, kvm2 and proxmox. The interface comment in internal/provider/provider.go says Create returns as soon as the request is submitted; the three VM providers on your own hypervisors block until the VM has started.
• Stopped and Terminated are produced inconsistently. kubernetes never produces Stopped, kvm, kvm2 and proxmox never produce Terminated, and the reconciler treats them the same.
• Different ListByTag partial-failure policies. gcp leaves failed parts out, kvm and kvm2 fail the whole list, and proxmox fails it when its one cluster-wide listing fails. Both are safe.
• Stale mock comments in internal/provider/provider.go, internal/provider/registry/registry.go and api/v1alpha1/proxy_types.go (and so the generated CRD's spec.provider description).
• GC does not know about installations. Two clusters sharing one project or hypervisor delete each other's instances. This was a merge-blocker in the 2026-08-10 PR review (docs/reviews) and is still open.
• Tracing. kvm has no child spans, so a kvm trace shows provider.create and nothing under it; correlation works only through traceID on its V(1) log lines. kvm2 has them, but only an in-memory recorder checks their topology. Only kubernetes is tested against a real collector, and that test checks only the operation spans; nothing checks that gcp's SDK spans, the kubernetes API spans, kvm2's spans or proxmox's HTTP spans nest under provider.* in a real collector.

10. Proposed features

Listed only. Nothing below is designed or implemented, except where a bullet says kvm2 already does it.

• Child spans in kvm: extend the tracingAPI decorator in kvm/trace.go, which already wraps every libvirt call, to also start a span per RPC, and add spans for the SSH dial and the image download. The seam's methods take no ctx today (hostPool.call hands fn only the API), so the span context has to be plumbed in. kvm2's observe.go over its ctx-first seam is the shape this would take.
• Tempo e2e checks for child spans, and for gcp, kvm and kvm2 through their real backend runbooks.
• A shared conformance suite: for example internal/provider/providertest, which runs the contract §11 tests against any provider plus a fake backend, so a new provider gets the contract tests for free.
• Verify the UID when Create adopts an instance. Return ErrPermanent on a mismatch instead of adopting an instance that belongs to someone else. kvm2 and proxmox do this; the other three do not.
• Report asynchronous create failures: gcp keeps the operation name, and Get reports the failure instead of NotFound.
• Capacity or quota signals for kvm, such as pool free space and host memory, mapped to ErrQuotaExceeded. kvm2 has the pool-space precheck; host memory is still unchecked in both.
• GC for secondary resources: mark volumes and similar resources, and let ListByTag or a second listing method cover them.
• Reload credentials and config without a restart. kvm2 re-reads its SSH material on every dial; the providers config is still read once per process for every type.
• Checker Details for kubernetes (server version) and gcp (quota headroom).
• Internal-IP mode for gcp, and a firewall rule managed per provider.
• An installation-identity mark, set on create and filtered on in ListByTag, so installations sharing a backend do not collect each other's instances.
• A permanent-failure latch that sees provider availability, so adding a missing provider releases the Proxies waiting on it.
• Validate provider names at load, and derive the config type allow-list from the constructor map.
• Per-provider RBAC as an optional kustomize component, and an OwnerReference on kubernetes Pods.