egress-proxies-operator/docs/providers-existing.md

The existing providers

This is the appendix to provider-contract.md: how the three providers that exist today (kubernetes, gcp, kvm) meet the contract, which profile each one chose on every axis, and what is still missing. It is reference material for whoever writes the next provider, not a user guide; the user guides are kvm-provider.md, gcp-vm-validation.md and gcp-in-specific-project.md. The README's "Implemented providers" table stays the short public summary.

The per-provider call flows are drawn in architecture.md §5, the health checks are tabulated in §11, and the judgment calls behind several rows below are recorded under its "Decisions" list. They are linked, not repeated.

Code references are a file path plus a function or symbol name.

1. Profile table

One row per axis of contract §4, in the same order, with the profile name and the symbol that shows it.

Axis	kubernetes	gcp	kvm
readiness	immediate: a Pod is scheduled within seconds (stateFromPod)	provisioning-window: minutes of boot (mapState)	provisioning-window: boot plus, on first use, an image download (imageCache.ensure)
create-return	submit: returns after client.Create	submit: Insert is never awaited, Operation.Wait is not called	blocking: waits through DomainCreate (ensureRunning), and the first image fetch runs inside (imageCache.fetch)
delete-return	returns-early: client.Delete, Pod terminates on its own	returns-early: Delete submits an operation each time	waits: destroy, undefine and volume removal complete before return (removeDomain, removeVolumes)
ip-source	assigned-at-create: PodIP (instanceFromPod)	assigned-at-create: the external NAT IP of the first NIC (toInstance)	discovered: ARP table first, then the guest agent, first routable IPv4 (address, firstIPv4)
gc-mark	native-labels: Pod labels (buildPod)	native-labels: instance labels (buildInsertRequest)	written-metadata: a namespaced <metadata> element written in DomainDefineXML (metadata.go)
created-at	backend-timestamp: CreationTimestamp	backend-timestamp: CreationTimestamp, parse error gives zero	written-at-create: createdAt in the metadata element (newInstanceMetadata)
list-partial	n/a: one call, all namespaces	omit-unreachable: ReturnPartialSuccess on AggregatedList	fail-whole per host, one unreadable domain skipped (ListByTag, managedInstance)
quota	absent: a quota 403 is Permanent (classify)	detected: 429, and 403 with reason quotaExceeded or rateLimitExceeded (classify, hasReason)	absent (classify)
not-found	status-404: apierrors.IsNotFound	status-404: isNotFound	error-code: ErrNoDomain, ErrNoStorageVol (hasCode)
primary-key	name-is-key: namespace/name	name-is-key: zones/<zone>/instances/<name>	name-is-key: hosts/<host>/domains/<name>
provisioning-state	native: PodPending	native: PROVISIONING, STAGING, REPAIRING	create-blocks: a domain is defined and started in one Create; nostate maps to Provisioning
async-ops	none	task-handle, but the handle is discarded: an insert that fails after acceptance is never read	none
secondary-resources	none	cascade: AutoDelete on the boot disk (buildInsertRequest)	explicit: overlay and seed removed by name (removeVolumes); base image kept
credentials	workload-identity: in-cluster config or kubeconfig (NewWithTransportWrapper)	environment: Application Default Credentials	mounted-files: SSH key and known_hosts from the optional kvm-ssh Secret, read lazily (newLazySSH)
client-kind	vendor-sdk: controller-runtime client.Client, no seam interface	vendor-sdk: cloud.google.com/go/compute behind instancesAPI	rpc-over-ssh: go-libvirt over an SSH-forwarded unix socket, behind libvirtAPI (conn.go)
timeouts	ctx-only: no rest.Config.Timeout is set	ctx-only: SDK defaults, with SDK retries on Get and List	per-call for the dial only (defaultHandshakeTimeout in dialSSH); RPCs are ctx-only and a cancelled call closes the connection
logging	none	tiered: V(1) outcomes and logAPIError, V(2) detail, V(5) wire log (wirelog.go)	tiered: V(1) outcomes and one libvirt call line per RPC (trace.go), V(2) XML and addresses; no wire tier
child-spans	transport-wrapper: tracing.RestConfigWrapper() on its own rest.Config, parent-gated	sdk-instrumentation: the SDK adds otelhttp itself; not parent-gated, but every call site has a parent	none: trace.go is a log decorator, not spans
backend-test	cluster-e2e: kind (test/e2e)	manual-runbook: gcp-vm-validation.md	integration-tag: kvm_integration_test.go, make test-integration-kvm

2. Other differences

Rows that are not profile axes but that a new provider author compares against.

Feature	kubernetes	gcp	kvm
Create verifies the adopted instance's UID	no	no	no
Placement used	none	Zone, MachineType, Image	Zone (host), MachineType, Image
CloudInit delivered	no: ignored	yes: user-data metadata	yes: NoCloud seed ISO built in-process (buildSeedISO)
Port used	yes: Squid config and containerPort (squidConf)	no: firewall is manual	no
States produced	Provisioning, Running, Terminated	all four	Provisioning, Running, Stopped
Deleting instance still reports its state	yes: a Pod being deleted reports its phase	n/a	n/a
Error.Provider is the configured name	no: hardcoded "kubernetes"	yes	yes
Checker target	apiserver	the project	each host, concurrently
Checker Details	none	none	user, pool, hostname, libvirtVersion, poolState, poolCapacity, poolAvailable
Credential rotation without restart	n/a	via ADC	no: cached after first successful load
Connection handling	client-go	SDK client, never closed	one pooled SSH plus libvirt connection per host, 15s keepalive (hostPool)
Unit-test double	controller-runtime fake client	fakeAPI behind instancesAPI, plus httptest for the real SDK client in check_test.go	fakeAPI behind libvirtAPI, plus an in-process x/crypto/ssh server in conn_test.go
Proxy software	generated Squid config (pod.go)	from cloudInit; the image must run cloud-init	from cloudInit, plus a vendor-data script that installs the guest agent (seed.go)
IP reachable from the operator	Pod network, so in-cluster only (not from make run-dev on a laptop)	external IP, once a firewall rule allows the cluster's egress IP	LAN address on the host's bridge; the operator needs a route to that LAN
Distinct egress IP per proxy	no: the cluster's egress	yes: ephemeral external IP	whatever the host's LAN routes through; not per VM by itself

3. Observability status

Layer	kubernetes	gcp	kvm
1. Operation spans and metrics	yes: decorators	yes: decorators	yes: decorators
2. Child spans per backend call	yes: tracing.RestConfigWrapper(), parent-gated (cmd/main.go, NewWithTransportWrapper)	yes: the SDK's own otelhttp; not parent-gated	no
3. Logs carry traceID and spanID	n/a: the package does not log	yes, including the V(5) wire log	yes: every libvirt call line at V(1)
Tested against a real collector	yes: the kind e2e checks provider.create and provider.delete in Tempo (test/e2e/tracing_test.go), operation spans only	no	no

4. kubernetes (internal/provider/kubernetes)

• Package and seam. kubernetes.go, pod.go, check.go. There is no seam interface: newWithClient injects a controller-runtime client.Client, and tests use the fake client. New calls NewWithTransportWrapper, which builds its own uncached client from ctrl.GetConfig() with an optional transport.WrapperFunc for tracing.

• Config. KubernetesConfig.Image only, defaulting to the Squid image constant in kubernetes.go.

• ID. providerID(namespace, name) gives namespace/name; parseProviderID uses cache.SplitMetaNamespaceKey and rejects an empty namespace with ErrPermanent. An empty name part is not rejected.

• Create. buildPod (pure) then client.Create; IsAlreadyExists is success. Only Name, Namespace, UID and Port are used. The Squid config is generated by squidConf from the port and passed through the SQUID_CONF environment variable, so there is exactly one object per proxy.

• Get and state. stateFromPod:

  Pod phase	PodIP	Provider state
  Pending	any	Provisioning
  Running	empty	Provisioning
  Running	set	Running
  Succeeded, Failed, Unknown	any	Terminated

  instanceFromPod sets IP only when the state is Running. A Pod with a deletion timestamp still reports its phase. UID from the LabelUID label, CreatedAt from CreationTimestamp.

• Delete. client.Delete, IsNotFound ignored.

• ListByTag. One client.List across all namespaces with MatchingLabels{LabelManaged: LabelManagedYes}; this is what the cluster-wide Pod RBAC is for.

• Errors. classify: IsNotFound NotFound; IsTooManyRequests, IsServerTimeout, IsTimeout Transient; IsForbidden, IsInvalid, IsBadRequest, IsUnauthorized Permanent; default Transient. Nothing is Quota, because a quota 403 and an RBAC 403 cannot be told apart.

• Check. Lists at most one managed Pod cluster-wide (client.Limit(1)), proving read and list access, not create. Target apiserver, address the rest config host. checkReason: context errors and timeouts ReasonTimeout; IsUnauthorized; IsForbidden; any APIStatus ReasonAPIError; else ReasonUnreachable.

• Logging and tracing. No log lines in the package. Child spans come from the parent-gated transport wrapper.

• Tests. kubernetes_test.go, pod_test.go, check_test.go over the fake client; the kind e2e (test/e2e) is the real-backend run.

• Deployment. RBAC markers for Pods live next to the reconciler's in internal/controller/proxy_controller.go. hack/providers-dev.yaml holds only this provider, for make run-dev.

• Known gaps.

  • Error.Provider is hardcoded to "kubernetes", so errors and logs show the wrong name when the configured name differs.
  • CloudInit and all of Placement are ignored.
  • No quota class: a quota 403 is Permanent.
  • A Pod being deleted still reports its phase, so it can read as Running.
  • No securityContext, no resources, and http_access allow all in pod.go.
  • No logging in the package, and no Checker Details.
  • No OwnerReference on Pods: a bypassed finalizer leaves the Pod behind.
  • The Pod RBAC is always installed, even when no kubernetes provider is configured.

5. gcp (internal/provider/gcp)

• Package and seam. gcp.go, insert.go, errors.go, check.go, wirelog.go. The seam is instancesAPI (Insert, Get, Delete, AggregatedList, Probe); realInstances adapts the SDK client and newWithAPI is the test constructor. New calls NewWithWireOptions, which takes the --gcp-wire-log-full-payloads flag from cmd/main.go.

• Config. GCPConfig: project required; network, networkTag, diskSizeGb defaulted in withDefaults.

• ID. formatProviderID(zone, name) gives zones/<zone>/instances/<name>; parseProviderID rejects anything else with ErrPermanent. The project comes from the config.

• Create. Validates Zone, MachineType and Image up front and returns ErrPermanent naming the missing field. buildInsertRequest (pure) builds one NIC with an ephemeral external IP (ONE_TO_ONE_NAT) and a boot disk with AutoDelete, labelled with both marks. Insert is submitted and never awaited; 409 (isAlreadyExists) is success.

• Get and state. mapState(status, ip):

  Instance status	NAT IP	Provider state
  PROVISIONING, STAGING, REPAIRING	any	Provisioning
  RUNNING	empty	Provisioning
  RUNNING	set	Running
  STOPPING, STOPPED, SUSPENDING, SUSPENDED	any	Stopped
  TERMINATED	any	Terminated
  anything else	any	Stopped

  toInstance reads the NAT IP of the first access config of the first NIC, UID from the label, and CreatedAt from CreationTimestamp with the parse error dropped (zero time on failure).

• Delete. Submits a delete operation and returns; 404 (isNotFound) is success. Every deletion poll submits a new operation.

• ListByTag. AggregatedList with managedFilter on the managed label only and ReturnPartialSuccess, so an unreachable zone is left out silently; the zone is recovered from the URL by lastPathSegment.

• Errors. classify: non-googleapi errors Transient; 404 NotFound; 429 Quota; 403 with reason quotaExceeded or rateLimitExceeded (hasReason) Quota; 400, 401 and other 403s Permanent; other codes Transient. logAPIError records the HTTP status and reasons at V(1) because classify collapses them.

• Check. Probe makes one AggregatedList page with maxResults=1 and the managed filter: credentials, the Compute API and compute.instances.list in one call. Target is the project, address compute.googleapis.com. checkReason: context errors ReasonTimeout; non-googleapi errors ReasonUnreachable; 401; 403; else ReasonAPIError.

• Logging. logger(ctx) from the context, named gcp with provider. V(2) for the curated insert request (never the user-data), V(1) per operation outcome, V(2) per listed instance. The wire log in wirelog.go runs at V(5) with payloads elided unless the flag is set.

• Tracing. The SDK adds otelhttp itself unless telemetry is disabled; not parent-gated.

• Tests. insert_test.go (pure builder), errors_test.go (code table), gcp_test.go (IDs, 409, state mapping, list filter and partial success) over fakeAPI; check_test.go also drives the real SDK client against httptest to prove exactly one request with maxResults=1. Real backend: the manual runbook in gcp-vm-validation.md.

• Deployment. Credentials by ADC; the runbook mounts a service-account key and points GOOGLE_APPLICATION_CREDENTIALS at it. A commented entry in both providers configs, config/samples/proxy_gcp.yaml, and demo scripts under docs/demo/.

• Known gaps.

  • Asynchronous insert failures are never read: an insert the API accepts but that fails later (quota, bad image) is seen only as NotFound on the next Get, and the reconciler creates again, so the reason never reaches status. gcp-vm-validation.md still says such errors do reach status.
  • External IP only: a VM without an external NAT IP stays Provisioning forever.
  • No firewall management: the rule for the proxy port is a manual prerequisite, and req.Port is unused.
  • The SDK client is never closed, and Placement.Region is unused.
  • Wrong label value in the docs: gcp-vm-validation.md filters on labels.proxy-operator-managed=yes; the value is true.
  • The sample cannot work: config/samples/proxy_gcp.yaml uses a debian-cloud image, which has no cloud-init, so its Squid install never runs (gcp-in-specific-project.md uses Ubuntu for that reason).

6. kvm (internal/provider/kvm)

The full user guide is kvm-provider.md; this section is the author's view.

• Package and seam. kvm.go (the four methods), conn.go (SSH dial, hostPool, libvirtAPI), image.go (imageCache), seed.go (NoCloud ISO and vendor-data), domainxml.go, volume.go, metadata.go, id.go, errors.go, check.go, trace.go. The seam is libvirtAPI, the twenty go-libvirt calls the provider uses; fake_test.go implements it in memory and newWithDialer injects the dialer and the HTTP client.

• Config. KVMConfig in internal/provider/config_kvm.go: sshKeyFile, knownHostsFile or insecureIgnoreHostKey, diskSizeGb, vendorData, installGuestAgent, images (url, sha256, format, firmware), machineTypes (vcpus, memoryMiB), hosts (name, address, user, socket, pool, bridge). validateKVM restricts image keys and host names with kvmIdentifier because they become a volume name and an ID segment.

• ID. formatProviderID(host, name) gives hosts/<host>/domains/<name>.

• Create. host resolves Placement.Zone (optional with one host), the machine type must be a catalogue key, resolveImage accepts a catalogue key or an http(s) URL. Then, inside one hostPool.call: DomainLookupByName, and if found ensureRunning; otherwise imageCache.ensure (download on first use, streamed through the operator into the pool with a sha256 marker volume), ensureOverlay (reused if it exists), writeSeed (always rebuilt: a cut-off upload must not be trusted), defineDomain with the metadata element, ensureRunning. Create returns only when the domain is running.

• Get and state. mapState(state, ip):

  libvirt state	IP	Provider state
  nostate	any	Provisioning
  running, blocked	empty	Provisioning
  running, blocked	set	Running
  paused, shutdown, shutoff, crashed, pmsuspended	any	Stopped

  address asks the ARP table first, then the guest agent, and firstIPv4 drops loopback, link-local and unspecified addresses. Metadata is read back by metadata and parsed by parseInstanceMetadata; a domain without the element reports an empty UID.

• Delete. removeDomain (destroy, tolerating not running or gone; then DomainUndefineFlags with NVRAM, managed save and snapshot metadata) and removeVolumes (overlay and seed by name, a missing pool is a no-op). A host no longer in the config is a logged no-op success.

• ListByTag. ConnectListAllDomains on every host; managedInstance keeps domains whose metadata says managed and skips a domain whose metadata or state cannot be read. One unreachable host fails the whole list.

• Errors. classify: isClassified errors keep their class; ErrNoDomain, ErrNoStorageVol NotFound; XML, unsupported, invalid-argument, auth and access errors, and a missing pool, Permanent; everything else Transient, including SSH handshake and auth failures. Nothing is Quota. logAPIError adds libvirtErrorFields.

• Check. checkHost per host concurrently through hostPool.probe, not call, so a check that runs out of time abandons its RPCs instead of closing the connection a reconcile may be using. Two cheap RPCs (ConnectGetLibVersion, ConnectGetHostname) then the pool must exist and be running. Reasons: the shared ones plus SSHMaterialUnavailable, SSHHandshakeFailed, LibvirtConnectFailed, LibvirtError, PoolNotFound, PoolNotActive, chosen by checkReason from the dialError stage. Details: user, pool, hostname, libvirtVersion, poolState, poolCapacity, poolAvailable.

• Connections. hostPool keeps one SSH plus libvirt connection per host, dialled lazily (lazySSH.dial, sshAuth.dial, dialSSH, connectLibvirt), with a 15s keepalive. hostEntry.acquire is a one-slot semaphore so waiting can end with the caller's context. A cancelled call closes the shared connection to unblock the RPC; probe abandons it and leaves the connection open. The SSH material is read once by lazySSH.load and cached.

• Logging. Logger from the context, named kvm, with provider, host and name. V(1): one line per operation outcome and one libvirt call line per RPC from the tracingAPI decorator in trace.go (rpc name, identifiers, duration, error fields). V(2): domain and volume XML, metadata, addresses. Upload and download bodies are never logged.

• Tracing. No child spans; trace.go is a log decorator.

• Tests. Pure builders (domainxml_test.go, volume_test.go, seed_test.go, id_test.go, metadata_test.go); errors_test.go, conn_test.go (pool behaviour and a real in-process SSH server); image_test.go, kvm_test.go (Create resumes after every partial step, every Delete already-gone case, ListByTag policies, logging tiers, a cloud-init sentinel never logged); trace_test.go; check_test.go. Real backend: kvm_integration_test.go behind //go:build integration, driven by KVM_TEST_* variables and make test-integration-kvm, described in testing.md.

• Deployment. The optional kvm-ssh Secret mounted at /etc/proxy-operator/kvm (config/manager/manager.yaml), make kvm-ssh-secret, a commented entry in both providers configs, config/samples/proxy_kvm.yaml.

• Known gaps (see also kvm-provider.md § Known gaps).

  • No quota or capacity detection.
  • Create blocks, including the first image download on each host.
  • GC blind spots: volumes without a domain are invisible to GC, and so are VMs on a host removed from the config, which Delete treats as done.
  • SSH material is cached after the first successful load, so rotating keys needs a restart.
  • Cancelling one operation cuts off the others on that host: a cancelled call closes the shared per-host connection.
  • SSH auth failures retry forever: SSH authentication and host-key failures are Transient, while libvirt auth errors are Permanent.
  • Create can return ErrNotFound when a volume or domain disappears during the call.
  • No reused-overlay check: an existing overlay is reused without checking its backing image or size.
  • Sample mismatch: config/samples/proxy_kvm.yaml places on host nas1, while both providers configs define only beelink.
  • Doc drift in kvm-provider.md: it says ARP is the fallback, but the code tries ARP first; it says host and name are on every log line, but the check and list paths lack them.

7. Gaps shared by the existing providers

• Adoption is not verified. None of the three checks LabelUID when Create finds an existing instance with the right name.
• The Create comment does not match kvm. The interface comment in internal/provider/provider.go says Create returns as soon as the request is submitted; kvm blocks until the domain has started.
• Stopped and Terminated are produced inconsistently. kubernetes never produces Stopped, kvm never produces Terminated, and the reconciler treats them the same.
• Different ListByTag partial-failure policies. gcp leaves failed parts out, kvm fails the whole list. Both are safe.
• Stale mock comments in internal/provider/provider.go, internal/provider/registry/registry.go and api/v1alpha1/proxy_types.go (and so the generated CRD's spec.provider description).
• GC does not know about installations. Two clusters sharing one project or hypervisor delete each other's instances. This was a merge-blocker in the 2026-08-10 PR review (docs/reviews) and is still open.
• Tracing. kvm has no child spans, so a kvm trace shows provider.create and nothing under it; correlation works only through traceID on its V(1) log lines. Only kubernetes is tested against a real collector, and that test checks only the operation spans; nothing checks that gcp's SDK spans or the kubernetes API spans nest under provider.*.

8. Proposed features

Listed only. Nothing below is designed or implemented.

• Child spans in kvm: extend the tracingAPI decorator in kvm/trace.go, which already wraps every libvirt call, to also start a span per RPC, and add spans for the SSH dial and the image download. The seam's methods take no ctx today (hostPool.call hands fn only the API), so the span context has to be plumbed in.
• Tempo e2e checks for child spans, and for gcp and kvm through their real backend runbooks.
• A shared conformance suite: for example internal/provider/providertest, which runs the contract §11 tests against any provider plus a fake backend, so a new provider gets the contract tests for free.
• Verify the UID when Create adopts an instance. Return ErrPermanent on a mismatch instead of adopting an instance that belongs to someone else.
• Report asynchronous create failures: gcp keeps the operation name, and Get reports the failure instead of NotFound.
• Capacity or quota signals for kvm, such as pool free space and host memory, mapped to ErrQuotaExceeded.
• GC for secondary resources: mark volumes and similar resources, and let ListByTag or a second listing method cover them.
• Reload credentials and config without a restart.
• Checker Details for kubernetes (server version) and gcp (quota headroom).
• Internal-IP mode for gcp, and a firewall rule managed per provider.
• An installation-identity mark, set on create and filtered on in ListByTag, so installations sharing a backend do not collect each other's instances.
• A permanent-failure latch that sees provider availability, so adding a missing provider releases the Proxies waiting on it.
• Validate provider names at load, and derive the config type allow-list from the constructor map.
• Per-provider RBAC as an optional kustomize component, and an OwnerReference on kubernetes Pods.