egress-proxies-operator/docs/providers.md

Providers: the blueprint

This page is for whoever writes the next provider. It explains the contract a provider must meet, which caller depends on each rule, how the three existing providers (kubernetes, gcp, kvm) meet it, and what is still missing.

Where the rest lives:

• Data flow: the calls each provider makes are in architecture.md §5.
• Health checks: the monitor that runs them is in architecture.md §11.
• Per-provider guides: kvm-provider.md and gcp-vm-validation.md.
• Code: the contract itself is internal/provider/provider.go, errors.go and check.go.

Rule keywords: MUST means breaking the rule breaks a caller. SHOULD means every current provider does it and a new one needs a reason not to. MAY means it's allowed.

Code references are path:line at the time of writing. When a line has moved, look for the function named next to it.

1. What a provider is

A provider is a Go type implementing provider.Provider: four methods over one VM-like instance, identified by an opaque providerID string. Nothing else in the operator knows how a backend works:

• The reconciler (internal/controller) calls Create, Get and Delete and branches only on the error class and Instance.State.
• Orphan GC (internal/gc) calls ListByTag and Delete.
• The provider health monitor (internal/providerhealth) calls the optional Checker.

The reconciler never watches backend resources. Everything it knows comes from polling Get (ProvisioningPoll and DeletionPoll 10s, RequeueNow 1s, QuotaRetry 5m, internal/controller/proxy_controller.go:452-467). Up to 3 Proxies reconcile at once (MaxConcurrentReconciles: 3, :442).

Providers are named instances of a type. gcp-eu and gcp-us can both have type gcp. The name is what Proxy.spec.provider refers to.

2. The contract, method by method

Create(ctx, CreateRequest) (providerID, error)

CreateRequest carries:

• Name: from NameFromUID, 22 characters, proxy- plus base32.
• UID: the owning Proxy's UID.
• Namespace and ProxyName.
• Placement: Region, Zone, MachineType, Image.
• CloudInit: already resolved from inline text or a Secret.
• Port.

Rules:

• MUST be idempotent by req.Name. A repeat call for a name that already exists returns that instance's providerID with a nil error. It must not return an error and must not create a duplicate. Why: the reconciler records the ID in status only after Create returns (proxy_controller.go:155-172). A crash between the two, or a status write that is lost, means Create runs again for the same Proxy. replaceInstance also depends on this to recover from a crash between its two writes (:240-244).

• MUST mark the instance as managed when it is created, with LabelManaged=true and LabelUID=<req.UID> or the backend's equivalent. kvm uses domain <metadata> because libvirt has no labels. Why: orphan GC finds instances only through that mark (internal/gc/gc.go:105), and never deletes one with an empty UID (gc.go:113-117). An instance that exists for a while without the mark leaks if the operator crashes.

• MUST return a providerID that is self-contained. Get and Delete receive only the ID, and the config or spec may have changed since Create, for example a zone edit during a replacement. So put the location in the ID:

  • kubernetes: <namespace>/<name>
  • gcp: zones/<zone>/instances/<name>
  • kvm: hosts/<host>/domains/<name>

  Validate any config values that end up in the ID so they can't contain the separator (config_kvm.go:108-114).

• MUST honour ctx. The reconcile context is cancelled on shutdown and when leadership is lost.

• SHOULD return once the request is submitted, not once the instance is running. Readiness is discovered by polling Get. A blocking Create holds one of the 3 reconcile workers for its whole duration. kvm breaks this rule on purpose, see §9.

• SHOULD reject a request it can never satisfy with ErrPermanent before touching the backend: a missing zone, machine type or image, or a name that is not in its catalogue (gcp gcp.go:137-141, kvm kvm.go:126-144, :160-164).

• MAY ignore Placement fields that don't apply (provider.go:56-57). If a field is ignored, the provider's doc page MUST say so, because the fields are still part of the spec hash. Changing an ignored field replaces the instance and has no other effect.

• SHOULD deliver CloudInit as user-data. A provider that can't (kubernetes) documents that it ignores the field.

• MUST NOT log CloudInit contents. It can hold secrets. Log its byte count instead. gcp and kvm test this with a sentinel string (kvm_test.go:975-995, gcp_test.go:330-407).

Get(ctx, providerID) (*Instance, error)

• MUST return ErrNotFound when, and only when, the instance is gone. Why: ErrNotFound triggers three actions:

  • In normal reconciling, the reconciler clears the ID and creates again (proxy_controller.go:197-202).
  • During replacement, it is the signal to create the new instance (:237-251).
  • During deletion, it releases the finalizer (:287-291).

  A false NotFound during an outage creates a duplicate or loses the instance. "Unreachable", "wrong credentials" and "can't tell" are Transient or Permanent, never NotFound.

• MUST keep returning the instance, not NotFound, while a delete is in progress and the name is still taken. Why: the replacement instance gets the same name. The reconciler waits for NotFound before creating it (:228-233 comment, :237-251). A NotFound too early would make Create adopt the dying instance.

• MUST report Running only with a non-empty IP. Running without an address yet is Provisioning. Why: the IP from a Running answer goes straight into status and the health engine (:212-215). All three providers do this.

• MUST report Stopped or Terminated only when the instance will not recover by itself. SHOULD map states it doesn't recognise to Stopped. Why: the reconciler deletes and recreates on either state (:216-226); the instances are treated as disposable. The two states cause exactly the same reaction.

• MUST fill UID and CreatedAt by reading them back from the backend, never from memory. Why: GC compares UID with the live Proxies. It deletes anything whose CreatedAt is older than MinAge (10m), and a zero CreatedAt counts as very old (gc.go:120). A provider that forgets CreatedAt makes instances still being created collectable. An unparsable timestamp gives the zero time in both gcp and kvm. That is accepted because it never shields an orphan forever (gcp.go:253-256).

• SHOULD return ErrPermanent for an ID it can never resolve, such as a malformed ID or a host no longer in the config. Why: the deletion path then tries Delete anyway and releases the finalizer if that works (:292-302). If the provider returned Transient, deletion would poll forever.

Delete(ctx, providerID) error

• MUST be idempotent. Deleting something that is already gone returns nil. Why: the reconciler calls Delete again on every poll until Get returns NotFound (:305-311). GC and the reconciler can also race.
• MUST remove everything Create made: disks, volumes, seeds, addresses. Either delete them explicitly or rely on the backend doing it (gcp AutoDelete on the boot disk). Why: GC sees only what ListByTag returns, so secondary resources have no other cleanup path.
• MAY return before the instance is gone (gcp, kubernetes), as long as Get keeps showing it until it is (see Get).
• SHOULD succeed for an ID whose Get fails permanently when there is nothing left it could reach. Example: kvm on a host that was removed from the config returns nil (kvm.go:466-476). Why: on the deletion path a permanent error isn't latched. It keeps retrying (deletionFailure, :349-359), so a Proxy is only released through the crawl.example.com/skip-cleanup annotation.

ListByTag(ctx) ([]Instance, error)

• MUST return only instances that carry the managed mark, each with an ID that Delete accepts, plus UID and CreatedAt. Why: GC deletes every listed instance whose UID isn't live and whose age is past MinAge (gc.go:111-131). Listing something without the mark is how an operator deletes a VM it doesn't own.
• MUST return an error rather than a list it can't vouch for. An error makes GC skip that provider for the sweep (gc.go:105-110).
• MAY leave out instances it couldn't read. Leaving something out is always safe, because GC never deletes what isn't listed. The current providers make different choices here:
  • gcp silently leaves out unreachable zones (ReturnPartialSuccess, gcp.go:229).
  • kvm fails the whole list when one host is down but skips a single unreadable domain (kvm.go:540-565, :580-596).
  • kubernetes makes a single call.
• SHOULD cover the whole scope the provider can create in: every namespace, zone or host. Anything outside that scope is never collected.
• Scope caveat: GC protects instances by UID, not by operator install. Two installs, or two clusters, sharing one GCP project or hypervisor will delete each other's instances (gcp-vm-validation.md). Two provider entries of one install that share a scope are safe, because they share the live-UID set.

3. Error taxonomy

Every error a provider method returns MUST be built with provider.Wrap(class, op, providerName, id, err). op is one of create, get, delete or list, and providerName is the configured name (cfg.Name), not the type.

Wrap keeps both errors.Is(err, class) and errors.As to the SDK error working (errors.go:53-62). The shared metrics and tracing decorators label results with the class (metrics.go:61-74, tracing.go:67-80).

Class	Use it for	During normal reconciling	While deleting
ErrNotFound	the instance doesn't exist	Get: recreate or advance (§2); other ops: returned as the error, so exponential backoff	Get: release the finalizer
ErrQuotaExceeded	quota or rate limit	Provisioned=False with reason QuotaExceeded, requeue after QuotaRetry (5m), not counted as an error (:337-340)	requeue after QuotaRetry
ErrTransient	network, 5xx, timeouts, cancellation, anything unknown	error returned, so workqueue exponential backoff	same
ErrPermanent	bad config, bad argument, auth or permission denied, malformed ID	latched: Provisioned=False with reason PermanentError, no more provider calls until metadata.generation changes (:133-139, :341-343)	error returned and retried forever; Get Permanent also triggers Delete (:292-302)

Pitfalls:

• Unclassified means Transient. provider.Class returns ErrTransient for any error not built with Wrap (errors.go:71-91). That is the safe default, but a permanent failure that was never classified retries forever, and a quota error backs off far too fast.
• Permanent is sticky. Use it only for errors that editing the Proxy spec or fixing the config can resolve. A 403 that could be a quota problem is the hard case: gcp checks the error reason (gcp/errors.go:31-34), while kubernetes can't tell quota from RBAC and picks Permanent (kubernetes.go:201-206).
• Don't return NotFound from Create. The reconciler treats it as Transient there, but it misleads the metrics and anyone reading logs. kvm can currently do this (§9).
• Classify in one classify function with a table-driven test. All three providers do.

4. Identity, names and GC marks

• Names: provider.NameFromUID(uid) gives proxy- plus 16 lowercase base32 characters (name.go:29-33). It is valid as a GCP name, a DNS label and a libvirt domain name. A backend with stricter rules SHOULD map it deterministically rather than invent a new scheme, because idempotency depends on the name being the same every time.
• providerID: opaque to everyone except the provider. Keep it <location>/<name>-shaped, parse it strictly, and return ErrPermanent for a malformed one (§2).
• Marks: LabelManaged = "proxy-operator-managed" with value "true", and LabelUID = "proxy-operator-uid" (provider.go:28-32). Use native labels or tags where the backend has them. Otherwise store them in metadata that is written in the same call that creates the instance (kvm: metadata.go).
• CreatedAt: read it from the backend's own creation timestamp. If the backend has none, write it into the metadata at creation (kvm: kvm.go:277-298).

5. Optional interfaces: Checker

provider.Checker is Check(ctx) []TargetStatus (check.go:37-46). cmd/main.go type-asserts it on the unwrapped provider (cmd/main.go:216) before adding the decorators (:219-222). The monitor treats a nil Checker as unsupported, not as failed (monitor.go:212).

• MUST never fail as a whole. Every failure is a TargetStatus with Ready=false.
• MUST return promptly once ctx ends. The monitor gives the whole round one timeout (monitor.go:180).
• MUST return one entry per target, in a stable order. Targets are what can fail independently: the apiserver, a project, each hypervisor.
• MUST NOT disrupt reconciles. A check that shares a connection with provisioning must not tear that connection down when it times out. kvm uses hostPool.probe for this, not call (kvm/check.go:65, kvm/conn.go:213-242).
• SHOULD use the shared reasons (Reachable, Timeout, Unreachable, Unauthorized, Forbidden, APIError). Add provider-specific reasons only for failures an admin would fix differently: kvm has SSHHandshakeFailed, PoolNotFound and others.
• SHOULD check with the same credentials, verb and scope that ListByTag and Create use, and be cheap: one page, one object.
• MAY fill Details with facts useful when triaging, such as versions or capacity. Only kvm does today.

6. Observability: tracing, logs and metrics

Tracing is OpenTelemetry, set up in internal/tracing from the standard OTEL_* environment variables. With no exporter configured, spans are no-ops and logs are unchanged (architecture.md §10). A provider has three layers to think about; the first comes for free.

1. Operation spans and metrics: the decorators, nothing to write. cmd/main.go:219-222 wraps every provider in WithTracing(WithMetrics(...)):

• provider.WithTracing opens one client span per call, provider.create, provider.get, provider.delete or provider.list, with provider.name, provider.id and provider.result (the error class) as attributes (internal/provider/tracing.go:27-80). ErrNotFound keeps the span status Ok, because polling to NotFound is expected.
• provider.WithMetrics counts each call by provider, op and result (metrics.go:15-74).
• The spans sit under Reconcile Proxy → reconcile.managed | reconcile.replaceInstance | reconcile.delete, or under gc.sweep.

Rules:

• MUST NOT open its own per-operation span or count its own requests. They would duplicate the decorators.
• MUST pass the caller's ctx to every backend call and log through logf.FromContext(ctx) at call time. The decorator starts the span with tracing.StartSpan, which puts a logger carrying traceID and spanID into ctx (internal/tracing/logger.go:32-44). A logger cached at construction, or a backend call made with context.Background(), loses the trace.
• Health checks bypass the decorators (cmd/main.go:211-218). A Checker runs under the monitor's providerhealth.check span (monitor.go:176), and the same ctx rules apply to it.

2. Child spans for backend calls: SHOULD. One operation span says that Create took 40 seconds. Child spans say which backend call took them. Every provider SHOULD produce one child span per backend request, under the operation span:

• HTTP backends: wrap the SDK's transport with otelhttp. Gate it on an existing parent span, as tracing.RestConfigWrapper does (internal/tracing/transport.go:12-51). Otherwise background traffic such as watches, token refreshes or keepalives becomes root spans. Check first whether the SDK already does this: Google's does.
• Non-HTTP backends (RPC over SSH, gRPC without interceptors, CLIs): start child spans yourself with tracing.Start(ctx, "<type>.<call>"), around each RPC and around long steps such as a dial or an image download. Use a decorator over the SDK seam, not call-site code.
• MUST NOT put secrets or CloudInit contents in span attributes. Put identifiers, sizes and error codes there.

3. Logs. V(1): one line per operation outcome and per failed backend call. V(2): curated detail. V(5): raw wire, if the backend has one. Log lines are matched to traces by the traceID key from layer 1, which is how kvm's per-RPC log lines join a trace today.

What each provider does now:

Layer	kubernetes	gcp	kvm
1. Operation spans and metrics	yes: decorators	yes: decorators	yes: decorators
2. Child spans per backend call	yes: tracing.RestConfigWrapper() on its rest.Config (cmd/main.go:196-200, kubernetes.go:58-60), parent-gated	yes: the Google SDK adds otelhttp itself unless telemetry is disabled (google.golang.org/api/transport/http/dial.go:308-312); not parent-gated, but every call site has a parent	no: kvm/trace.go is a log decorator, not spans
3. Logs carry traceID/spanID	n/a: the package doesn't log	yes, including the V(5) wire log (gcp/wirelog.go:54-59)	yes: every libvirt call line at V(1)
Tested against a real collector	yes: kind e2e checks the provider.create and provider.delete traces in Tempo (test/e2e/tracing_test.go:200-238), operation spans only	no	no

7. Feature matrix

Legend: yes / partial / no, with a short note. The evidence is in the per-provider notes below the table.

Feature	kubernetes	gcp	kvm
providerID self-contained	yes: ns/name	yes: zones/z/instances/n; project from config	yes: hosts/h/domains/n
Create idempotent	yes: AlreadyExists = success	yes: 409 = success	yes: lookup by name; overlay reused, seed rebuilt
Create verifies adopted instance's UID	no	no	no
Create non-blocking	yes	yes: operation not awaited	no: waits for DomainCreate; first use of an image downloads it
Placement used	none	Zone, MachineType, Image	Zone (host), MachineType, Image
CloudInit delivered	no: ignored	yes: user-data metadata	yes: NoCloud seed ISO
Port used	yes: Squid config and containerPort	no: firewall is manual	no
GC marks on create	yes: Pod labels	yes: instance labels	yes: domain <metadata>; volumes unmarked
States produced	Provisioning, Running, Terminated	all four	Provisioning, Running, Stopped
IP source	PodIP	external NAT IP only	ARP, then guest agent; first IPv4
UID / CreatedAt readback	yes	yes	yes
Delete idempotent	yes	yes	yes, at each step
Delete waits until gone	no	no	yes
Delete cleans secondary resources	n/a	yes: boot disk auto-deleted	yes: overlay and seed; base image kept
ListByTag scope	cluster-wide	one project, all zones	configured hosts
ListByTag on partial failure	n/a: single call	leaves out failed zones	whole list fails if a host is down; unreadable domain skipped
Quota detection	no	yes: 429 and 403 quota reasons	no
Error.Provider = configured name	no: hardcoded "kubernetes"	yes	yes
Checker	yes: apiserver	yes: the project	yes: each host
Checker Details	no	no	yes
Credentials	in-cluster config / kubeconfig	ADC	SSH key + known_hosts files, usually a Secret
Credential rotation without restart	n/a	via ADC	no: cached after first load
Connection handling	client-go	SDK client; SDK retries on Get and List	one pooled SSH+libvirt connection per host, keepalive
Logging	none in the package	V(1) ops, V(2) detail, wire log	V(1) ops and RPCs, V(2) detail
Tracing: operation spans (§6)	yes	yes	yes
Tracing: child spans per backend call	yes: parent-gated transport	yes: SDK's otelhttp	no
Unit tests with a fake	yes: controller-runtime fake client	yes: fakeAPI + httptest	yes: fakeAPI + in-process SSH server
Real-backend test	kind e2e	manual runbook	//go:build integration

kubernetes notes (internal/provider/kubernetes)

• ID: kubernetes.go:152-165. An empty name part is not rejected.
• Create: kubernetes.go:90-100 and pod.go:23-50. Only Name, Namespace, UID and Port are used. Squid config is generated from the port and passed through SQUID_CONF (pod.go:38-75).
• State: stateFromPod, kubernetes.go:185-199. Pending, or Running with no PodIP, is Provisioning. Succeeded, Failed and Unknown are Terminated. A Pod being deleted still reports its phase.
• Delete: kubernetes.go:118-127. ListByTag: kubernetes.go:134-146.
• Errors: classify, kubernetes.go:201-218. 429 and timeouts are Transient. Forbidden, Invalid, BadRequest and Unauthorized are Permanent. Nothing is Quota.
• Check: check.go:21-53. It lists at most one managed Pod cluster-wide, so it proves read and list access, not create.
• Config: only kubernetes.image (config.go:45-48, default at kubernetes.go:28).

gcp notes (internal/provider/gcp)

• ID: gcp.go:289-299.
• Create: gcp.go:135-167 and insert.go:34-74. Operation.Wait is never called. There is one NIC with an ephemeral external IP, and the boot disk has AutoDelete.
• State: mapState, gcp.go:271-287. RUNNING without a NAT IP is Provisioning. Unknown statuses are Stopped.
• Get: gcp.go:172-192. Delete: gcp.go:196-219. ListByTag: gcp.go:224-244.
• Errors: errors.go:20-39. Non-googleapi errors are Transient. 404 is NotFound. 429 and 403 with reason quotaExceeded or rateLimitExceeded are Quota. 400, 401 and other 403s are Permanent.
• Check: check.go:30-78. One AggregatedList page with maxResults=1.
• Config: project is required. network, networkTag and diskSizeGb have defaults (config.go:51-61, insert.go:12-30).
• Logging: V(1) and V(2) tiers in gcp.go. The wire log in wirelog.go runs at V(5), with payloads elided unless --gcp-wire-log-full-payloads is set.

kvm notes (internal/provider/kvm)

• ID: id.go:11-21. Host names are restricted by config_kvm.go:108-114.
• Create: kvm.go:154-221:
  • Looks up the domain by name and starts it if needed (ensureRunning, :300-318).
  • Otherwise: base image (image.go), then overlay (reused if it exists, :225-245), then seed (always rebuilt, :249-275), then define with metadata (:277-298), then start.
• Get: kvm.go:323-356. States come from mapState (:442-453): nostate is Provisioning, running or blocked without an IP is Provisioning, everything else is Stopped.
• Addresses: address (:397-428) tries ARP first, then the guest agent.
• Delete: kvm.go:461-538. Destroy, undefine (with NVRAM, managed save and snapshot metadata), then remove the overlay and seed by name. A host that is no longer configured is a success.
• ListByTag: kvm.go:545-596.
• Errors: errors.go:21-42. No domain or no volume is NotFound. XML, unsupported, invalid-argument, auth and access errors, and a missing pool, are Permanent. Everything else is Transient, including SSH handshake and auth failures. Nothing is Quota.
• Check: check.go:41-139. Runs concurrently per host through hostPool.probe. Reasons are tagged by the dial stage.
• Connections: conn.go. One connection per host with a 15s keepalive. The pool dials on first use; with provider checks on (the default), the leader's first health check opens it at startup. On cancel, call closes the shared connection; probe abandons the RPC and leaves the connection open.
• Full guide: kvm-provider.md.

8. Adding a provider: checklist

1. Package: create internal/provider/<type>/ with the provider type and a compile-time interface assertions: var _ provider.Checker = (*Provider)(nil) as in kubernetes/check.go:15, and the same for provider.Provider, which the existing providers don't assert yet.

   • Put the backend SDK behind a small interface (gcp instancesAPI, kvm libvirtAPI) so a fake can stand in for it in tests.
   • Build the logger with logf.FromContext(ctx).WithName("<type>").WithValues("provider", name).

2. Config: in internal/provider/config.go:

   • Add a <Type>Config block and a ProviderConfig field.
   • Add the block to setBlocks.
   • Add the type to the switch p.Type allow-list (config.go:102-108).
   • Add required-field validation, in its own config_<type>.go if it is large.
   • Add rows to config_test.go.

   The file is parsed with UnmarshalStrict, so unknown keys already fail.

3. Constructor: New(ctx, provider.ProviderConfig) (provider.Provider, error), registered in the registry.Build map in cmd/main.go:195-205.

   • A constructor error stops the manager.
   • So fail only on config errors, and don't dial the backend here: one unreachable hypervisor or API must not take down the other providers (kvm logs a warning and connects lazily, kvm.go:89-101).

4. Decorators: nothing to do. cmd/main.go:219-222 wraps every provider in WithTracing(WithMetrics(...)). Don't count requests or open per-operation spans yourself.

5. Contract: implement each rule in §2, and test each one against the fake:

   • repeat Create
   • Get after Delete
   • Delete twice
   • malformed ID
   • unknown state mapped to Stopped
   • Running requires an IP
   • UID and CreatedAt read back
   • ListByTag excludes unmarked instances

6. Errors: one classify plus a table test covering every class, including "not an SDK error → Transient". Use cfg.Name in provider.Wrap.

7. Checker: shared reasons first, a table test for the reasons, and a test that Check returns promptly with an expired ctx.

8. Tracing and logging (§6):

   • Child spans for backend calls: an existing SDK instrumentation, a parent-gated transport wrapper, or a span decorator over the SDK seam. Add a test with an in-memory span recorder (sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(tracetest.NewSpanRecorder())), as in internal/provider/tracing_test.go) asserting the spans are children of the operation span.
   • Take the logger from ctx on every call so log lines carry traceID.
   • V(1): one line per operation outcome, plus a logAPIError-style line with the raw backend error.
   • V(2): request and response detail.
   • Never put CloudInit contents in logs or span attributes; add a sentinel test for it.

9. Real-backend test: behind //go:build integration, skipped when its environment variables are unset. It runs the whole lifecycle: create, create again, poll until Running, list, delete, delete again, and confirm it is gone and cleaned up. See kvm/kvm_integration_test.go and testing.md.

10. RBAC: if the provider talks to the cluster, add +kubebuilder:rbac markers in their own comment block, not inside a doc comment. The kubernetes provider's Pod markers live next to the reconciler's (internal/controller/proxy_controller.go:86-88); controller-gen scans ./..., so either place works. Then run make manifests and check git diff config/rbac/role.yaml.

11. Docs:

    • A docs/<type>-provider.md covering prerequisites, config reference, placement mapping, what lands on the backend, the health check, and known gaps.
    • A row in the architecture.md components table, and a block in §5 and §11. If the provider adds spans, update the span topology in §10.
    • A column in the §7 matrix above.
    • A sample in config/samples/.

12. Optional: a demo script under docs/demo/.

9. Gaps and proposed features

These are listed only. Nothing below is designed or implemented.

Current gaps

Across providers and the contract

• Adoption isn't verified. Create adopts any existing instance with the right name without checking its LabelUID. None of the three providers check it.
• The Create comment doesn't match kvm. The contract says Create returns as soon as the request is submitted (provider.go:90-94). kvm blocks until the domain has started, and on first use of an image, for the whole download.
• Stopped and Terminated are indistinguishable to the reconciler. kubernetes never produces Stopped, and kvm never produces Terminated.
• Different ListByTag partial-failure policies. gcp leaves failed parts out, kvm fails the whole list. Both are safe for GC; the contract doesn't say which to prefer.
• Config is read once per process. Adding a provider or host, or rotating credentials, needs a manager restart.
• Stale comments. provider.go:3-4 and registry/registry.go:5-12 still describe a mock subpackage, which was replaced by the kubernetes provider.

kubernetes

• Wrong provider name in errors. Error.Provider is hardcoded to "kubernetes" (kubernetes.go:105-216), so errors and logs show the wrong name when the configured name differs.
• Spec fields ignored. CloudInit and all of Placement are ignored.
• No quota class. A quota 403 is Permanent.
• Deleting Pods look healthy. A Pod being deleted still reports its phase, so it can read as Running.
• Unguarded Pods. No securityContext, no resources, and http_access allow all (pod.go).
• No logging in the package, and no Checker Details.

Tracing

• kvm has no child spans. A kvm trace shows provider.create and nothing under it: no libvirt RPCs, no SSH dial, no image download, although that is where the time goes. Correlation works only through the traceID on its V(1) log lines, which needs debug logging on.
• Only kubernetes is tested against a real collector, and that test checks only the operation spans. Nothing checks that gcp's SDK spans or the kubernetes API spans actually nest under provider.*.

gcp

• Asynchronous insert failures are never read. An insert that the API accepts but that fails later (quota, bad image) is never seen. The next Get returns NotFound and the reconciler creates it again, so the reason never reaches status. gcp-vm-validation.md says such errors do reach status.
• External IP only. A VM without an external NAT IP stays Provisioning forever.
• No firewall management. The rule for the proxy port is a manual prerequisite, and req.Port is unused.
• The SDK client is never closed, and Placement.Region is unused.
• Wrong label value in the docs. gcp-vm-validation.md filters on labels.proxy-operator-managed=yes; the value is true.

kvm (see also kvm-provider.md § Known gaps)

• No quota or capacity detection.
• Create blocks, including the first image download on each host.
• GC blind spots. Volumes without a domain are invisible to GC, and so are VMs on a host removed from the config. Delete treats the latter as done.
• SSH material is cached after the first successful load, so rotating keys needs a restart.
• Cancelling one operation cuts off the others on that host. A cancelled call closes the shared per-host connection, aborting other operations in flight on that host (conn.go:199-211).
• SSH auth failures retry forever. SSH authentication and host-key failures are Transient, while libvirt auth errors are Permanent.
• Create can return ErrNotFound when a volume or domain disappears during the call.
• No reused-overlay check. An existing overlay is reused without checking its backing image or size.
• Doc drift in kvm-provider.md:
  • It says ARP is the fallback; the code tries ARP first.
  • It says host and name are on every log line; the check and list paths lack them.

Proposed features (not implemented)

• Child spans in kvm: extend the tracingAPI decorator in kvm/trace.go, which already wraps every libvirt call, to also start a span per RPC, and add spans for the SSH dial and the image download. The seam's methods take no ctx today (hostPool.call hands fn only the API), so the span context has to be plumbed in.
• Tempo e2e checks for child spans, and for gcp and kvm through their real backend runbooks.
• A shared conformance suite: for example internal/provider/providertest, which runs the §2 rules against any provider plus a fake backend, so a new provider gets the contract tests for free.
• Verify the UID when Create adopts an instance. Return ErrPermanent on a mismatch instead of adopting an instance that belongs to someone else.
• Report asynchronous create failures: gcp keeps the operation name, and Get reports the failure instead of NotFound.
• Capacity or quota signals for kvm, such as pool free space and host memory, mapped to ErrQuotaExceeded.
• GC for secondary resources: mark volumes and similar resources, and let ListByTag or a second listing method cover them.
• Reload credentials and config without a restart.
• Checker Details for kubernetes (server version) and gcp (quota headroom).
• Internal-IP mode for gcp, and a firewall rule managed per provider.