egress-proxies-operator/docs/provider-contract.md

The provider contract

This document is the specification a provider must meet. It is written for whoever builds the next one, human or agent, and it deliberately says nothing about how any existing provider is built. Those details are in providers-existing.md; the questionnaire a new provider's design must answer is provider-spec-template.md. Read the three in that order: this contract, then the template, then the appendix for worked examples.

Rule keywords: MUST means breaking the rule breaks a caller, and every MUST below names the caller and what breaks. SHOULD means every current provider does it and a new one needs a written reason not to. MAY means it is allowed and the choice must be recorded in the provider's spec.

Code references are a file path plus a function or symbol name, never a line number. Where a rule is enforced by a caller, that caller is one of:

• the reconciler, internal/controller/proxy_controller.go;
• orphan GC, internal/gc/gc.go;
• the provider health monitor, internal/providerhealth/monitor.go;
• the decorators, internal/provider/metrics.go and internal/provider/tracing.go;
• the composition root, cmd/main.go.

1. What a provider is and who calls it

1.1 The types

A provider is a Go type implementing provider.Provider (internal/provider/provider.go): four methods over one VM-like instance, identified by an opaque providerID string.

• Create(ctx, CreateRequest) (providerID, error)
• Get(ctx, providerID) (*Instance, error)
• Delete(ctx, providerID) error
• ListByTag(ctx) ([]Instance, error)

CreateRequest carries Name (from NameFromUID in internal/provider/name.go, 22 characters, proxy- plus base32), UID (the owning Proxy's UID), Namespace, ProxyName, Placement (Region, Zone, MachineType, Image), CloudInit (already resolved from inline text or a Secret) and Port.

Instance carries ID, IP, State (one of StateProvisioning, StateRunning, StateStopped, StateTerminated), UID and CreatedAt.

The GC marks are the constants LabelManaged (value LabelManagedYes) and LabelUID, also in provider.go.

Providers are named instances of a type: two entries of one type with different config blocks are two providers. The configured name is what Proxy.spec.provider refers to, and it is immutable, so a Proxy never moves between providers.

1.2 The callers

Nothing above the provider watches backend resources. Everything the operator knows about an instance comes from polling Get.

• The reconciler calls Create, Get and Delete from reconcileManaged, replaceInstance and reconcileDelete, and branches only on the error class (providerFailure, deletionFailure) and on Instance.State. It maps the CRD's placement to Placement one to one in placementFrom. Its poll intervals are set in applyDefaults: ProvisioningPoll and DeletionPoll 10s, DriftPoll 2m, RequeueNow 1s, QuotaRetry 5m. Up to 3 Proxies reconcile at once (MaxConcurrentReconciles in SetupWithManager).
• Orphan GC calls ListByTag and Delete from Sweeper.sweep, every Interval (10m), and never deletes an instance younger than MinAge (10m). It loops over providers one after another in a single goroutine.
• The provider health monitor calls the optional Checker from Monitor.checkOnce, every --provider-check-interval, all providers concurrently under one timeout.
• The composition root builds every provider through registry.Build (internal/provider/registry/registry.go) before the manager starts, on every replica, then wraps each one in WithTracing(WithMetrics(...)).

Where the calls come from. The reconciler, GC, the health engine and the monitor run on the leader only. The constructors run on every replica at startup.

Concurrency. Up to 3 reconciles, a GC sweep and a health-check round can call the same provider at the same time, and GC's Delete can overlap the reconciler's Delete for the same instance. A provider MUST be safe for concurrent use, including two calls for the same instance name. Nothing above the provider serialises them.

1.3 What the instance is for

The instance is a forward proxy with its own egress path. Two parties connect to http://<Instance.IP>:<spec.port>:

• the health engine, from the operator pod, which sends a real CONNECT through it (probe in internal/health/probe.go);
• clients, which get ip and port from the discovery API (viewOf in internal/discovery/handlers.go).

So the IP a provider reports MUST be reachable on that port from the operator's pod network and from the clients. Whatever that takes on the backend (firewall rules, security groups, routing) is either done by the provider or documented as a prerequisite in the provider's guide.

The provider does not know which proxy software runs. The proxy is installed and configured by the Proxy's cloudInit; building install logic into the operator is a stated non-goal (docs/prompts/__initial-prompt.md, non-goals). A backend that cannot take user-data (see readiness in §4) documents that it ignores the field and how the proxy gets configured instead.

1.4 Where it runs

The manager container (config/manager/manager.yaml) runs as non-root with a read-only root filesystem, a 64Mi emptyDir at /tmp and a 128Mi memory limit, from a distroless static image (Dockerfile) that carries the public CA roots and nothing else. So a provider:

• MUST stream large payloads rather than buffering them in memory;
• MUST write scratch files only under /tmp, and delete them;
• gets credentials either as mounted files (the providers ConfigMap at /etc/proxy-operator, optional Secret mounts) or from the environment or the pod's own identity. See §7.

1.5 Time and cancellation

No caller puts a deadline on the context. The reconcile and GC contexts are cancelled on shutdown and on loss of leadership, and otherwise never. The only bounded context is the health monitor's (Monitor.checkOnce, one timeout per round).

• MUST honour ctx: return promptly once it is cancelled, on every method.
• SHOULD bound every backend call itself: a dial timeout, a per-request timeout, a cap on any wait loop. What breaks otherwise: a hung Create, Get or Delete pins one of the 3 reconcile workers until the manager stops; a hung ListByTag stalls orphan GC for every provider, because Sweeper.sweep runs them sequentially; a Check that ignores its context wedges a monitor round. Choose budgets against the polls: a Get that routinely outlives ProvisioningPoll gains nothing.

2. The contract, method by method

2.1 Create(ctx, CreateRequest) (providerID, error)

• MUST be idempotent by req.Name. A repeat call for a name that already exists returns that instance's providerID with a nil error, creates no duplicate and returns no error. Caller: the reconciler records the ID in status only after Create returns (reconcileManaged). A crash between the two, or a lost status write, runs Create again for the same Proxy; replaceInstance recovers from a crash between its two writes the same way.
• MUST mark the instance as managed, with LabelManaged=LabelManagedYes and LabelUID=req.UID or the backend's equivalent, in the same call that creates it. When the backend's create call cannot carry the mark, the window between creation and marking MUST be far shorter than MinAge and documented in the provider's spec (gc-mark in §4). Caller: orphan GC finds instances only through the mark and never deletes one whose UID is empty (Sweeper.sweep). An instance that exists without the mark leaks if the operator crashes before marking it.
• MUST return a providerID that is self-contained. Get and Delete receive only the ID, and the config or spec may have changed since Create (a zone edit during a replacement). Put whatever Get needs to find the instance into the ID, and validate config values that end up in the ID so they cannot contain the separator. See primary-key in §4 for backends that allocate their own identifier.
• MUST NOT let Get answer NotFound for an ID that Create just returned. A backend with eventual consistency between create and read needs a read-after-write guarantee or a retry inside Create. Caller: on NotFound the reconciler clears the ID and creates again (reconcileManaged), so a lagging read makes a duplicate.
• MUST honour ctx (§1.5).
• MAY return once the request is submitted, or block until the instance is running (create-return in §4). Submit-and-return is preferred: readiness is discovered by polling Get, and a blocking Create holds one of the 3 workers for its whole duration. A provider that blocks MUST still honour cancellation and SHOULD cap the wait.
• SHOULD reject a request it can never satisfy with ErrPermanent before touching the backend: a missing or unknown zone, machine type or image, or any placement value not in its catalogue. The CRD cannot validate placement, because it is provider-agnostic, so this error is the user's only feedback. The message SHOULD name the offending field and value.
• Catalogue changes do not replace instances. The spec hash (specHashInput in internal/controller/spechash.go) covers placement, the resolved cloudInit and port, not the provider config. When a placement value names a config entry, changing what that entry means affects only new instances. The provider's guide MUST say so and tell users to add a new key rather than edit one in use.
• MAY ignore Placement fields that do not apply. If a field is ignored, the provider's guide MUST say so, because the field is still part of the spec hash: changing it replaces the instance and has no other effect.
• SHOULD deliver CloudInit as user-data, byte for byte, without parsing or re-rendering it. A provider that cannot documents that it ignores the field.
• MUST NOT log CloudInit contents, put them in span attributes, or include them in error messages. It can hold secrets. Log its byte count. Test this with a sentinel string (§11).

2.2 Get(ctx, providerID) (*Instance, error)

• MUST return ErrNotFound when, and only when, the instance is gone. Caller: ErrNotFound triggers three actions. In normal reconciling the reconciler clears the ID and creates again (reconcileManaged). During replacement it is the signal to create the new instance (replaceInstance). During deletion it releases the finalizer (reconcileDelete). A false NotFound during an outage creates a duplicate or loses the instance. "Unreachable", "wrong credentials" and "cannot tell" are Transient or Permanent, never NotFound.
• MUST keep returning the instance, not NotFound, while a delete is in progress and the name is still taken. Caller: the replacement instance gets the same name. replaceInstance waits for NotFound before creating it; a NotFound too early makes Create adopt the dying instance.
• MUST report StateRunning only with a non-empty IP. Running without an address yet is StateProvisioning. Caller: the IP from a Running answer goes straight into status and to the health engine (reconcileManaged).
• MUST report StateStopped or StateTerminated only when the instance will not recover by itself. SHOULD map states it does not recognise to StateStopped. Caller: the reconciler reacts to both states identically: it calls Delete and requeues after DeletionPoll without clearing the ID, then polls Get and deletes again until NotFound, then creates (reconcileManaged). Instances are disposable, so this is the safe answer for anything unknown, but it makes the next rule critical.
• MUST NOT report a freshly created instance that has not been started yet as StateStopped. A backend with no provisioning state of its own, where "defined but not yet started" and "stopped" look the same, MUST resolve this in one of two ways (provisioning-state in §4): block Create until the instance is started, or record enough at creation (written metadata, a task handle) for Get to answer StateProvisioning. Caller: the same branch of reconcileManaged would delete the instance right after it was created, forever.
• During an asynchronous operation or a backend lock (a clone or a delete still running, an instance locked by the backend), Get MUST return either the last known state or ErrTransient, never NotFound and never Permanent (async-ops in §4).
• MUST fill UID and CreatedAt by reading them back from the backend, never from memory. Caller: GC compares UID with the live Proxies and deletes anything whose CreatedAt is older than MinAge. A zero CreatedAt counts as very old (Sweeper.sweep), so a provider that forgets it makes instances still being created collectable. An unparsable timestamp giving the zero time is accepted: it never shields an orphan forever.
• SHOULD return ErrPermanent for an ID it can never resolve: a malformed ID, a location no longer in the config. Caller: reconcileDelete then tries Delete anyway and releases the finalizer if that works. If the provider returned Transient, deletion would poll forever.

2.3 Delete(ctx, providerID) error

• MUST be idempotent. Deleting something already gone returns nil. Caller: GC and the reconciler race, and a crash repeats the call.
• MUST accept being called again while an earlier delete is still in progress. Treat "already deleting" as success, and SHOULD keep that cheap. Caller: while Get still returns the instance, reconcileDelete and the Stopped branch of reconcileManaged call Delete again every DeletionPoll until Get returns NotFound.
• MUST remove everything Create made: disks, volumes, seeds, addresses, uploaded files. Either delete them explicitly or rely on the backend cascading (secondary-resources in §4). Caller: GC sees only what ListByTag returns, so secondary resources have no other cleanup path.
• MAY return before the instance is gone (delete-return in §4), as long as Get keeps showing it until it is.
• SHOULD succeed for an ID whose Get fails permanently when there is nothing left the provider could reach, and log that it did nothing. Caller: on the deletion path a permanent error is not latched; it retries forever (deletionFailure), and the Proxy is then only released through the AnnotationSkipCleanup annotation (api/v1alpha1/proxy_types.go).

2.4 ListByTag(ctx) ([]Instance, error)

• MUST return only instances that carry the managed mark, each with an ID that Delete accepts, plus UID and CreatedAt. State and IP are not used by GC. Caller: GC deletes every listed instance whose UID is not live and whose age is past MinAge (Sweeper.sweep). Listing something without the mark is how an operator deletes an instance it does not own.
• MUST return an error rather than a list it cannot vouch for. An error makes GC skip that provider for this sweep (Sweeper.sweep).
• MAY leave out instances it could not read. Leaving something out is always safe, because GC never deletes what is not listed. Pick a policy (list-partial in §4) and document it.
• SHOULD cover the whole scope the provider can create in: every namespace, zone, node or host. Anything outside that scope is never collected.
• Scope caveat: GC protects instances by UID, not by installation. Two operator installs sharing one backend delete each other's instances (§13). Two provider entries of one install that share a scope are safe, because they share the live-UID set.

3. What a container-style backend and a VM-style backend have in common

The contract above is the same for both, but the profiles in §4 differ. A backend that runs a container has an instance that is ready seconds after Create, gets its address from the platform, is deleted in one call and has no secondary resources; a backend that boots a VM has a provisioning window of minutes, may discover the address only after the guest reports it, may need several deletes, and leaves disks and seeds behind unless told otherwise. Neither is preferred. What matters is that the provider's spec names its profile on every axis, because the appendix table and the test list are keyed by those names.

4. Behaviour profiles

Each axis below has a stable code. The provider's spec (the template) answers every axis; the appendix maps each existing provider to its profile on every axis. A profile is not a permission to skip a MUST in §2; it says which of the allowed ways of meeting it the provider chose.

4.1 readiness

When is an instance usable after Create?

• immediate: the platform starts it at once; Get reports Running within a poll or two.
• provisioning-window: minutes of booting; Get reports Provisioning for that time.

Callers tolerate both: the reconciler polls at ProvisioningPoll either way. An immediate provider still MUST report Provisioning until the IP is known.

4.2 create-return

Does Create return once the request is submitted or once the instance is running?

• submit: preferred. Readiness comes from Get.
• blocking: allowed. Costs one of 3 workers for the duration, and any first-use setup (an image download) runs inside it. MUST honour ctx, SHOULD cap the wait, and MUST keep every step idempotent so a cut-off attempt resumes.

4.3 delete-return

Does Delete return before the instance is gone?

• returns-early: Get keeps returning the instance until it is gone (§2.2), and repeated Delete calls are cheap.
• waits: Delete returns nil only when nothing is left. Bound the wait and honour ctx.

4.4 ip-source

Where does Instance.IP come from?

• assigned-at-create: the backend reports it with the instance.
• discovered: an agent in the guest, a DHCP lease, a neighbour table. Discovery failures mean "no address yet", never an error; Get returns Provisioning. The provider's guide MUST say what the guest needs (an agent package, outbound access to install it) and what happens without it.

4.5 gc-mark

Where do LabelManaged and LabelUID live?

• native-labels: labels or tags the backend indexes; ListByTag can filter server-side.
• written-metadata: a description, an annotation or a metadata element written by the provider; ListByTag lists everything and filters client-side.
• mark-window: the create call cannot carry the mark and a second call writes it. Allowed only when the window is a few seconds, documented, and the unmarked instance is cleaned by the same Proxy's next reconcile (§2.1).

4.6 created-at

Where does CreatedAt come from?

• backend-timestamp: the backend's own creation time.
• written-at-create: the provider writes the time into the mark metadata in the create call. A backend with no timestamp MUST do this.

4.7 list-partial

What does ListByTag do when part of its scope is unreachable?

• fail-whole: one unreachable part fails the list; GC skips the provider this sweep. Nothing is collected until every part answers.
• omit-unreachable: unreachable parts are left out and the rest is returned. Orphans in the reachable parts are collected; the provider MUST log what it left out.

Both are safe (§2.4). A single unreadable instance MAY always be skipped.

4.8 quota

Can the provider tell "no capacity" from other failures?

• detected: a recognisable quota, rate-limit or out-of-resources signal maps to ErrQuotaExceeded.
• absent: capacity errors map to ErrTransient. The guide MUST say so, and the Checker SHOULD expose a capacity figure in Details instead.

4.9 not-found

How does the provider know an instance is gone?

• status-404: a distinct not-found status or error code from the backend.
• error-code: a code or message the backend uses for "does not exist", matched exactly and covered by the classify table test with real captured responses.
• list-and-miss: absence from a listing. Allowed only when the listing can never be partial; otherwise a missing part reads as "gone" and §2.2 is broken.

4.10 primary-key

Is req.Name the backend's key?

• name-is-key: the backend addresses instances by name. Create looks the name up, the ID is <location>/<name>.
• allocated-id: the backend allocates an identifier and the name is a label. Create MUST first find an existing instance by name (idempotency), then allocate. Allocation MUST be safe against a concurrent Create for another Proxy: a deterministic identifier derived from req.UID, or an allocate-then-verify loop. The ID SHOULD embed the allocated key and the name; Get and Delete use the allocated key. If instances can move between locations (migration), the ID MUST NOT depend on the location.

4.11 provisioning-state

Does the backend have a state for "being created"?

• native: the backend reports a provisioning or staging state, mapped to StateProvisioning.
• create-blocks: no such state; Create blocks until the instance is started, so Get never sees "defined but not started".
• marker: no such state and Create returns early; the provider records at creation (metadata or a task handle) what lets Get answer StateProvisioning for an instance that was never started, and StateStopped for one that stopped afterwards.

4.12 async-ops

Does the backend run create and delete as asynchronous tasks?

• none: calls complete synchronously.
• task-handle: the backend returns a task the provider can poll. The provider SHOULD keep the handle where Get can find it (metadata or the ID) so a failed create can be reported instead of read as NotFound.
• backend-lock: the backend locks the instance during a task. Get returns the last known state or Transient while locked (§2.2); Delete on a locked instance returns Transient, not Permanent.

4.13 secondary-resources

What else does Create make, and who removes it?

• none.
• cascade: the backend deletes them with the instance (auto-delete flags). The provider MUST set those flags at create.
• explicit: the provider deletes them by name in Delete, tolerating their absence. Resources that can exist without an instance (a crash between creating them and creating the instance) are a documented GC blind spot unless ListByTag covers them.

4.14 credentials

Where do credentials and trust material come from?

• mounted-files: paths in the config block, pointing into an optional Secret mount (§7).
• environment: standard SDK environment discovery.
• workload-identity: the pod's own identity.

TLS trust for an HTTPS backend with a private CA is one of system-roots, ca-file (a path in the config block) or insecure (a boolean whose name starts with insecure, warned about at startup).

4.15 client-kind

What talks to the backend?

• vendor-sdk: put it behind a small interface (the seam) so a fake can stand in.
• thin-client: a net/http client written in the provider package. Preferred when no maintained SDK exists (CLAUDE.md: stdlib over third-party). It MUST set transport timeouts, MUST carry the backend's status and body in one error type that classify reads, and SHOULD wrap its transport for child spans (§9).
• rpc-over-ssh or another non-HTTP transport: one pooled connection per target, a keepalive, and a documented answer to "what does a cancelled call do to the shared connection".

4.16 timeouts

How is each backend call bounded? (§1.5)

• per-call: every request has its own deadline, independent of ctx.
• ctx-only: the provider relies on ctx. Allowed only for backends whose client already enforces a timeout; the spec MUST cite it.

4.17 logging

Which log tiers does the provider produce? The logger MUST come from logf.FromContext(ctx) at call time (§9).

• tiered: V(1) one line per operation outcome and per failed backend call; V(2) request and response detail; V(5) raw wire if there is one.
• outcome-only: V(1) only.
• none: allowed only for a backend whose client logs on its own.

4.18 child-spans

How do backend calls appear under the operation span? (§9)

• sdk-instrumentation: the SDK emits spans itself.
• transport-wrapper: a parent-gated otelhttp transport.
• span-decorator: a decorator over the seam that starts one span per call.
• none: a documented gap, with log correlation through traceID only.

4.19 backend-test

How is the provider proven against the real backend?

• cluster-e2e: the kind end-to-end suite.
• integration-tag: a test behind //go:build integration, skipped without its environment variables, with a make test-integration-<type> target and a section in docs/testing.md.
• manual-runbook: a documented by-hand run. Allowed only when the backend cannot be reached from CI or a developer machine.

4.20 Summary

Axis	Profiles	Who tolerates the difference	Template entry
readiness	immediate, provisioning-window	reconciler polls either way	7
create-return	submit, blocking	reconciler; worker budget	10
delete-return	returns-early, waits	reconciler re-calls Delete	11
ip-source	assigned-at-create, discovered	reconciler gates Running on IP	7
gc-mark	native-labels, written-metadata, mark-window	GC	8
created-at	backend-timestamp, written-at-create	GC MinAge	8
list-partial	fail-whole, omit-unreachable	GC	12
quota	detected, absent	reconciler QuotaRetry	13
not-found	status-404, error-code, list-and-miss	reconciler	13
primary-key	name-is-key, allocated-id	reconciler idempotency	2
provisioning-state	native, create-blocks, marker	reconciler Stopped branch	9
async-ops	none, task-handle, backend-lock	reconciler	10
secondary-resources	none, cascade, explicit	GC blind spots	11
credentials	mounted-files, environment, workload-identity	deployment	14
client-kind	vendor-sdk, thin-client, rpc-over-ssh	tests, tracing	17
timeouts	per-call, ctx-only	worker and GC budget	18
logging	tiered, outcome-only, none	operators	20
child-spans	sdk-instrumentation, transport-wrapper, span-decorator, none	tracing	17
backend-test	cluster-e2e, integration-tag, manual-runbook	CI	21

5. Error taxonomy

Every error a provider method returns MUST be built with provider.Wrap(class, op, providerName, id, err) (internal/provider/errors.go). op is one of create, get, delete or list, and providerName is the configured name (ProviderConfig.Name), not the type. Caller: the decorators label metrics and spans with the class (resultLabel), and the reconciler branches on provider.Class.

Wrap keeps both errors.Is(err, class) and errors.As to the backend's error type working (Error.Unwrap returns both).

Class	Use it for	During normal reconciling	While deleting
ErrNotFound	the instance does not exist	Get: recreate or advance (§2.2); other ops: returned as the error, so exponential backoff	Get: release the finalizer
ErrQuotaExceeded	quota, rate limit, out of capacity	Provisioned=False with reason QuotaExceeded, requeue after QuotaRetry, not counted as an error (providerFailure)	requeue after QuotaRetry (deletionFailure)
ErrTransient	network, 5xx, timeouts, cancellation, anything unknown	error returned, so workqueue exponential backoff	same
ErrPermanent	bad config, bad argument, auth or permission denied, malformed ID	latched: Provisioned=False with reason PermanentError, no more provider calls until metadata.generation changes (reconcileManaged)	error returned and retried forever; a Permanent Get also triggers Delete (reconcileDelete)

Pitfalls:

• Unclassified means Transient. provider.Class returns ErrTransient for any error not built with Wrap. That is the safe default, but a permanent failure that was never classified retries forever, and a quota error backs off far too fast.
• Permanent is sticky. Use it only for errors that editing the Proxy spec or fixing the config can resolve. A permission-denied status that the backend also uses for quota is the hard case: read the backend's reason if it gives one, and if it does not, pick Permanent and document it.
• Do not return NotFound from Create. The reconciler treats it as Transient there, but it misleads the metrics and anyone reading logs.
• NotFound detection follows the not-found profile (§4.9). A backend that reports "does not exist" as a generic server error needs the exact message or code matched, with real captured responses in the test table.
• Classify in one classify function with a table-driven test covering every class, including "not a backend error at all, so Transient".

6. Identity, names, marks, CreatedAt

• Names: NameFromUID(uid) gives proxy- plus 16 lowercase base32 characters. It is a valid DNS label. A backend with stricter rules SHOULD map it deterministically rather than invent a new scheme, because idempotency depends on the name being the same every time. A backend that keys on an allocated identifier follows primary-key (§4.10).
• providerID: opaque to everyone except the provider. Keep it <location>/<name>-shaped or <location>/<allocated-key>-shaped, parse it strictly, and return ErrPermanent for a malformed one (§2.2).
• Marks: LabelManaged with value LabelManagedYes, and LabelUID. Use native labels or tags where the backend has them; otherwise write them into metadata in the create call (§4.5).
• CreatedAt: the backend's creation timestamp, or written into the mark metadata at creation (§4.6).
• Provider names: a configured name MUST be a lowercase DNS-1123 label of at most 63 characters (spec.provider in api/v1alpha1/proxy_types.go). The ProviderHealth object is named after the provider; Monitor.Start warns and Monitor.publish skips the object for a name that is not a valid object name. Config loading does not check this yet (§13).

7. Credentials, TLS and secret hygiene

• The constructor MUST NOT dial the backend. registry.Build aborts the whole provider set on a constructor error, and the manager exits: one unreachable backend must not take the other providers down. Fail in the constructor only on config errors.
• Credential files SHOULD be read lazily. A mount marked optional: true is empty when the Secret is missing; the constructor logs a warning and the first call fails Transient until the Secret lands. A file that exists but does not parse is Permanent.
• Credentials MUST NOT appear in logs, span attributes, error strings or Error.Err. Backend errors that echo a request URL or header with a token in it MUST be sanitised before wrapping. The same holds for CloudInit (§2.1). Test both with a sentinel (§11).
• TLS trust follows credentials (§4.14). An insecure toggle MUST be named so that it reads as insecure, default off, and MUST be logged as a warning at startup when on.
• Rotation. State in the spec whether rotating the Secret takes effect without a restart. Reading files on every dial makes rotation free; caching them after first load needs a restart and must say so.
• Scratch files only under /tmp (§1.4), removed when the call ends.

8. Optional interfaces: Checker

provider.Checker is Check(ctx) []TargetStatus (internal/provider/check.go). cmd/main.go type-asserts it on the unwrapped provider before adding the decorators, and the monitor treats a nil Checker as unsupported, not failed (Monitor.run).

• MUST never fail as a whole. Every failure is a TargetStatus with Ready=false. Caller: Monitor.checkOnce has no error path per provider.
• MUST return promptly once ctx ends. The monitor gives the whole round one timeout.
• MUST return one entry per target, in a stable order. Targets are what can fail independently: an apiserver, a project, each host or node. Caller: the monitor logs transitions per target (logTransitions) and the gauges are labelled {provider,target}.
• MUST NOT disrupt reconciles. A check that shares a connection with provisioning must not tear that connection down when it times out.
• SHOULD use the shared reasons (ReasonReachable, ReasonTimeout, ReasonUnreachable, ReasonUnauthorized, ReasonForbidden, ReasonAPIError). Add provider-specific reasons only for failures an admin would fix differently, and document each one in the guide.
• SHOULD check with the same credentials, verb and scope that ListByTag and Create use, and be cheap: one page, one object.
• MAY fill Details with facts useful when triaging: versions, capacity, the account in use.
• The status.type field of ProviderHealth (api/v1alpha1/providerhealth_types.go) lists the known types in its doc comment; add the new type there and run make manifests.

9. Observability

Tracing is OpenTelemetry, set up in internal/tracing from the standard OTEL_* environment variables. With no exporter configured, spans are no-ops and logs are unchanged (architecture.md §10).

Layer 1, operation spans and metrics: nothing to write. cmd/main.go wraps every provider in WithTracing(WithMetrics(...)):

• WithTracing opens one client span per call, named provider.create, provider.get, provider.delete or provider.list, with provider.name, provider.id and provider.result (the error class) as attributes. ErrNotFound keeps the span status Ok, because polling to NotFound is expected.
• WithMetrics counts each call as proxy_operator_provider_requests_total{provider,op,result} (internal/metrics/metrics.go). A Checker feeds proxy_operator_provider_target_up, proxy_operator_provider_check_duration_seconds and proxy_operator_provider_check_timestamp_seconds, labelled {provider,target}.

Rules:

• MUST NOT open its own per-operation span or count its own requests. They would duplicate the decorators.
• MUST pass the caller's ctx to every backend call and take the logger from logf.FromContext(ctx) at call time. tracing.StartSpan (internal/tracing/logger.go) puts a logger carrying traceID and spanID into ctx. A logger cached at construction, or a backend call made with context.Background(), loses the trace.
• Health checks bypass the decorators. A Checker runs under the monitor's providerhealth.check span, and the same ctx rules apply.

Layer 2, child spans per backend call: SHOULD. One operation span says that Create took 40 seconds; child spans say which backend call took them. Follow child-spans (§4.18):

• HTTP backends: wrap the transport with otelhttp, gated on an existing parent span as tracing.RestConfigWrapper does (internal/tracing/transport.go), so background traffic (token refreshes, keepalives) does not become root spans. Check first whether the SDK already does this.
• Non-HTTP backends: start child spans yourself with tracing.Start(ctx, "<type>.<call>") around each call and around long steps such as a dial or a download, in a decorator over the seam, not at call sites.
• MUST NOT put secrets or CloudInit in span attributes. Put identifiers, sizes and error codes there.

Layer 3, logs. Follow logging (§4.17). Log lines join a trace through the traceID key from layer 1.

10. Configuration and wiring

The providers file is parsed by LoadConfig (internal/provider/config.go) with UnmarshalStrict, so unknown keys already fail.

• Add a <Type>Config struct and a ProviderConfig field for it.
• Add the block to setBlocks, so a block that does not match Type is rejected.
• Add the type to the allow-list in Config.validate. The same list exists a second time as the constructor map in cmd/main.go; both must change (§13).
• Add required-field validation in its own config_<type>.go when it is more than a few lines, with rows in config_test.go.
• Validate every config value that is pasted into an identifier: a location name that becomes part of the providerID must not contain the separator; a catalogue key that becomes a file or volume name must be a safe filename.
• The constructor has the signature registry.Constructor: func(ctx, provider.ProviderConfig) (provider.Provider, error). Register it in the map in cmd/main.go. Flags a provider needs are defined there too and passed through a NewWith... variant, never read from the environment inside the package.
• Decorators are applied by the composition root, in the order WithTracing(WithMetrics(p)), after the Checker has been collected from the unwrapped provider.

11. Testing obligations

CLAUDE.md applies: stdlib testing, table-driven, tests next to source, t.Parallel() where safe.

• A fake behind the seam. Every unit test runs against an in-memory fake of the backend, never the network.
• Contract tests, each against the fake:
  • repeat Create returns the same ID and creates nothing;
  • Get after Delete returns ErrNotFound;
  • Delete twice succeeds;
  • malformed ID is ErrPermanent from Get and Delete;
  • an unknown backend state maps to StateStopped;
  • StateRunning requires an IP, and a running instance without one is StateProvisioning;
  • UID and CreatedAt are read back from the fake, not remembered;
  • ListByTag excludes unmarked instances and instances with an empty UID are still listed with UID == "";
  • Get during an in-progress delete still returns the instance;
  • Get during an asynchronous operation or lock returns the last state or Transient (async-ops);
  • a created but not yet started instance is never StateStopped (provisioning-state);
  • a cancelled ctx makes every method return promptly.
• Classify: one table test covering every class, every backend code the provider matches, and "not a backend error at all, so Transient". Use captured real responses for message matching.
• Checker: a table test for the reasons, a test that Check returns promptly with an expired ctx, and a test that it returns one entry per target in config order.
• Sentinel tests: a CloudInit containing a sentinel string and a credential containing another never appear in any log line, span attribute or error string.
• Tracing: with an in-memory span recorder (sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(tracetest.NewSpanRecorder())), as in internal/provider/tracing_test.go), the child spans are children of the operation span.
• Real-backend test per backend-test (§4.19). It runs the whole lifecycle: create, create again, poll until Running, list, delete, delete again, confirm it is gone and that no secondary resource remains. Keep it compiling with go vet -tags=integration ./....

12. Checklist for adding a provider

1. Spec first. Copy docs/provider-spec-template.md to docs/<type>-provider-spec.md and answer every entry. Unanswered entries block implementation.
2. Package internal/provider/<type>/ with the provider type, a compile-time assertion for provider.Provider and one for provider.Checker, the seam interface, and a logger built from logf.FromContext(ctx).WithName("<type>").WithValues("provider", name).
3. Config per §10, including config_test.go rows.
4. Constructor New(ctx, provider.ProviderConfig) registered in cmd/main.go; no dialing.
5. Contract per §2, tested per §11.
6. Errors per §5, classify plus its table test, ProviderConfig.Name in every Wrap.
7. Checker per §8, with the status.type comment updated and make manifests run.
8. Tracing and logging per §9 and §4.17, §4.18.
9. Real-backend test per §4.19, with a make test-integration-<type> target and a section in docs/testing.md.
10. RBAC: if the provider talks to the cluster's own API, add +kubebuilder:rbac markers in their own comment block, never inside a doc comment, then make manifests and check git diff config/rbac/role.yaml.
11. Deployment: an optional: true credentials mount in config/manager/manager.yaml; a commented entry in config/manager/providers_config.yaml and config/samples/providers-config.yaml; a config/samples/proxy_<type>.yaml whose cloud-init installs a proxy on the sample image; a Makefile helper for the Secret if there is one; hack/providers-dev.yaml only if the type needs no credentials.
12. Docs: docs/<type>-provider.md (prerequisites, config reference, placement mapping, what lands on the backend, the health check, logging, known gaps); a row and a section in providers-existing.md; a row in the architecture.md components table and a block in its §5 and §11 (§10 too if the provider adds spans); a row in the README "Implemented providers" table; a CHANGELOG.md entry once the user has confirmed it works.
13. Optional: a demo script under docs/demo/.

13. Known limits of the contract

These are facts a new provider has to live with today. None is designed or scheduled.

• Adoption is not verified. Create adopts any existing instance with the right name without checking its LabelUID.
• StateStopped and StateTerminated are indistinguishable to the reconciler; both mean delete and recreate.
• Config is read once per process. Adding a provider or a location, or rotating credentials a provider caches, needs a manager restart.
• GC has no installation identity. Two installs sharing one backend delete each other's instances, because GC only knows its own live UIDs.
• An unconfigured provider latches the Proxy. A Proxy naming a provider missing from the config is latched PermanentError; adding the provider and restarting does not change metadata.generation, and spec.provider is immutable, so the Proxy stays latched until it is recreated.
• Provider names are not validated as DNS-1123 at config load.
• The type list exists twice: Config.validate and the constructor map in cmd/main.go.
• Delete is resubmitted on every deletion poll until Get says NotFound.
• Callers set no deadline (§1.5); each provider bounds itself.
• There is no conformance suite. The §11 contract tests are written per provider; internal/provider/providertest is a proposal in the appendix.
• Stale comments in internal/provider/provider.go, internal/provider/registry/registry.go and api/v1alpha1/proxy_types.go still describe a mock provider that no longer exists.

14. Related documents

• providers.md: the index.
• provider-spec-template.md: the questionnaire.
• providers-existing.md: how the existing providers meet this contract, and their gaps.
• architecture.md §5, §10 and §11.
• testing.md.