egress-proxies-operator/docs/plans-executions/2026-09-16-0841-provider-blueprint.md

Execution: provider blueprint

Plan: docs/plans/2026-09-16-0841-provider-blueprint.md

• Step 0 — Save the plan; wait for the merge
• Step 1 — Collect evidence (read-only, in parallel)
• Step 2 — Write docs/providers.md
• Step 3 — Verify, then open the MR

Step 0 — Save the plan; wait for the merge

The plan was committed on feat/provider-health (80b2e9c) and reached main through PR #6. After the user confirmed the merge, the branch was cut from the updated main:

git switch main && git pull --ff-only
git switch -c feat/provider-blueprint

Worth noting: .claude/settings.json has local, uncommitted permission edits that carried over through the switch. They are deliberately left out of this branch.

Step 1 — Collect evidence (read-only, in parallel)

Three Explore agents answered the plan's 12-item questionnaire, one each for kubernetes, gcp and kvm. Meanwhile I read the shared layer: internal/provider/*.go, registry.go, the provider wiring in cmd/main.go, reconcileManaged/replaceInstance/reconcileDelete in the controller, internal/gc/gc.go, and how the monitor uses Checker.

Before relying on the agents' claims, I spot-checked them against the code. One was wrong and did not go into the doc: the kvm agent said Delete skips volume removal when DomainDestroy returns ErrNoDomain. In fact removeDomain returns nil there, and Delete still calls removeVolumes afterwards (kvm.go:477-482).

Rules that exist only in the callers, and not in any comment, were found by reading the controller and GC. They went into the doc as MUST rules:

• A zero CreatedAt makes an instance immediately collectable (gc.go:120).
• Get returning ErrPermanent on the deletion path makes the reconciler call Delete anyway.
• Replacement waits for Get to return NotFound before creating an instance with the same name.
• Stopped and Terminated both lead to delete-and-recreate.

Worth noting, found outside this plan's scope and not changed here:

• docs/gcp-vm-validation.md filters on labels.proxy-operator-managed=yes, but the value is true.
• According to the gcp agent, docs/gcp-in-specific-project.md holds a real project ID, IAM output and an egress IP. The user was told.

Step 2 — Write docs/providers.md

Wrote the eight sections the plan lists. Linked the new doc from the Provider contract row in docs/architecture.md and from the CLAUDE.md "Architecture" section.

Judgment calls:

• The file:line evidence sits in per-provider notes under the matrix, as planned. The contract rules in §2 cite the reconciler and GC lines each rule protects.
• The gap list includes documentation drift (kvm-provider.md's ARP order and log keys, gcp-vm-validation.md's label value and its claim that async errors reach status), as well as code gaps. Both would mislead the next provider author in the same way.
• "Proposed features" stays a flat list with no design, as the user decided.

Worth noting: the line numbers in the doc are a snapshot. The doc says so and asks readers to look for the named function when a line has moved.

Step 3 — Verify, then open the MR

An Explore agent re-checked the doc against the code: about 150 claims, including every matrix cell. It found four mismatches, and all four were fixed:

• Startup dial: I had listed kvm-provider.md's "connection kept open from startup" as doc drift. It isn't: the pool dials lazily, but the provider health monitor runs a check as soon as the leader starts (monitor.go:147, default interval 1m), and that check opens the connection. The bullet was removed and the kvm notes now say this.
• Checker wording: §5 said the monitor type-asserts Checker. It is cmd/main.go:216; the monitor only sees nil or not.
• Reasons, not conditions: §3 called QuotaExceeded and PermanentError conditions. They are reasons on Provisioned.
• Latch lines: the line range is :133-139, not :130-135.

It also noted that none of the providers assert var _ provider.Provider, and that the kubernetes provider's RBAC markers live in the controller package. The §7 checklist now says both.

Link check (every relative ](…) target and anchor in the new and edited docs resolves), then the sanity build:

go build ./... && go test -short ./...
# all packages ok

Worth noting: no CHANGELOG entry yet. Per CLAUDE.md it goes in after the user confirms the doc.

Follow-up — tracing was missing from the blueprint

The user pointed out, after the PR was opened, that the blueprint said almost nothing about tracing. The plan's questionnaire item 10 had collected the evidence, but the doc reduced it to a matrix row about logging and a checklist line telling authors not to open spans. That line even suggests tracing needs no work. A provider author needs the opposite message: the operation span comes free, and child spans for backend calls are theirs to add.

Added a new §6, "Observability: tracing, logs and metrics", and renumbered the matrix, checklist and gaps sections to §7–§9. It covers:

• Three layers: decorator operation spans; child spans per backend call, with parent-gated HTTP transports or a span decorator over the SDK seam; and logs.
• The ctx and logger rules that keep traceID on log lines.
• A per-provider status table:
  • kubernetes: parent-gated RestConfigWrapper spans.
  • gcp: the Google SDK's own otelhttp, never disabled by the provider.
  • kvm: no spans. trace.go is a log decorator.

Also added two tracing rows to the §7 matrix, rewrote checklist item 8 as "Tracing and logging" with an in-memory span-recorder test, and added gaps (kvm has no child spans; only kubernetes is checked against Tempo, and only at operation level). A proposal notes that the kvm libvirtAPI seam takes no ctx, so per-RPC spans need that plumbed in.

The links were re-checked, including the new architecture.md#10-tracing-internaltracing anchor.

Follow-up — a completeness sweep

The user asked for one more pass over everything provider-related. After the tracing miss, this pass read beyond the provider packages:

• the Proxy API types and the spec hash;
• the health engine and discovery call sites that consume Instance.IP;
• the manager Deployment, the providers ConfigMap and the samples;
• README, testing.md, kvm-provider.md and the GCP docs;
• the initial prompt, the proxy-operator and kvm plans, and the 2026-08-10 PR review;
• the CHANGELOG.

What the blueprint was missing, now added:

• What the instance is for. The IP and port must be reachable from the operator's health probes and from discovery clients. Providers don't know which proxy runs; the proxy comes from cloud-init, except for kubernetes.
• Concurrency. Three reconciles, GC and checks can call a provider at once, so it MUST be safe for concurrent use.
• The runtime envelope: read-only root filesystem, a 64Mi /tmp, a 128Mi memory limit, credentials as optional mounts.
• Create: placement can't be validated by the CRD, so error messages must name the field. Catalogue edits in config aren't in the spec hash.
• Delete is re-called every DeletionPoll while a delete is in progress.
• Provider names must be DNS-1123 for ProviderHealth, and spec.provider is immutable.
• Metric names in §6.
• Three matrix rows: proxy software, IP reachability, distinct egress.
• A deployment checklist item: credentials mount, config entries, sample Proxy, Makefile helper.
• Gaps:
  • still open from the PR review: GC has no installation identity; an unconfigured provider latches Proxies; kubernetes Pod RBAC always installed and no OwnerReference on Pods;
  • the duplicated type list;
  • samples that can't work: proxy_gcp.yaml uses a debian-cloud image without cloud-init, and proxy_kvm.yaml places on nas1 where configs define beelink;
  • the "mock" example in the spec.provider doc.

Outside the plan's "leave other docs alone" line, README.md changed:

• Its "Adding a provider" section restated the contract, with the old "Create never blocks" rule that kvm contradicts. It now points to docs/providers.md instead of keeping a second copy that disagrees.
• The "Implemented providers" table gained the missing kvm row.
• The deploy note no longer claims the default ConfigMap ships only the kubernetes provider.

The links were re-checked for providers.md and README.md.