egress-proxies-operator/docs/plans-executions/2026-09-16-0841-provider-blueprint.md

# Execution: provider blueprint
Plan: [docs/plans/2026-09-16-0841-provider-blueprint.md](../plans/2026-09-16-0841-provider-blueprint.md)

- [x] Step 0 — Save the plan; wait for the merge
- [x] Step 1 — Collect evidence (read-only, in parallel)
- [x] Step 2 — Write `docs/providers.md`
- [x] Step 3 — Verify, then open the MR

## Step 0 — Save the plan; wait for the merge

The plan was committed on `feat/provider-health` (`80b2e9c`) and reached `main`
through PR #6. After the user confirmed the merge, the branch was cut from the
updated `main`:

```bash
git switch main && git pull --ff-only
git switch -c feat/provider-blueprint
```

Worth noting: `.claude/settings.json` has local, uncommitted permission edits
that carried over through the switch. They are deliberately left out of this
branch.

## Step 1 — Collect evidence (read-only, in parallel)

Three Explore agents answered the plan's 12-item questionnaire, one each for
kubernetes, gcp and kvm. Meanwhile I read the shared layer:
`internal/provider/*.go`, `registry.go`, the provider wiring in `cmd/main.go`,
`reconcileManaged`/`replaceInstance`/`reconcileDelete` in the controller,
`internal/gc/gc.go`, and how the monitor uses `Checker`.

Before relying on the agents' claims, I spot-checked them against the code. One
was wrong and did not go into the doc: the kvm agent said `Delete` skips volume
removal when `DomainDestroy` returns `ErrNoDomain`. In fact `removeDomain`
returns nil there, and `Delete` still calls `removeVolumes` afterwards
(`kvm.go:477-482`).

Rules that exist only in the callers, and not in any comment, were found by
reading the controller and GC. They went into the doc as MUST rules:
- A zero `CreatedAt` makes an instance immediately collectable (`gc.go:120`).
- `Get` returning `ErrPermanent` on the deletion path makes the reconciler call
  `Delete` anyway.
- Replacement waits for `Get` to return NotFound before creating an instance
  with the same name.
- `Stopped` and `Terminated` both lead to delete-and-recreate.

Worth noting, found outside this plan's scope and not changed here:
- `docs/gcp-vm-validation.md` filters on `labels.proxy-operator-managed=yes`,
  but the value is `true`.
- According to the gcp agent, `docs/gcp-in-specific-project.md` holds a real
  project ID, IAM output and an egress IP. The user was told.

## Step 2 — Write `docs/providers.md`

Wrote the eight sections the plan lists. Linked the new doc from the Provider
contract row in `docs/architecture.md` and from the CLAUDE.md "Architecture"
section.

Judgment calls:
- The file:line evidence sits in per-provider notes under the matrix, as
  planned. The contract rules in §2 cite the reconciler and GC lines each rule
  protects.
- The gap list includes documentation drift (kvm-provider.md's ARP order and
  log keys, gcp-vm-validation.md's label value and its claim that async errors
  reach status), as well as code gaps. Both would mislead the next
  provider author in the same way.
- "Proposed features" stays a flat list with no design, as the user decided.

Worth noting: the line numbers in the doc are a snapshot. The doc says so and
asks readers to look for the named function when a line has moved.

## Step 3 — Verify, then open the MR

An Explore agent re-checked the doc against the code: about 150 claims,
including every matrix cell. It found four mismatches, and all four were fixed:
- **Startup dial:** I had listed kvm-provider.md's "connection kept open from
  startup" as doc drift. It isn't: the pool dials lazily, but the provider
  health monitor runs a check as soon as the leader starts
  (`monitor.go:147`, default interval 1m), and that check opens the connection.
  The bullet was removed and the kvm notes now say this.
- **Checker wording:** §5 said the monitor type-asserts `Checker`. It is
  `cmd/main.go:216`; the monitor only sees nil or not.
- **Reasons, not conditions:** §3 called `QuotaExceeded` and `PermanentError`
  conditions. They are reasons on `Provisioned`.
- **Latch lines:** the line range is `:133-139`, not `:130-135`.

It also noted that none of the providers assert `var _ provider.Provider`, and
that the kubernetes provider's RBAC markers live in the controller package. The
§7 checklist now says both.

Link check (every relative `](…)` target and anchor in the new and edited docs
resolves), then the sanity build:

```bash
go build ./... && go test -short ./...
# all packages ok
```

Worth noting: no CHANGELOG entry yet. Per CLAUDE.md it goes in after the user
confirms the doc.

### Follow-up — tracing was missing from the blueprint

The user pointed out, after the PR was opened, that the blueprint said almost
nothing about tracing. The plan's questionnaire item 10 had collected the
evidence, but the doc reduced it to a matrix row about logging and a checklist
line telling authors not to open spans. That line even suggests tracing needs
no work. A provider author needs the opposite message: the operation span comes
free, and child spans for backend calls are theirs to add.

Added a new §6, "Observability: tracing, logs and metrics", and renumbered the
matrix, checklist and gaps sections to §7–§9. It covers:
- **Three layers:** decorator operation spans; child spans per backend call,
  with parent-gated HTTP transports or a span decorator over the SDK seam; and
  logs.
- **The ctx and logger rules** that keep `traceID` on log lines.
- **A per-provider status table:**
  - kubernetes: parent-gated `RestConfigWrapper` spans.
  - gcp: the Google SDK's own `otelhttp`, never disabled by the provider.
  - kvm: no spans. `trace.go` is a log decorator.

Also added two tracing rows to the §7 matrix, rewrote checklist item 8 as
"Tracing and logging" with an in-memory span-recorder test, and added gaps (kvm
has no child spans; only kubernetes is checked against Tempo, and only at
operation level). A proposal notes that the kvm `libvirtAPI` seam takes no
`ctx`, so per-RPC spans need that plumbed in.

The links were re-checked, including the new `architecture.md#10-tracing-internaltracing` anchor.

### Follow-up — a completeness sweep

The user asked for one more pass over everything provider-related. After the
tracing miss, this pass read beyond the provider packages:
- the Proxy API types and the spec hash;
- the health engine and discovery call sites that consume `Instance.IP`;
- the manager Deployment, the providers ConfigMap and the samples;
- README, testing.md, kvm-provider.md and the GCP docs;
- the initial prompt, the proxy-operator and kvm plans, and the 2026-08-10 PR
  review;
- the CHANGELOG.

What the blueprint was missing, now added:
- **What the instance is for.** The IP and port must be reachable from the
  operator's health probes and from discovery clients. Providers don't know
  which proxy runs; the proxy comes from cloud-init, except for kubernetes.
- **Concurrency.** Three reconciles, GC and checks can call a provider at once,
  so it MUST be safe for concurrent use.
- **The runtime envelope:** read-only root filesystem, a 64Mi `/tmp`, a 128Mi
  memory limit, credentials as optional mounts.
- **Create:** `placement` can't be validated by the CRD, so error messages must
  name the field. Catalogue edits in config aren't in the spec hash.
- **Delete** is re-called every `DeletionPoll` while a delete is in progress.
- **Provider names** must be DNS-1123 for `ProviderHealth`, and `spec.provider`
  is immutable.
- **Metric names** in §6.
- **Three matrix rows:** proxy software, IP reachability, distinct egress.
- **A deployment checklist item:** credentials mount, config entries, sample
  Proxy, Makefile helper.
- **Gaps:**
  - still open from the PR review: GC has no installation identity; an
    unconfigured provider latches Proxies; kubernetes Pod RBAC always installed
    and no `OwnerReference` on Pods;
  - the duplicated type list;
  - samples that can't work: `proxy_gcp.yaml` uses a debian-cloud image without
    cloud-init, and `proxy_kvm.yaml` places on `nas1` where configs define
    `beelink`;
  - the `"mock"` example in the `spec.provider` doc.

Outside the plan's "leave other docs alone" line, README.md changed:
- Its "Adding a provider" section restated the contract, with the old "Create
  never blocks" rule that kvm contradicts. It now points to `docs/providers.md`
  instead of keeping a second copy that disagrees.
- The "Implemented providers" table gained the missing kvm row.
- The deploy note no longer claims the default ConfigMap ships only the
  kubernetes provider.

The links were re-checked for providers.md and README.md.