egress-proxies-operator/docs/plans-executions/2026-09-16-0841-provider-blueprint.md

# Execution: provider blueprint
Plan: [docs/plans/2026-09-16-0841-provider-blueprint.md](../plans/2026-09-16-0841-provider-blueprint.md)

- [x] Step 0 — Save the plan; wait for the merge
- [x] Step 1 — Collect evidence (read-only, in parallel)
- [x] Step 2 — Write `docs/providers.md`
- [x] Step 3 — Verify, then open the MR

## Step 0 — Save the plan; wait for the merge

The plan was committed on `feat/provider-health` (`80b2e9c`) and reached `main`
through PR #6. After the user confirmed the merge, the branch was cut from the
updated `main`:

```bash
git switch main && git pull --ff-only
git switch -c feat/provider-blueprint
```

Worth noting: `.claude/settings.json` has local, uncommitted permission edits
that carried over through the switch. They are deliberately left out of this
branch.

## Step 1 — Collect evidence (read-only, in parallel)

Three Explore agents answered the plan's 12-item questionnaire, one each for
kubernetes, gcp and kvm. Meanwhile I read the shared layer:
`internal/provider/*.go`, `registry.go`, the provider wiring in `cmd/main.go`,
`reconcileManaged`/`replaceInstance`/`reconcileDelete` in the controller,
`internal/gc/gc.go`, and how the monitor uses `Checker`.

Before relying on the agents' claims, I spot-checked them against the code. One
was wrong and did not go into the doc: the kvm agent said `Delete` skips volume
removal when `DomainDestroy` returns `ErrNoDomain`. In fact `removeDomain`
returns nil there, and `Delete` still calls `removeVolumes` afterwards
(`kvm.go:477-482`).

Rules that exist only in the callers, and not in any comment, were found by
reading the controller and GC. They went into the doc as MUST rules:
- A zero `CreatedAt` makes an instance immediately collectable (`gc.go:120`).
- `Get` returning `ErrPermanent` on the deletion path makes the reconciler call
  `Delete` anyway.
- Replacement waits for `Get` to return NotFound before creating an instance
  with the same name.
- `Stopped` and `Terminated` both lead to delete-and-recreate.

Worth noting, found outside this plan's scope and not changed here:
- `docs/gcp-vm-validation.md` filters on `labels.proxy-operator-managed=yes`,
  but the value is `true`.
- According to the gcp agent, `docs/gcp-in-specific-project.md` holds a real
  project ID, IAM output and an egress IP. The user was told.

## Step 2 — Write `docs/providers.md`

Wrote the eight sections the plan lists. Linked the new doc from the Provider
contract row in `docs/architecture.md` and from the CLAUDE.md "Architecture"
section.

Judgment calls:
- The file:line evidence sits in per-provider notes under the matrix, as
  planned. The contract rules in §2 cite the reconciler and GC lines each rule
  protects.
- The gap list includes documentation drift (kvm-provider.md's ARP order and
  log keys, gcp-vm-validation.md's label value and its claim that async errors
  reach status), as well as code gaps. Both would mislead the next
  provider author in the same way.
- "Proposed features" stays a flat list with no design, as the user decided.

Worth noting: the line numbers in the doc are a snapshot. The doc says so and
asks readers to look for the named function when a line has moved.

## Step 3 — Verify, then open the MR

An Explore agent re-checked the doc against the code: about 150 claims,
including every matrix cell. It found four mismatches, and all four were fixed:
- **Startup dial:** I had listed kvm-provider.md's "connection kept open from
  startup" as doc drift. It isn't: the pool dials lazily, but the provider
  health monitor runs a check as soon as the leader starts
  (`monitor.go:147`, default interval 1m), and that check opens the connection.
  The bullet was removed and the kvm notes now say this.
- **Checker wording:** §5 said the monitor type-asserts `Checker`. It is
  `cmd/main.go:216`; the monitor only sees nil or not.
- **Reasons, not conditions:** §3 called `QuotaExceeded` and `PermanentError`
  conditions. They are reasons on `Provisioned`.
- **Latch lines:** the line range is `:133-139`, not `:130-135`.

It also noted that none of the providers assert `var _ provider.Provider`, and
that the kubernetes provider's RBAC markers live in the controller package. The
§7 checklist now says both.

Link check (every relative `](…)` target and anchor in the new and edited docs
resolves), then the sanity build:

```bash
go build ./... && go test -short ./...
# all packages ok
```

Worth noting: no CHANGELOG entry yet. Per CLAUDE.md it goes in after the user
confirms the doc.

### Follow-up — tracing was missing from the blueprint

The user pointed out, after the PR was opened, that the blueprint said almost
nothing about tracing. The plan's questionnaire item 10 had collected the
evidence, but the doc reduced it to a matrix row about logging and a checklist
line telling authors not to open spans. That line even suggests tracing needs
no work. A provider author needs the opposite message: the operation span comes
free, and child spans for backend calls are theirs to add.

Added a new §6, "Observability: tracing, logs and metrics", and renumbered the
matrix, checklist and gaps sections to §7–§9. It covers:
- **Three layers:** decorator operation spans; child spans per backend call,
  with parent-gated HTTP transports or a span decorator over the SDK seam; and
  logs.
- **The ctx and logger rules** that keep `traceID` on log lines.
- **A per-provider status table:**
  - kubernetes: parent-gated `RestConfigWrapper` spans.
  - gcp: the Google SDK's own `otelhttp`, never disabled by the provider.
  - kvm: no spans. `trace.go` is a log decorator.

Also added two tracing rows to the §7 matrix, rewrote checklist item 8 as
"Tracing and logging" with an in-memory span-recorder test, and added gaps (kvm
has no child spans; only kubernetes is checked against Tempo, and only at
operation level). A proposal notes that the kvm `libvirtAPI` seam takes no
`ctx`, so per-RPC spans need that plumbed in.

The links were re-checked, including the new `architecture.md#10-tracing-internaltracing` anchor.