egress-proxies-operator/docs/plans-executions/2026-08-07-1747-proxy-operator.md

# Execution log: proxy-operator

Pairs with [docs/plans/2026-08-07-1747-proxy-operator.md](../plans/2026-08-07-1747-proxy-operator.md).

## Status

- [x] Step 0 — Branch and scaffold
- [x] Step 1 — API types (`api/v1alpha1/proxy_types.go`)
- [x] Step 2 — Provider contract (`internal/provider/`)
- [x] Step 3 — Kubernetes pod provider (`internal/provider/kubernetes/`; first built as an in-memory mock, then replaced — see the two Step 3 sections below)
- [x] Step 4 — Reconciler (`internal/controller/`)
- [x] Step 5 — Health engine (`internal/health/`)
- [x] Step 6 — Lease store (`internal/lease/`)
- [x] Step 7 — Discovery API (`internal/discovery/`)
- [x] Step 8 — GCP provider (`internal/provider/gcp/`)
- [x] Step 9 — Orphan GC + metrics
- [ ] Step 10 — Wiring, config, docs
- [ ] Step 11 — Tests
- [ ] Verification (vet/test/kind e2e) + commit, push, open MR

## Step 0 — Branch and scaffold

Branched off the unborn `main`:

```bash
git checkout -b feat/proxy-operator
```

Installed kubebuilder v4.15.0 into a scratch `GOBIN` rather than the default
`$(go env GOPATH)/bin`, since the module's package layout changed and
`go install .../cmd/kubebuilder@v4.15.0` (the path from the plan) 404s — the binary is
now the module root itself:

```bash
GOBIN=<scratch>/bin go install sigs.k8s.io/kubebuilder/v4@v4.15.0
```

Scaffolded in place, with `kubebuilder` on `PATH`:

```bash
kubebuilder version
# KubeBuilder: v4.15.0, Kubernetes: 1.36.0

kubebuilder init --domain example.com \
  --repo gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator --plugins go/v4
# WARN: target directory not empty (expected — CLAUDE.md, docs/, .claude/ already existed)

kubebuilder create api --group crawl --version v1alpha1 --kind Proxy \
  --resource --controller
```

`create api` auto-ran `make manifests` at the end, which pulled and ran
`controller-gen` itself:

```bash
sigs.k8s.io/controller-tools/cmd/controller-gen@v0.21.0
"$(bin)/controller-gen" object:headerFile="hack/boilerplate.go.txt",year=2026 paths="./..."
```

Confirmed the CRD group landed correctly (no doubling — `--domain example.com
--group crawl` was used specifically to avoid the `crawl.crawl.example.com` trap
called out in the plan):

```bash
grep -A2 "GroupVersion =" api/v1alpha1/groupversion_info.go
# SchemeGroupVersion = schema.GroupVersion{Group: "crawl.example.com", Version: "v1alpha1"}
```

Dropped the scaffolded GitHub Actions workflows (remote is Gitea, not GitHub):

```bash
git rm -r --cached .github 2>/dev/null; rm -rf .github
```

Ran the full manifest/codegen pass once more to confirm the toolchain is reproducible
end to end:

```bash
make manifests generate
# controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
# controller-gen object:headerFile="hack/boilerplate.go.txt",year=2026 paths="./..."
```

then `go build ./...` and `go vet ./...`, both clean with no output.

Worth noting: `CONTROLLER_TOOLS_VERSION` in the generated `Makefile` came out at
`v0.21.0` by default in this kubebuilder release, so the Makefile edit the plan
anticipated wasn't needed. Pre-existing `CLAUDE.md`/`CHANGELOG.md` content survived
untouched; kubebuilder added its own `README.md`, `AGENTS.md`, `.golangci.yml`,
`.devcontainer/`, `Dockerfile` on top of them — those get edited or left as-is in
later steps. Committed as `076bc66`.

## Step 1 — API types (`api/v1alpha1/proxy_types.go`)

Wrote the full `ProxySpec`/`ProxyStatus`/`Proxy` types per the plan, including the
four corrections called out there (`MaxLeases *int32`, `HealthCheck` with
`+kubebuilder:default={}`, `MinLength=1` on `Provider`/`CloudInit.Inline`,
`Conditions` with `+listType=map`), the 7 CEL `XValidation` rules (6 on `ProxySpec`,
1 on `CloudInitSpec`), and pure helpers in `helpers.go`
(`EffectivePort`/`EffectiveHost`/`HealthCheckOrDefault`/`MaxLeasesOrDefault`) with
table-driven tests in `helpers_test.go`.

Regenerated deepcopy and the CRD:

```bash
make manifests generate
# controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
# controller-gen object:headerFile="hack/boilerplate.go.txt",year=2026 paths="./..."
```

Confirmed all 7 CEL rules and the `healthCheck` `default: {}` block landed in the
generated CRD as expected:

```bash
grep -B1 "rule:" config/crd/bases/crawl.example.com_proxies.yaml
# 7 matches, one per XValidation marker written
```

Ran the full suite, not just `go build`/`go vet`, since this was a good opportunity to
confirm envtest itself works end to end for the first time:

```bash
make test
# Setting up envtest binaries for Kubernetes version 1.36...
# .../bin/k8s/1.36.2-darwin-arm64   (confirms the plan's envtest version note)
```

This failed on the first run — not because of anything in the new types, but because
kubebuilder's scaffolded placeholder test in `proxy_controller_test.go` creates a bare
`Proxy{}` with no `spec.mode`, which our new required/enum field correctly rejects:

```text
Proxy.crawl.example.com "test-resource" is invalid: [spec.mode: Unsupported value: "":
supported values: "Managed", "External", ...]
```

That's a real envtest apiserver enforcing our schema for the first time, which is
useful confirmation on its own. Patched just the resource literal in that scaffold
test to a minimal valid spec (`Mode: External` + `Endpoint.Host`) rather than
rewriting the file — that whole test gets replaced in Step 4 alongside the real
reconciler, so a deeper fix now would be thrown away. `make test` then passed clean:
`api/v1alpha1` at 20.5% coverage (helpers only — CEL itself isn't unit-testable, it's
exercised by the real apiserver as shown above), `internal/controller` at 66.7%.

Worth noting: the `go test ./...` command from the plan's own verification section
does *not* work directly for the envtest suite — it needs `KUBEBUILDER_ASSETS` set,
which only `make test` does via `setup-envtest`. Plain `go test ./...` fails the
`internal/controller` package with a `/usr/local/kubebuilder/bin/etcd: no such file`
error that has nothing to do with the code. Use `make test`, not `go test ./...`,
whenever the controller package is in scope.

## Step 2 — Provider contract (`internal/provider/`)

Wrote the `Provider` interface (`Create`/`Get`/`Delete`/`ListByTag`), `Instance`,
`Placement`, `CreateRequest`, and the GC-contract label constants
(`provider.go`); the error taxonomy with multi-error `Unwrap() []error` so
`errors.Is` and `errors.As` both work off the same wrapped value
(`errors.go`); deterministic instance naming via SHA-256 → base32 → 16
chars (`name.go`); and `--providers-config` YAML parsing with fail-fast
validation (`config.go`).

One deliberate deviation from the plan's file layout: the plan listed
`internal/provider/metrics.go` as part of this step, but the
`provider.WithMetrics` decorator it describes is Step 9's concern (it needs
the Prometheus vectors that don't exist until the metrics package is
built) and nothing in this step depends on it existing yet. Deferred to
Step 9 rather than writing a decorator with nowhere to register its
metrics.

The registry package (`internal/provider/registry/registry.go`) came out
slightly different from the plan's sketch, and better for it: instead of a
package-level `var builtin = map[string]Constructor{"mock": mock.New, "gcp":
gcp.New}` living inside the registry package, `Build` takes the
`map[string]Constructor` as a parameter. This means `registry` has zero
import on `internal/provider/mock` or `internal/provider/gcp` — neither of
which exists yet at this point in the plan (mock is Step 3, gcp is Step 8)
— so the package compiles today instead of only once both are done, and the
explicit wiring lives at the composition root (`cmd/main.go`, Step 10)
rather than being smeared into the registry package itself. Still fully
avoids the import-cycle trap the plan called out.

Ran the full suite:

```bash
go mod tidy   # sigs.k8s.io/yaml (already an indirect dep of the k8s.io toolchain) promoted to direct
go build ./... && go vet ./...
go test -race -v ./internal/provider/...
make test
```

`internal/provider` landed at 96.2% coverage, `internal/provider/registry` at
100%. `make test` also ran `go fmt ./...`, which reformatted `errors.go`'s
struct-field comment alignment before its first commit — no logic change,
just gofmt on a brand-new file.

Worth noting: `sigs.k8s.io/yaml` (not `gopkg.in/yaml.v3`) was picked for
`--providers-config` parsing specifically because it has `UnmarshalStrict`
built in (rejects unknown fields, which is what "fail fast on unknown type"
in the plan actually needs) and was already pulled in transitively by the
k8s.io toolchain, so no new dependency was added — `go mod tidy` just
promoted it from indirect to direct.

## Step 3 — Mock provider (`internal/provider/mock/`)

Started from the plan's design (state as a pure function of an injectable
clock, no background timers) but had to redesign the "real proxy" part
before writing any code, once a load-bearing assumption turned out false.

The plan (and the earlier decision to make the mock run a real proxy)
assumed each instance could get its own loopback address —
`127.0.0.1:<port>` per instance, "a fake IP from a private range." Verified
that assumption directly before committing to it:

```bash
cat <<'EOF' > /tmp/loopbacktest.go
package main
import ("fmt"; "net")
func main() {
	for _, addr := range []string{"127.0.0.2:0", "127.0.0.55:0", "127.1.2.3:0"} {
		l, err := net.Listen("tcp", addr)
		if err != nil { fmt.Printf("%s: FAIL: %v\n", addr, err); continue }
		fmt.Printf("%s: OK\n", addr)
		l.Close()
	}
}
EOF
go run /tmp/loopbacktest.go
# 127.0.0.2:0: FAIL: listen tcp 127.0.0.2:0: bind: can't assign requested address
# 127.0.0.55:0: FAIL: listen tcp 127.0.0.55:0: bind: can't assign requested address
# 127.1.2.3:0: FAIL: listen tcp 127.1.2.3:0: bind: can't assign requested address
```

Only `127.0.0.1` binds on macOS without `sudo ifconfig lo0 alias ... up` —
Linux routes the whole `127.0.0.0/8` block to loopback by default, macOS
doesn't. That's not something the operator can or should do at runtime, so
per-instance loopback IPs were out. There's also a second problem the
per-instance-IP design didn't solve anyway: `EffectivePort()`
(`api/v1alpha1/helpers.go`, Step 1) is computed purely from `spec.port`
with no channel for a provider to report back a different *port* — so
whatever a mock instance actually listens on has to be the literal port the
reconciler will pass through `CreateRequest.Port`, not an OS-assigned
ephemeral one.

Redesigned around one real listener **per port**, shared and
reference-counted across every instance that uses it, rather than one
listener per instance (`proxy.go`, `sharedProxies`). This fixes both
problems at once: every instance binds the same `127.0.0.1` (no OS issue),
and any number of instances can share a port without conflict since it's
the exact same underlying listener. Deliberately made the refcounting
**package-level**, not a field on `mock.Provider`, because a bound TCP port
is a genuinely process-global OS resource — two separately configured
mock-typed provider entries (e.g. two named `"mock"` instances in
`providers-config.yaml`) would otherwise both try to bind the same default
port and the second one would just fail. This is a case where a package
global is the correct model, not a shortcut: it mirrors an OS-level
singleton, not application state.

Instance lifecycle otherwise follows the plan exactly: `Get`/`ListByTag`
derive `Provisioning → Running → Terminated → purged (ErrNotFound)` from
`createdAt`/`deletedAt` compared against an injectable clock, with the real
proxy listener acquired lazily on the first observed `Running` and released
on `Delete` (or lazily on purge, so orphaned records can't leak a
reference). `Create` is idempotent by name. Fault injection is wired both
ways per the plan: `MockConfig.FailNextCreates`/`FailWith` for the demo
config, `InjectCreateFailures(n, class)` for tests.

Tests initially had a real flake, caught by running with `-race -count=3`
rather than trusting one green run:

```bash
go test -race -v ./internal/provider/mock/... 2>&1 | tail -5
# --- FAIL: TestAcquireProxy_sharedAcrossAcquires
#     proxy_test.go:37: port not released after last reference: bind: address already in use
```

Root cause wasn't the refcounting logic — it was the test helper. `freePort`
asked the OS for a free port by binding to `:0` and immediately closing it,
which is a classic TOCTOU race under `t.Parallel()`: two tests can be handed
the same "free" port before either actually claims it, since nothing holds
it open in between. Fixed by replacing the OS-asks approach with a
monotonic counter (`20000 + atomic.Int32`) — these tests only need a port
unique *within this test run*, not one verified free by the OS at an
instant in time, so guaranteeing uniqueness outright is both simpler and
correct where the "ask and hope" approach wasn't. Reran `-race -count=3`
clean afterward.

Added tests beyond the plan's list to close real coverage gaps rather than
stopping at "green": config-override branches in `New`, all four
`failClassFromString` branches, `ListByTag`'s purge-on-list and
Running/IP-inclusion paths, and a plain-`http://` forwarding test
(`handleForward`) alongside the CONNECT one, since a `probeURL` override
could use either scheme. Landed at 91.1% coverage; the remainder is
OS-failure branches (bind errors, hijack failures) not worth simulating for
a prototype.

The `TestProvider_realProxyTunnelsConnect` test is the one that actually
matters most here: it opens real sockets end to end — mock `Create` →
`Get` past `provisionDelay` → real `http.Client` with
`Transport.Proxy` dialing through the mock's CONNECT tunnel to a real
`httptest.NewTLSServer` — and gets a real `204` back. That's the concrete
proof the "mock runs a real proxy" decision actually delivers a genuine
end-to-end healthcheck, not a simulated one.

`internal/provider/mock` at 91.1% coverage. `make test` green across the
whole repo.

## Step 3 (revised) — Kubernetes pod provider replaces the mock

After the mock provider above was built and working, the user pushed back:
it felt too far from the real system to build confidence in, and they'd
rather have simpler, more "real" code than a fast-but-simulated test
double — a `kind`-based verification pass "once in a while" is an
acceptable trade. They proposed replacing it outright with a provider that
creates real Pods in the operator's own cluster, rather than keeping mock
around as a fallback.

**Talked through the trade-off before agreeing to it, since it wasn't free
of downsides.** Pods sharing a cluster's egress IPs don't solve what this
operator actually exists for (routing around IP-based rate limiting needs
genuinely distinct egress paths — that's still only `gcp`), and `envtest`
has no kubelet, so a Pod-based provider can never be exercised by the fast
test suite either. Net effect: this is a replacement for the mock's role
in local dev/CI confidence-building, not a new alternative to GCP, and the
reconciler's own state-machine tests (Step 4) still need a minimal
in-test stub `Provider` — a handful of lines in the test file, not a
package with its own config format or fault-injection surface, which is
the specific kind of complexity the user was pushing back on.

**Picked the proxy software by actually checking what's out there,** not
by guessing a Docker Hub path:

```bash
curl -s "https://hub.docker.com/v2/repositories/vimagick/tinyproxy/" | head -c 400
# last_updated 2021-07-22 — stale
curl -s "https://hub.docker.com/v2/repositories/ubuntu/squid/" | head -c 400
# last_updated 2026-08-07T04:36:14Z — updated the same day, Canonical-published, 50M+ pulls
```

`ubuntu/squid` won clearly: actively maintained (rebuilt the same day this
check ran), official publisher, and — since it's a public image — `kind`
nodes pull it directly, no build/load step needed for the quickstart.
Pinned to `6.6-24.04_edge` (Ubuntu 24.04 LTS base) rather than floating
`latest`, for reproducibility.

**Design, in `internal/provider/kubernetes/`:**

- **`pod.go`** — `buildPod` is a pure function (mirrors the GCP provider's
  planned `buildInsertRequest`): builds a `corev1.Pod` with one Squid
  container. Config is generated in Go and passed via a `SQUID_CONF` env
  var that the container's command writes to `/etc/squid/squid.conf`
  before `exec squid` — deliberately not a separate `ConfigMap`, so
  there's still exactly one Kubernetes object per proxy instance to
  create, track, and clean up. The config itself is intentionally
  permissive (`http_access allow all`, `via off`, `forwarded_for off`) —
  documented in-code as a prototype-for-a-private-cluster choice, the same
  posture the spec already takes toward cloud-init on the GCP provider
  (installing/configuring proxy software is explicitly out of scope
  there; this provider doesn't try to do more).
- **`kubernetes.go`** — `Create`/`Get`/`Delete`/`ListByTag` against a
  `client.Client`. **`providerID` is `<namespace>/<podName>`**, parsed
  with `k8s.io/client-go/tools/cache.SplitMetaNamespaceKey` — the exact
  same reasoning as the GCP provider's planned zone-qualified providerID
  (Step 8): `Get`/`Delete` need to be self-contained without re-deriving
  where the resource lives. State mapping collapses `Succeeded`/`Failed`/
  `Unknown` all into `Terminated`, since the reconciler already treats
  Stopped and Terminated identically (delete + recreate) — no finer
  distinction would change any behavior. `Running` with no `PodIP` yet
  maps to `Provisioning`, not `Running`, so an empty IP is never
  published — the same rule the plan already called out for GCP's
  `RUNNING`-without-`NatIP` case.
- **Client construction is the one genuinely new pattern this provider
  needed** that GCP/mock didn't: it builds its own `client.Client` via
  `ctrl.GetConfig()`, which auto-detects in-cluster config when running as
  a Pod and falls back to the local kubeconfig otherwise. That's what
  makes `make run` against a local `kind` cluster and running in-cluster
  use the exact same code path with zero provider-specific wiring in
  `cmd/main.go`. The corresponding risk: `New()` must never run under
  `go test` — it would happily connect to whatever real cluster the
  developer's kubeconfig points at. Solved the same way `mock.Provider`
  solved clock injection: an unexported `newWithClient(c, cfg)` constructor
  that tests call directly, bypassing `ctrl.GetConfig()` entirely.
  `New()` sits at 0% test coverage, deliberately — it's the one function
  that must stay untested by design.
- **RBAC implication worth flagging now** (implemented in Step 10):
  `ListByTag` lists Pods across every namespace, not just the namespace(s)
  Proxies live in, because orphan GC needs to find every Pod this operator
  tagged regardless of where it landed. That means the operator's Pod
  permissions have to be a `ClusterRole`, not scoped to a single
  namespace — a real trade-off against the "fleet lives in one namespace,
  keeps RBAC simple" principle the spec states for the CRD itself.
  Documented rather than special-cased away.
- **One known simplification, documented in a code comment rather than
  solved:** Kubernetes surfaces both RBAC-denied and quota-exceeded as the
  same 403 Forbidden, and `apierrors` has no helper to tell them apart.
  Both classify as `ErrPermanent` — the safer of the two defaults (stop
  retrying rather than hammering an API server that will never allow the
  request), but a real ResourceQuota failure that would clear once other
  proxies are deleted won't get the 5-minute-backoff-and-retry treatment
  `ErrQuotaExceeded` gives on the GCP path.

**Testing:** unlike the mock, this package's tests use
`sigs.k8s.io/controller-runtime/pkg/client/fake` — real `corev1.Pod`
objects, the real `client.Client` interface, not a hand-rolled in-memory
map. That's a strictly more realistic test double than what mock.Provider
was, while still being fast (no cluster, no kubelet) — it just can't prove
a container actually starts and serves traffic, which is exactly the gap
the `kind`-based verification pass is for. 77.6% coverage; the only
meaningfully uncovered function is `New()` itself, deliberately.

`make test` green across the whole repo (`go build`/`go vet` clean,
`internal/provider` 96.0%, `internal/provider/kubernetes` 77.6%,
`internal/provider/registry` 100%, unchanged).

## Cleanup — removed unused cert-manager/webhook scaffolding

Not a plan step; the user asked for this directly after I explained what
`make test-e2e` currently does, and didn't want unused scaffold machinery
carried forward into later work.

kubebuilder's generic scaffold assumes a project *might* grow admission
webhooks later, so it wires up webhook TLS-cert machinery and an
unconditional cert-manager install in the e2e suite defensively. Checked
whether any of it was actually load-bearing before touching anything:

```bash
grep -rn "cert-manager\|certmanager\|CertManager" config/
# only inside commented-out [CERTMANAGER] blocks in kustomization.yaml —
# the whole block is inert, never uncommented
grep -n "webhook" PROJECT
# no output — kubebuilder create webhook was never run
```

Confirmed nothing here does anything for this project — no `config/webhook/`
exists, no `+kubebuilder:webhook` markers exist anywhere, and the spec's
own non-goals explicitly rule out admission webhooks and cert-manager
wiring forever, not just "not yet."

Removed:

- **`cmd/main.go`**: the `webhook` import, the three `webhook-cert-*`
  flags, the `webhookServerOptions`/`webhookServer` construction, and the
  `WebhookServer:` field on `ctrl.Options` — the manager runs with no
  webhook server at all now, which is correct since nothing registers one.
  Left the metrics-cert flags alone (`--metrics-cert-path` etc.) — those
  let real certs be mounted for the metrics endpoint without cert-manager,
  which is unrelated to webhooks and still useful. Reworded a comment that
  said "TODO(user): If you enable certManager..." since that will never
  happen here.
- **`test/e2e/e2e_suite_test.go`**: the unconditional cert-manager install
  in `BeforeSuite` and matching uninstall in `AfterSuite`
  (`setupCertManager`/`teardownCertManager`/`shouldCleanupCertManager`),
  and the doc comment claiming the suite "requires Kind and CertManager"
  (it only requires Kind now).
- **`test/utils/utils.go`**: `InstallCertManager`, `UninstallCertManager`,
  `IsCertManagerCRDsInstalled`, and their now-unused `warnError` helper and
  `certmanagerVersion`/`certmanagerURLTmpl` constants. Also removed
  `UncommentCode` — grepped first and confirmed it had zero callers even
  before this cleanup; it was dead scaffold code from the start, unrelated
  to cert-manager, just found while in there.

Left the inert commented-out `[WEBHOOK]`/`[CERTMANAGER]` blocks in the
`config/*/kustomization.yaml` files and the
`+kubebuilder:scaffold:e2e-webhooks-checks`-style marker comments in
`test/e2e/e2e_test.go` alone — those are standard kubebuilder codegen
anchors and pure comments with no runtime behavior, unlike the cert-manager
install this cleanup actually removed. Stripping every trace of "webhook"
from every scaffold comment across the tree would be a much bigger, purely
cosmetic diff for no behavioral benefit; this cleanup targeted the things
that were actually *doing* something.

Verified with both build tags, since `test/e2e` only compiles under `-tags=e2e`:

```bash
go build ./... && go vet ./...
go build -tags=e2e ./... && go vet -tags=e2e ./...
```

Both clean. `make test` green across the whole repo, unchanged from before
the cleanup.

## Cleanup — lean-down audit of Steps 0–3

Planned and approved separately in
[docs/plans/2026-08-08-1335-lean-scaffold-cleanup.md](../plans/2026-08-08-1335-lean-scaffold-cleanup.md).
Prompted by the user asking (a) why webhook machinery existed at all when the
spec's §12 says "do NOT build: admission webhooks, cert-manager wiring", and
(b) for a full audit of completed work so the project starts as lean as
possible.

**Why the webhook machinery existed — for the record.** `kubebuilder init`
emits it unconditionally: the active webhook-server wiring in `cmd/main.go`,
cert-manager install in the e2e utils, and commented kustomize anchor blocks
all arrive with `init`, not with `create webhook` (never run here). At Step 0
the scaffold was deliberately committed untouched as a reviewable baseline
(only `.github/` stripped), and spec §3's "config/ (scaffold-generated, kept
working)" was read as license to keep the rest. The process gap: §12's
non-goals deserved a pruning pass immediately *after* the baseline commit,
especially for the parts that actually did something. The user caught it, not
the build process.

**Could `init` have skipped it?** No — verified against the v4.15.0 binary,
not from memory:

```bash
kubebuilder init --help
# flags: --domain --repo --owner --license(-file) --multigroup --namespaced
#        --fetch-deps --skip-go-version-check --project-version --plugins
# nothing subtracts features; optional plugins (helm, grafana, deploy-image,
# autoupdate) are all additive — there is no "minimal" plugin
```

Scaffold-then-prune is the only supported path to a lean baseline. Worth
knowing at the *next* project bootstrap: plan the pruning pass as part of
scaffolding, not as a later discovery. (`--namespaced` was the one
arguably-applicable flag, but it conflicts with the kubernetes-pod provider's
cluster-wide `ListByTag` for orphan GC — cluster-scoped was correct.)

**Audit result for the Go code: clean.** Nothing beyond spec that isn't a
justified, already-logged deviation (`Instance.UID`/`CreatedAt`,
`MaxLeases *int32`, `HealthCheck default={}`, registry-as-parameter,
kubernetes provider per user decision). Trivial extras (`shortName=px`,
`MaxProperties=32`) stay.

**Scope lines drawn by the user during plan review** — recorded because they
shape what "lean" means for this repo going forward:

- Developer tooling is exempt: `.golangci.yml`, `.custom-gcl.yml`,
  `.devcontainer/`, `AGENTS.md`, Makefile `lint`/`docker-buildx`/
  `build-installer` targets all stay ("tooling around the project is cool, i
  just want the application code produced to start as lean as possible").
- Monitoring manifests stay: all of `config/prometheus/`, plus the paired
  metrics-TLS-via-cert-manager plumbing
  (`config/default/cert_metrics_manager_patch.yaml` and the metrics-certs/
  ServiceMonitor halves of the commented replacements block) — removing half
  of a pair would leave dangling comment references.
- All RBAC manifests stay, including the `proxy_admin/editor/viewer` helper
  ClusterRoles whose own headers say "not used by the project itself".
- Network policies go: "we do not need any network policies at the moment".

**Removed:**

- `config/network-policy/` (kustomization.yaml, allow-metrics-traffic.yaml)
  plus its commented `#- ../network-policy` line in the default kustomization.
- Webhook-only remnants in `config/default/kustomization.yaml`: the commented
  `#- ../webhook` / `#- ../certmanager` resource lines, the
  `manager_webhook_patch.yaml` patch reference, and the webhook halves of the
  replacements block (serving-cert → Validating/Mutating WebhookConfiguration
  cainjection, conversion webhook, and the
  `+kubebuilder:scaffold:crdkustomizecainjection*` markers — anchors only for
  `kubebuilder create webhook`, permanently a non-goal).
- The two commented `[WEBHOOK]` blocks in `config/crd/kustomization.yaml`
  (conversion patches + the `configurations:` reference), along with the empty
  `patches:` key they lived under and the `crdkustomizewebhookpatch` marker.
  Kept the one-line `crdkustomizeresource` marker — `kubebuilder create api`
  could legitimately run again.
- `config/crd/kustomizeconfig.yaml` — only consumer was the removed
  `configurations:` block.

Verified: `bin/kustomize build config/default` and `... config/crd` both
render cleanly (no dangling references), `go build`/`go vet` clean with and
without `-tags=e2e`, `make test` green with coverage numbers identical to
pre-cleanup.

## Step 4 — Reconciler (`internal/controller/`)

Implemented the state machine per the plan's action table:
`proxy_controller.go` (dispatch + managed/external/delete paths, cloud-init
resolution, spec-hash annotation persistence, Secret→Proxy watch mapping),
`status.go` (condition reasons, `computePhase`, the single deferred
`patchStatusIfChanged`), and `spechash.go` (explicit
`{placement, resolved cloud-init, port}` hash input, SHA-256 hex). Tests:
the action-table suite against a fake client with an in-test `stubProvider`
(`reconcile_test.go`), `computePhase` truth table, spec-hash
stability/normalization/sensitivity tables, and a rewritten envtest suite
(`proxy_controller_test.go`) driving full lifecycles — provision→Running,
spec-change replacement, finalizer deletion, External tracking — against
the real apiserver with real CRD defaulting.

**Deviation from the plan's `Requeue: true` rows:** `ctrl.Result{Requeue}`
is deprecated in controller-runtime v0.24 (verified in the vendored source,
`pkg/reconcile/reconcile.go`: "Deprecated: Use `RequeueAfter` instead"), and
golangci's staticcheck would flag it. Those rows use a fifth configurable
interval instead, `RequeueNow` (default 1s) — same "process the next state
promptly" semantics, still shrinkable in tests like the other four.

**A real bug the new tests caught on their first run** (both the fake-client
and envtest suites, independently): in the replacement path's
instance-is-gone branch, the status clear (`ProviderID = ""`) was staged
*before* `setSpecHash`'s metadata `Update` — and `client.Update` refreshes
the whole object from the server's response, *including status*, so the
staged clear was silently overwritten and the proxy wedged with a stale
providerID. Fix: stage status changes only after any metadata Update
(the create branch already did it in that order). Worth remembering for
every future reconciler: **`r.Update` clobbers in-memory status staged
before it.**

Two judgment calls the plan left open, now documented in code:

- `computePhase` maps Provisioned=True with no Healthy verdict yet to
  `Provisioning`, not `Ready` — a proxy nobody has probed shouldn't be
  advertised as Ready. Health (Step 5) flips it.
- `deletionFailure` (the finalizer path's error handler) never latches
  `ErrPermanent` the way `providerFailure` does — latching there would
  wedge the object forever with no retry; it keeps retrying visibly
  instead.

The permanent-failure latch compares the condition's `observedGeneration`
against the CR generation, so a spec edit automatically clears Failed and
retries — no manual annotation-poking needed to recover.

Verification:

```bash
make test    # regenerates manifests (role.yaml gains secrets get;list;watch), envtest green
KUBEBUILDER_ASSETS="$PWD/bin/k8s/1.36.2-darwin-arm64" go test -race ./internal/controller/
go test -short ./internal/controller/   # 0.6s — envtest suite correctly skipped
go build -tags=e2e ./... && go vet -tags=e2e ./...
```

`internal/controller` at 75.9% coverage; the envtest suite runs in ~6s and
is now guarded by `testing.Short()` per the testing conventions.

Worth noting: the envtest specs simulate instance state by mutating the
stub between direct `Reconcile` calls rather than running the manager —
deterministic and fast, at the cost of not exercising watch-driven
requeues; Step 11's manager-driven cases cover that. The Secret watch is
wired in `SetupWithManager` but the label-restricted Secret cache it
assumes arrives with `cmd/main.go` in Step 10.

## Step 5 — Health engine (`internal/health/`)

Implemented the engine per the plan's design: `probe.go` (through-the-proxy
probe with the plan's exact transport — fresh per probe,
`DisableKeepAlives: true` so every probe re-exercises CONNECT) and
`engine.go` (leader-elected manager Runnable: 1 s scheduler tick + a pool
of 8 workers, per-proxy threshold state under one mutex, transition-only
emission over a buffered `chan event.GenericEvent`). The reconciler side
landed in the same step: a `HealthSnapshotter` interface + `applyHealth`
staging the Healthy condition/latency/lastHealthCheckTime from
`Engine.Snapshot`, and a conditional
`WatchesRawSource(source.Channel(...))` in `SetupWithManager`. Everything
tolerates nil (engine unwired) until `cmd/main.go` connects the two in
Step 10.

Deviations and judgment calls beyond the plan text:

- **First-probe scheduling is split by whether a verdict was seeded.** The
  plan's startup jitter (`nextDue = now + rand(0, interval)`) applies only
  to proxies whose state was seeded from an existing Healthy condition —
  the restart case it exists for. A never-probed proxy is probed on the
  next tick instead; making a brand-new proxy wait up to a full interval
  for its first verdict would be pure lag with no thundering-herd benefit.
- **The latency-change emission rule only applies while the verdict is
  healthy.** Caught by the first test run, not foreseen: a success streak
  still below `successThreshold` (verdict unhealthy, reported unhealthy)
  satisfied the plan's rule (c) — latency delta vs a stale reported value,
  rate window open — and emitted a pointless latency-only update for a
  proxy still reported as unhealthy. Guarded with `res.ok && *st.healthy`.
- **`ProbeTLSConfig` field added to the engine** (nil = system roots). The
  probe function needs a CA override to be testable against
  `httptest.NewTLSServer`, and the same knob is genuinely useful for
  probing targets signed by a private CA. Not a test-only backdoor.
- **State pruning doubles as replacement hygiene:** any proxy with no
  probeable host (provisioning, mid-replacement, deleting) has its state
  dropped each tick, so a replacement instance always starts with fresh
  counters. Complementarily, the reconciler's create branch removes the
  stale Healthy condition and latency fields — a new VM shouldn't wear its
  predecessor's verdict.

Tests: a real CONNECT-capable proxy stub (hijack + bidirectional
`io.Copy`) probing a real `httptest.NewTLSServer` — CONNECT success,
refused CONNECT, unexpected status, dead proxy, plain-http forwarding;
table-driven threshold/suppression/seeding/pruning tests driving
`record`/`tick` directly; an end-to-end `Start` test (fake reader, fake
probeFn, 5 ms tick) asserting event delivery, snapshot content, and clean
shutdown on context cancel; and controller-side tests with a
`fakeSnapshotter` proving Running+healthy ⇒ `Ready`, Running+unhealthy ⇒
`Unhealthy`, no-verdict ⇒ no condition, and stale-verdict cleanup on
replacement.

Verification (all green):

```bash
make test                          # envtest + units; health 93.4%, controller 77.4%
KUBEBUILDER_ASSETS="$PWD/bin/k8s/1.36.2-darwin-arm64" go test -race ./...
go test -race -count=2 ./internal/health/   # shook out the emission-rule bug above
```

Worth noting: `docs/architecture.md` (created between Steps 4 and 5 on
user request) gained a §6 for the engine and now shows the third workqueue
feed (`source.Channel`). The `Healthy` condition reasons live in the
controller package (`ReasonProbeSucceeded`/`ReasonProbeFailed`) — the
engine deliberately knows nothing about conditions except reading one at
seed time, keeping the state/representation split honest. The
`hint`-driven `wg.Go` idiom (Go 1.25+) replaced the classic
`wg.Add/defer wg.Done` in the worker pool.

## Step 6 — Lease store (`internal/lease/`)

Implemented `store.go` per the plan: `Acquire` takes the whole candidate
set so selection and insertion happen under the one store mutex (no
overcommit between concurrent requests), selection is a linear scan +
`slices.SortFunc` on `(activeLeases asc, latency asc, name asc)`,
`AcquireStats{Considered, AtCapacity, InCooldown}` feeds Step 7's 409
body, cooldowns live in a `map[{proxy, target}]time.Time` (empty target =
global pool), and expired leases are retained for `CooldownWindow` past
their TTL so a late `Report` — arriving exactly when a proxy is being
rate-limited — still resolves and records its cooldown.

Semantics pinned against the spec (§8) rather than guessed:

- Report results are exactly `ok | rate_limited | banned` (`ParseResult`
  gives the API layer its 400 check). `rate_limited` and `banned` both
  record a cooldown for the same window; `ok` records nothing.
  Distinguishing ban duration from rate-limit duration would be a second
  knob the spec doesn't ask for — noted for the Decisions section.
- Cooldown scoping: the global cooldown (empty target) always applies; a
  target-scoped cooldown additionally blocks acquisitions for that target;
  acquisitions without a target see only the global pool ("a proxy
  rate-limited by one site is still fine for everyone else").
- A `Report` without a target falls back to the lease's own target before
  falling back to global — so a client that leased with a target doesn't
  accidentally poison the whole proxy by omitting it in the report.

Design notes:

- **Correctness never depends on the sweep.** Every read path
  (`Acquire`/`ActiveCount`/`Counts`) compares `ExpiresAt` against the
  injected clock, so TTL expiry frees capacity immediately even if the
  background loop hasn't run; the sweep is purely garbage collection. The
  plan's `ExpireLoop` became `Start(ctx)` + `NeedLeaderElection() false`
  so the store satisfies `manager.Runnable` directly — Step 10 just
  `mgr.Add(store)`s it. Not leader-elected because lease state is
  per-process and must expire wherever the discovery API is serving.
- The store knows nothing about Proxy objects — `Candidate` carries the
  opaque key, `MaxLeases`, and latency; the discovery layer does the
  health/attribute filtering. The spec's `LeaseStore` interface will be
  defined consumer-side in `internal/discovery` (Step 7), per Go idiom;
  this package exports only the concrete in-memory `*Store`.
- Lease IDs come from `crypto/rand.Text()` (Go 1.24+); returned `Lease`
  values are copies so callers can't mutate store internals.

Tests (94.8% coverage, `-race -count=2` clean): capacity + release
freeing slots, `MaxLeases=0` unleasable, least-loaded/latency/name
selection order, target-scoped vs global cooldown scoping, cooldown
expiry via the injected fake clock, TTL freeing capacity with no sweep,
report-on-expired-but-retained lease (then `ErrUnknownLease` after
retention), `ok` recording nothing, idempotent release, `ParseResult`,
40 concurrent acquires against `MaxLeases=5` granting exactly 5, and the
`Start` loop sweeping then stopping cleanly on cancel.

```bash
go test -race -count=2 ./internal/lease/
make test    # whole repo green, other packages' coverage unchanged
```

Worth noting: `docs/architecture.md` was not extended this step — the
lease store is HTTP-driven, not cluster-event-driven, so its diagram
belongs with the discovery API and lands in Step 7 (banner updated to say
so).

## Step 7 — Discovery API (`internal/discovery/`)

Implemented `server.go` (Runnable + middleware chain) and `handlers.go`
(the four endpoints + `proxyView` wire shape) per the plan: stdlib
`http.ServeMux` method+wildcard routing (no third-party router — see the
plan clarification commit: this is a stdlib feature since Go 1.22, the
project stays on the pinned Go 1.26), middleware outermost-first recover →
request-log → `MaxBytesReader(64KiB)` → constant-time bearer auth with
`/healthz` exempt, empty `DISCOVERY_TOKEN` serving unauthenticated with a
loud startup warning, `NeedLeaderElection() = false` with the plan's
runnable-ordering rationale in the doc comment, and graceful `Shutdown`
with a 10 s grace on ctx cancel.

The `LeaseStore` interface landed consumer-side in this package (spec §8
wants handlers swappable to a CRD/Redis store); `internal/lease.*Store`
satisfies it without modification.

Judgment calls the plan/spec left open:

- **409 arithmetic:** the store only ever sees healthy candidates, so its
  `Considered` excludes unhealthy matches. The handler counts unhealthy
  selector-matches itself and reports `considered = healthy + unhealthy`,
  keeping the plan's example arithmetic (7 = 2+2+3) consistent.
- **TTL handling:** omitted/zero `ttlSeconds` → 300 s default; negative or
  above `MaxLeaseTTL` (default 1h, flag in Step 10) → 400 `invalid_ttl`
  rather than silent clamping — a client asking for a week-long lease
  should find out, not get an hour quietly.
- **Grant response includes the fresh `activeLeases`** (the just-granted
  lease counted), read back via `Store.Counts()` after the acquire.
- Proxies with a deletionTimestamp are filtered out of both list and
  candidate selection — a proxy mid-teardown shouldn't be advertised.

Tests (87.3% coverage, `-race -count=2` clean, green on first run):
httptest over the real handler chain with a fake cache reader and a real
`lease.Store` — auth on/off/wrong-token/healthz-exempt, list filtering
(attributes, healthy, combined, empty-is-200), grant shape (201, default
TTL, lowest-latency pick, RFC3339 expiresAt, activeLeases=1), the full
409 body arithmetic, invalid TTL/body/result, idempotent 204 release,
report→cooldown→409 round-trip, 404 on unknown lease, and a real
`Start` on `127.0.0.1:0` (via the new `BoundAddr()` accessor) serving
healthz then shutting down cleanly on cancel.

```bash
go test -race -count=2 ./internal/discovery/
make test    # whole repo green
```

Worth noting: `go mod tidy` promoted `github.com/go-logr/logr` from
indirect to direct (the server holds a `logr.Logger` field). The
`--discovery-addr`, `--max-lease-ttl` flags and the `DISCOVERY_TOKEN`
Secret mount arrive with `cmd/main.go` in Step 10. `docs/architecture.md`
gained §7 covering the whole HTTP path and the store's sweep Runnable.

## Step 8 — GCP provider (`internal/provider/gcp/`)

Implemented per the plan: `gcp.go` (Provider + the flattened `instancesAPI`
test seam + providerID handling + state mapping), `insert.go` (pure
`buildInsertRequest` + config defaults), `errors.go` (HTTP-code → taxonomy
classification). Only the four calls the spec allows — instances Insert /
Get / Delete / AggregatedList — and `Operation.Wait` is never called:
Create/Delete return once the operation is submitted, `409 alreadyExists`
on Insert and `404` on Delete both count as success, which is what makes
repeat calls after a crash correct.

Dependency added (the plan's environment check pinned it):

```bash
go get cloud.google.com/go/compute@v1.65.0 google.golang.org/api@latest
# resolved google.golang.org/api v0.292.0; go mod tidy pulled the auth/gax chain
```

Key shapes, all straight from the plan:

- **providerID `zones/<zone>/instances/<name>`** — Get/Delete parse the
  zone out of the ID instead of re-reading `spec.placement.zone`, which is
  wrong exactly when a zone edit is the replacement being processed.
- **The seam is not an SDK mirror** — verified the plan's premise against
  the vendored source before designing around it:
  `InstancesScopedListPairIterator` has an unexported `nextFunc`, so a
  fake cannot construct one. The seam flattens `AggregatedList` to
  `[]*computepb.Instance` and returns operations as just their name.
- `AggregatedList` sets `ReturnPartialSuccess: true` (one unreachable
  zone must not fail a GC sweep) and filters by
  `labels.proxy-operator-managed = true`.
- `RUNNING` without a `NatIP` maps to `Provisioning` — never publish an
  empty IP. Unknown/new GCP statuses map to `Stopped`: the reconciler's
  response is delete-and-recreate, always safe for cattle.
- Classification: 404→NotFound; 429 and 403-with-
  `quotaExceeded`/`rateLimitExceeded`→Quota; 400/401/403-other→Permanent;
  everything else (408, 5xx, network, unknown)→Transient.

Judgment call: `placement.zone`/`machineType`/`image` are all required at
`Create` — missing values fail as `ErrPermanent` with a message naming
the empty fields, rather than inventing defaults the spec doesn't define.
A wrong guess here would silently create billable VMs of an arbitrary
shape; a Failed condition telling the user what to set is strictly better.

Tests (75.2%, `-race -count=2` clean, green on first run): the plan's
primary field-by-field `buildInsertRequest` assertion (machine-type URL,
boot disk, the exact `{External NAT, ONE_TO_ONE_NAT}` access config,
user-data metadata, GC labels, network tag) plus config overrides and
no-metadata-without-cloud-init; the full classification table including
`errors.Is` AND `errors.As` through the multi-unwrap; and fake-seam tests
for zone-qualified IDs, 409-is-success, permanent-on-bad-placement (no
API call made), the nine-row state-mapping table, 404 paths, malformed
providerIDs, and the ListByTag filter/partial-success assertions. The
uncovered remainder is `New()` (dials real Google with ADC) and the
`realInstances` adapter — the same deliberately-untested posture as the
kubernetes provider's `New()`.

Worth noting: gopls suggested replacing `proto.String(x)` with Go 1.26's
`new(x)` expression; left as `proto.String` — it is the universal
protobuf-construction idiom and matches every example in the SDK docs.
Registry wiring (`"gcp": gcp.New`) happens at the composition root in
Step 10, as designed in Step 2. `docs/architecture.md` §5 now shows both
providers' call mappings.

## Step 9 — Orphan GC + metrics

Implemented `internal/gc/gc.go` (the `Sweeper` Runnable) and
`internal/metrics/metrics.go` (explicit-registration metric set), plus the
`provider.WithMetrics` decorator deferred from Step 2 into
`internal/provider/metrics.go`, and the observation hooks in the health
engine and discovery server.

**GC**, per the plan: `NeedLeaderElection() = true` (destructive ⇒ single
writer), 10 min ticker with the first sweep one full interval after start,
per-provider `ListByTag` with log-and-continue on provider errors, and a
kill requires all three of: our UID label present, older than `MinAge`
(10 min), and the UID matching no existing CR — where a CR with a
`deletionTimestamp` still counts as live (its finalizer owns that
deletion; GC racing it would double-delete). Two safety behaviors worth
naming: a failed Proxy `List` skips the whole sweep (an unreadable live
set proves nothing orphaned), and the namespace-scope guard makes `Start`
refuse to run against a namespace-restricted cache unless
`--gc-allow-namespaced` is explicit, with the flag named in the error.
One deviation of record: the plan says kills log "at Warn", but logr has
no Warn level — kills log at Info with a `WARNING:` prefix carrying
provider, providerID, and UID, same convention as the discovery server's
empty-token warning.

**Metrics**, per the plan: no `init()` — `Metrics.Register(reg, phases,
leases)` is called explicitly by the composition root (Step 10), which is
also what lets every test use a fresh registry (asserted by a test that
registers two sets on two registries). `proxy_operator_proxies{phase}`
and `proxy_operator_leases_active` are scrape-time collectors fed by
closures — a reconcile-incremented gauge drifts and leaks series on
delete; reading the source of truth at scrape time cannot. The
probe vectors observe **every** probe (status stays transition-only;
metrics carry the high-frequency signal), and the health engine calls
`ForgetProxy` when it prunes a state entry so per-proxy series don't leak.
`provider_requests_total{provider,op,result}` comes from the
`WithMetrics` decorator — the one place `Class()` is called purely for
observability, with results labelled ok / not_found / quota_exceeded /
permanent / transient.

**Decoupling shape:** each consuming package defines its own small
recorder interface (`health.ProbeMetrics`, `discovery.LeaseMetrics`,
`provider.RequestRecorder`); `metrics.Metrics` satisfies all of them
structurally. Only `cmd/main.go` will import `internal/metrics`.

Tests (`-race -count=2` clean): GC's true-orphan matrix in one sweep
(live kept, deleting-CR kept, young kept, unlabelled kept, orphan
deleted), broken-provider isolation, list-failure skips sweep, the
namespace guard both ways, and the Start loop sweeping then stopping;
metrics via `prometheus/testutil` — `GatherAndCompare` on the scrape-time
collectors, ForgetProxy dropping series, outcome/result label counts;
the decorator's five-way classification table with error passthrough
asserted. Coverage: gc 86.1%, metrics 95.0%, provider up to 97.1%;
health/discovery re-ran green with the hooks in place.

```bash
go test -race -count=2 ./internal/gc/ ./internal/metrics/ ./internal/provider/ ./internal/health/ ./internal/discovery/
make test    # whole repo green
```

Worth noting: `prometheus/client_golang` was already in the module via
controller-runtime's metrics server, so no new dependency — `go mod tidy`
just promoted it to direct. `docs/architecture.md` gained §8 (GC sweep)
and §9 (metrics shape).