egress-proxies-operator/docs/plans-executions/2026-08-07-1747-proxy-operator.md

# Execution log: proxy-operator

Pairs with [docs/plans/2026-08-07-1747-proxy-operator.md](../plans/2026-08-07-1747-proxy-operator.md).

## Status

- [x] Step 0 — Branch and scaffold
- [x] Step 1 — API types (`api/v1alpha1/proxy_types.go`)
- [x] Step 2 — Provider contract (`internal/provider/`)
- [x] Step 3 — Mock provider (`internal/provider/mock/`)
- [ ] Step 4 — Reconciler (`internal/controller/`)
- [ ] Step 5 — Health engine (`internal/health/`)
- [ ] Step 6 — Lease store (`internal/lease/`)
- [ ] Step 7 — Discovery API (`internal/discovery/`)
- [ ] Step 8 — GCP provider (`internal/provider/gcp/`)
- [ ] Step 9 — Orphan GC + metrics
- [ ] Step 10 — Wiring, config, docs
- [ ] Step 11 — Tests
- [ ] Verification (vet/test/kind e2e) + commit, push, open MR

## Step 0 — Branch and scaffold

Branched off the unborn `main`:

```bash
git checkout -b feat/proxy-operator
```

Installed kubebuilder v4.15.0 into a scratch `GOBIN` rather than the default
`$(go env GOPATH)/bin`, since the module's package layout changed and
`go install .../cmd/kubebuilder@v4.15.0` (the path from the plan) 404s — the binary is
now the module root itself:

```bash
GOBIN=<scratch>/bin go install sigs.k8s.io/kubebuilder/v4@v4.15.0
```

Scaffolded in place, with `kubebuilder` on `PATH`:

```bash
kubebuilder version
# KubeBuilder: v4.15.0, Kubernetes: 1.36.0

kubebuilder init --domain example.com \
  --repo gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator --plugins go/v4
# WARN: target directory not empty (expected — CLAUDE.md, docs/, .claude/ already existed)

kubebuilder create api --group crawl --version v1alpha1 --kind Proxy \
  --resource --controller
```

`create api` auto-ran `make manifests` at the end, which pulled and ran
`controller-gen` itself:

```bash
sigs.k8s.io/controller-tools/cmd/controller-gen@v0.21.0
"$(bin)/controller-gen" object:headerFile="hack/boilerplate.go.txt",year=2026 paths="./..."
```

Confirmed the CRD group landed correctly (no doubling — `--domain example.com
--group crawl` was used specifically to avoid the `crawl.crawl.example.com` trap
called out in the plan):

```bash
grep -A2 "GroupVersion =" api/v1alpha1/groupversion_info.go
# SchemeGroupVersion = schema.GroupVersion{Group: "crawl.example.com", Version: "v1alpha1"}
```

Dropped the scaffolded GitHub Actions workflows (remote is Gitea, not GitHub):

```bash
git rm -r --cached .github 2>/dev/null; rm -rf .github
```

Ran the full manifest/codegen pass once more to confirm the toolchain is reproducible
end to end:

```bash
make manifests generate
# controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
# controller-gen object:headerFile="hack/boilerplate.go.txt",year=2026 paths="./..."
```

then `go build ./...` and `go vet ./...`, both clean with no output.

Worth noting: `CONTROLLER_TOOLS_VERSION` in the generated `Makefile` came out at
`v0.21.0` by default in this kubebuilder release, so the Makefile edit the plan
anticipated wasn't needed. Pre-existing `CLAUDE.md`/`CHANGELOG.md` content survived
untouched; kubebuilder added its own `README.md`, `AGENTS.md`, `.golangci.yml`,
`.devcontainer/`, `Dockerfile` on top of them — those get edited or left as-is in
later steps. Committed as `076bc66`.

## Step 1 — API types (`api/v1alpha1/proxy_types.go`)

Wrote the full `ProxySpec`/`ProxyStatus`/`Proxy` types per the plan, including the
four corrections called out there (`MaxLeases *int32`, `HealthCheck` with
`+kubebuilder:default={}`, `MinLength=1` on `Provider`/`CloudInit.Inline`,
`Conditions` with `+listType=map`), the 7 CEL `XValidation` rules (6 on `ProxySpec`,
1 on `CloudInitSpec`), and pure helpers in `helpers.go`
(`EffectivePort`/`EffectiveHost`/`HealthCheckOrDefault`/`MaxLeasesOrDefault`) with
table-driven tests in `helpers_test.go`.

Regenerated deepcopy and the CRD:

```bash
make manifests generate
# controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
# controller-gen object:headerFile="hack/boilerplate.go.txt",year=2026 paths="./..."
```

Confirmed all 7 CEL rules and the `healthCheck` `default: {}` block landed in the
generated CRD as expected:

```bash
grep -B1 "rule:" config/crd/bases/crawl.example.com_proxies.yaml
# 7 matches, one per XValidation marker written
```

Ran the full suite, not just `go build`/`go vet`, since this was a good opportunity to
confirm envtest itself works end to end for the first time:

```bash
make test
# Setting up envtest binaries for Kubernetes version 1.36...
# .../bin/k8s/1.36.2-darwin-arm64   (confirms the plan's envtest version note)
```

This failed on the first run — not because of anything in the new types, but because
kubebuilder's scaffolded placeholder test in `proxy_controller_test.go` creates a bare
`Proxy{}` with no `spec.mode`, which our new required/enum field correctly rejects:

```text
Proxy.crawl.example.com "test-resource" is invalid: [spec.mode: Unsupported value: "":
supported values: "Managed", "External", ...]
```

That's a real envtest apiserver enforcing our schema for the first time, which is
useful confirmation on its own. Patched just the resource literal in that scaffold
test to a minimal valid spec (`Mode: External` + `Endpoint.Host`) rather than
rewriting the file — that whole test gets replaced in Step 4 alongside the real
reconciler, so a deeper fix now would be thrown away. `make test` then passed clean:
`api/v1alpha1` at 20.5% coverage (helpers only — CEL itself isn't unit-testable, it's
exercised by the real apiserver as shown above), `internal/controller` at 66.7%.

Worth noting: the `go test ./...` command from the plan's own verification section
does *not* work directly for the envtest suite — it needs `KUBEBUILDER_ASSETS` set,
which only `make test` does via `setup-envtest`. Plain `go test ./...` fails the
`internal/controller` package with a `/usr/local/kubebuilder/bin/etcd: no such file`
error that has nothing to do with the code. Use `make test`, not `go test ./...`,
whenever the controller package is in scope.

## Step 2 — Provider contract (`internal/provider/`)

Wrote the `Provider` interface (`Create`/`Get`/`Delete`/`ListByTag`), `Instance`,
`Placement`, `CreateRequest`, and the GC-contract label constants
(`provider.go`); the error taxonomy with multi-error `Unwrap() []error` so
`errors.Is` and `errors.As` both work off the same wrapped value
(`errors.go`); deterministic instance naming via SHA-256 → base32 → 16
chars (`name.go`); and `--providers-config` YAML parsing with fail-fast
validation (`config.go`).

One deliberate deviation from the plan's file layout: the plan listed
`internal/provider/metrics.go` as part of this step, but the
`provider.WithMetrics` decorator it describes is Step 9's concern (it needs
the Prometheus vectors that don't exist until the metrics package is
built) and nothing in this step depends on it existing yet. Deferred to
Step 9 rather than writing a decorator with nowhere to register its
metrics.

The registry package (`internal/provider/registry/registry.go`) came out
slightly different from the plan's sketch, and better for it: instead of a
package-level `var builtin = map[string]Constructor{"mock": mock.New, "gcp":
gcp.New}` living inside the registry package, `Build` takes the
`map[string]Constructor` as a parameter. This means `registry` has zero
import on `internal/provider/mock` or `internal/provider/gcp` — neither of
which exists yet at this point in the plan (mock is Step 3, gcp is Step 8)
— so the package compiles today instead of only once both are done, and the
explicit wiring lives at the composition root (`cmd/main.go`, Step 10)
rather than being smeared into the registry package itself. Still fully
avoids the import-cycle trap the plan called out.

Ran the full suite:

```bash
go mod tidy   # sigs.k8s.io/yaml (already an indirect dep of the k8s.io toolchain) promoted to direct
go build ./... && go vet ./...
go test -race -v ./internal/provider/...
make test
```

`internal/provider` landed at 96.2% coverage, `internal/provider/registry` at
100%. `make test` also ran `go fmt ./...`, which reformatted `errors.go`'s
struct-field comment alignment before its first commit — no logic change,
just gofmt on a brand-new file.

Worth noting: `sigs.k8s.io/yaml` (not `gopkg.in/yaml.v3`) was picked for
`--providers-config` parsing specifically because it has `UnmarshalStrict`
built in (rejects unknown fields, which is what "fail fast on unknown type"
in the plan actually needs) and was already pulled in transitively by the
k8s.io toolchain, so no new dependency was added — `go mod tidy` just
promoted it from indirect to direct.

## Step 3 — Mock provider (`internal/provider/mock/`)

Started from the plan's design (state as a pure function of an injectable
clock, no background timers) but had to redesign the "real proxy" part
before writing any code, once a load-bearing assumption turned out false.

The plan (and the earlier decision to make the mock run a real proxy)
assumed each instance could get its own loopback address —
`127.0.0.1:<port>` per instance, "a fake IP from a private range." Verified
that assumption directly before committing to it:

```bash
cat <<'EOF' > /tmp/loopbacktest.go
package main
import ("fmt"; "net")
func main() {
	for _, addr := range []string{"127.0.0.2:0", "127.0.0.55:0", "127.1.2.3:0"} {
		l, err := net.Listen("tcp", addr)
		if err != nil { fmt.Printf("%s: FAIL: %v\n", addr, err); continue }
		fmt.Printf("%s: OK\n", addr)
		l.Close()
	}
}
EOF
go run /tmp/loopbacktest.go
# 127.0.0.2:0: FAIL: listen tcp 127.0.0.2:0: bind: can't assign requested address
# 127.0.0.55:0: FAIL: listen tcp 127.0.0.55:0: bind: can't assign requested address
# 127.1.2.3:0: FAIL: listen tcp 127.1.2.3:0: bind: can't assign requested address
```

Only `127.0.0.1` binds on macOS without `sudo ifconfig lo0 alias ... up` —
Linux routes the whole `127.0.0.0/8` block to loopback by default, macOS
doesn't. That's not something the operator can or should do at runtime, so
per-instance loopback IPs were out. There's also a second problem the
per-instance-IP design didn't solve anyway: `EffectivePort()`
(`api/v1alpha1/helpers.go`, Step 1) is computed purely from `spec.port`
with no channel for a provider to report back a different *port* — so
whatever a mock instance actually listens on has to be the literal port the
reconciler will pass through `CreateRequest.Port`, not an OS-assigned
ephemeral one.

Redesigned around one real listener **per port**, shared and
reference-counted across every instance that uses it, rather than one
listener per instance (`proxy.go`, `sharedProxies`). This fixes both
problems at once: every instance binds the same `127.0.0.1` (no OS issue),
and any number of instances can share a port without conflict since it's
the exact same underlying listener. Deliberately made the refcounting
**package-level**, not a field on `mock.Provider`, because a bound TCP port
is a genuinely process-global OS resource — two separately configured
mock-typed provider entries (e.g. two named `"mock"` instances in
`providers-config.yaml`) would otherwise both try to bind the same default
port and the second one would just fail. This is a case where a package
global is the correct model, not a shortcut: it mirrors an OS-level
singleton, not application state.

Instance lifecycle otherwise follows the plan exactly: `Get`/`ListByTag`
derive `Provisioning → Running → Terminated → purged (ErrNotFound)` from
`createdAt`/`deletedAt` compared against an injectable clock, with the real
proxy listener acquired lazily on the first observed `Running` and released
on `Delete` (or lazily on purge, so orphaned records can't leak a
reference). `Create` is idempotent by name. Fault injection is wired both
ways per the plan: `MockConfig.FailNextCreates`/`FailWith` for the demo
config, `InjectCreateFailures(n, class)` for tests.

Tests initially had a real flake, caught by running with `-race -count=3`
rather than trusting one green run:

```bash
go test -race -v ./internal/provider/mock/... 2>&1 | tail -5
# --- FAIL: TestAcquireProxy_sharedAcrossAcquires
#     proxy_test.go:37: port not released after last reference: bind: address already in use
```

Root cause wasn't the refcounting logic — it was the test helper. `freePort`
asked the OS for a free port by binding to `:0` and immediately closing it,
which is a classic TOCTOU race under `t.Parallel()`: two tests can be handed
the same "free" port before either actually claims it, since nothing holds
it open in between. Fixed by replacing the OS-asks approach with a
monotonic counter (`20000 + atomic.Int32`) — these tests only need a port
unique *within this test run*, not one verified free by the OS at an
instant in time, so guaranteeing uniqueness outright is both simpler and
correct where the "ask and hope" approach wasn't. Reran `-race -count=3`
clean afterward.

Added tests beyond the plan's list to close real coverage gaps rather than
stopping at "green": config-override branches in `New`, all four
`failClassFromString` branches, `ListByTag`'s purge-on-list and
Running/IP-inclusion paths, and a plain-`http://` forwarding test
(`handleForward`) alongside the CONNECT one, since a `probeURL` override
could use either scheme. Landed at 91.1% coverage; the remainder is
OS-failure branches (bind errors, hijack failures) not worth simulating for
a prototype.

The `TestProvider_realProxyTunnelsConnect` test is the one that actually
matters most here: it opens real sockets end to end — mock `Create` →
`Get` past `provisionDelay` → real `http.Client` with
`Transport.Proxy` dialing through the mock's CONNECT tunnel to a real
`httptest.NewTLSServer` — and gets a real `204` back. That's the concrete
proof the "mock runs a real proxy" decision actually delivers a genuine
end-to-end healthcheck, not a simulated one.

`internal/provider/mock` at 91.1% coverage. `make test` green across the
whole repo.