egress-proxies-operator/docs/plans/2026-08-07-1747-proxy-operator.md

# Plan: proxy-operator — Kubernetes operator for crawling-proxy VMs
**Created:** 2026-08-07 17:47

## Context

The crawling department runs a small fleet (tens) of HTTP proxy VMs across cloud
providers to dodge rate limiting. This builds a production-quality **prototype**
operator making each proxy VM a first-class Kubernetes object: GitOps-managed,
actively health-checked *through the proxy*, and discoverable by crawler clients via
an HTTP list/lease API.

The repo is empty — `go.mod`, `CLAUDE.md`, `CHANGELOG.md`, `.claude/`, and the spec at
[docs/prompts/__initial-prompt.md](../prompts/__initial-prompt.md). Zero commits.
All greenfield. Governing principle: **proxies are immutable cattle** — any meaningful
spec change deletes and recreates the VM. No in-place update logic.

### Decisions locked with the user

| Question | Decision |
|---|---|
| Module path | Keep `gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator` (spec's `github.com/CHANGEME/...` was a placeholder) |
| API group | `crawl.example.com`, `v1alpha1`, kind `Proxy`, **namespaced** |
| Git flow | Scaffold commit, then work on `feat/proxy-operator`, MR via `tea`, no merge/delete from CLI |
| kubebuilder | `go install sigs.k8s.io/kubebuilder/v4/cmd/kubebuilder@v4.15.0` |
| Local/CI provider | **Kubernetes pod provider**, not an in-memory mock: creates real `ubuntu/squid` pods in-cluster. Revised mid-build — see Step 3 |
| Verification | vet + unit + envtest, then a throwaway kind cluster running the README quickstart, then delete it |

### Verified environment — no version substitutions needed

Go 1.26.4 · kubebuilder v4.15.0 · controller-runtime v0.24.1 · k8s.io/* v0.36.3 ·
controller-tools v0.21.0 · compute v1.65.0 · kind v0.32.0 · kubectl v1.36.1 · docker
29.6.2 · gcloud present. envtest bundles exist for k8s **1.36.0 and 1.36.2** on
darwin/arm64 — the scaffold Makefile derives the *minor* (`1.36`) from `k8s.io/api`
and resolves the latest patch; do not "fix" it to 1.36.3, which has no bundle.

Every pin in the spec is satisfiable. The README will say so explicitly rather than
omitting the substitutions section.

### Milestone order — each ends green on `go build ./... && go vet ./...`

1. Scaffold + pins → 2. `api/v1alpha1` + CEL → 3. `internal/provider` + kubernetes-pod →
4. reconciler + envtest → 5. health engine → 6. lease + discovery → 7. GCP provider →
8. orphan GC + metrics → 9. `cmd/main.go` wiring + `config/` → 10. docs + kind run.

---

## Step 0 — Branch and scaffold

```bash
git checkout -b feat/proxy-operator          # main is unborn; branch starts empty
go install sigs.k8s.io/kubebuilder/v4/cmd/kubebuilder@v4.15.0
kubebuilder init --domain example.com \
  --repo gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator --plugins go/v4
kubebuilder create api --group crawl --version v1alpha1 --kind Proxy --resource --controller
make manifests generate
```

**Trap:** `--domain crawl.example.com --group crawl` yields group
`crawl.crawl.example.com`. It must be `--domain example.com --group crawl`.

If `init` refuses the non-empty directory, scaffold into an empty temp dir with the
identical `--repo` and copy the tree in — don't fight the emptiness check. Copy this
plan to `docs/plans/2026-08-07-1747-proxy-operator.md` per CLAUDE.md. **Commit the
untouched scaffold on its own** so every later diff is reviewable.

Post-scaffold hand-edits: `CONTROLLER_TOOLS_VERSION ?= v0.21.0` in the Makefile (CEL
emission at the 1.36 API level); add a `run-dev` target wired to a sample
`--providers-config` (Step 10); delete the scaffolded `.github/workflows/` (the
remote is Gitea).

---

## Step 1 — API types (`api/v1alpha1/proxy_types.go`)

Structs exactly as the spec dictates, with these **four corrections that are silent
bugs otherwise**:

- **`MaxLeases *int32`**, not `int32`. With a value type + `omitempty` + `default=5`,
  an explicit `0` is dropped on any Go round-trip and re-defaulted to 5 — "0 =
  unleasable" becomes unreachable.
- **`HealthCheck *HealthCheckSpec` needs `+kubebuilder:default={}`.** Structural
  defaulting only descends into values that exist; without it a nil `healthCheck` gets
  *none* of its nested defaults and the spec's "all defaulted" quietly fails.
- `+kubebuilder:validation:MinLength=1` on `Provider` and `CloudInit.Inline`, so an
  explicit `""` fails OpenAPI validation and the CEL `has()` rules stay simple.
- Conditions get `+listType=map +listMapKey=type`.

**CEL at the `ProxySpec` struct level** — cross-field rules cannot live on a field:

```go
// +kubebuilder:validation:XValidation:rule="self.mode == oldSelf.mode",message="mode is immutable"
// +kubebuilder:validation:XValidation:rule="has(self.provider) == has(oldSelf.provider) && (!has(self.provider) || self.provider == oldSelf.provider)",message="provider is immutable"
// +kubebuilder:validation:XValidation:rule="self.mode != 'Managed' || has(self.provider)",message="provider is required when mode is Managed"
// +kubebuilder:validation:XValidation:rule="self.mode != 'External' || !has(self.provider)",message="provider must not be set when mode is External"
// +kubebuilder:validation:XValidation:rule="self.mode != 'External' || has(self.endpoint)",message="endpoint is required when mode is External"
// +kubebuilder:validation:XValidation:rule="self.mode != 'Managed' || !has(self.endpoint)",message="endpoint must not be set when mode is Managed"
```
plus on `CloudInitSpec`:
`rule="has(self.inline) != has(self.secretRef)"` (CEL `!=` on booleans is XOR).

Semantics that matter: a rule mentioning `oldSelf` is **skipped on CREATE**, so
immutability and required-iff must be *separate markers* — `&&`-ing them together
would skip the required-iff check on create. The immutability rule uses the
`has(self.x) == has(oldSelf.x) && ...` form because `Provider` is optional and a
field-level rule wouldn't fire when the field is absent on either side.

Defaults: `port=3128`, `maxLeases=5`, probeURL `https://www.gstatic.com/generate_204`,
interval 30s, timeout 5s, failureThreshold 3, successThreshold 1,
expectedStatusCodes `{200,204}`. Printer columns Mode/Provider/Phase/IP/Healthy/Age,
`shortName=px`, `+kubebuilder:subresource:status`.

Pure helpers in `helpers.go` (unit-tested): `EffectivePort()`, `EffectiveHost()`,
`HealthCheckOrDefault()`, `MaxLeasesOrDefault()`. **Port lives in two places**
(`spec.port` for Managed, `spec.endpoint.port` for External) — one helper, used by
health, discovery, and the hash.

---

## Step 2 — Provider contract (`internal/provider/`)

```
internal/provider/{provider,errors,name,config,metrics}.go
internal/provider/registry/registry.go   # type→constructor — SEPARATE package
internal/provider/{kubernetes,gcp}/
```

**Import-cycle trap:** a registry inside `internal/provider` would have to import
`internal/provider/kubernetes` (or `.../gcp`), both of which import `internal/provider`
for the interface. `init()` self-registration is banned by CLAUDE.md, so the registry
goes in its own leaf-importing package, and its `Build` function takes the
type→constructor map as a parameter instead — see Step 2's execution log entry for why
this ended up better than a package-level map even beyond avoiding the cycle.

`Instance` needs **two fields the spec omits**, or orphan GC is unimplementable:
`UID string` (from the label, for the liveness match) and `CreatedAt time.Time` (for
the "skip < 10 min" rule).

**Error taxonomy** — the key trick is multi-value unwrap:
```go
type Error struct{ Class error; Op, Provider, ID string; Err error }
func (e *Error) Unwrap() []error { return []error{e.Class, e.Err} }
func Class(err error) error  // returns the sentinel; unclassified → ErrTransient
```
So `errors.Is(err, ErrQuotaExceeded)` **and** `errors.As(err, &googleapiErr)` both work
on the same value. `Class()` defaulting to `ErrTransient` matters: retrying is always
safer than latching `Failed`.

**Deterministic naming** — SHA-256, first 10 bytes, RFC 4648 base32 lowercased,
unpadded → 16 chars, `proxy-` + that = 22 total:
```go
func NameFromUID(uid types.UID) string
```
GCP requires `^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$` ≤63. base32 lowercases to `[a-z2-7]`
— all legal; base64's `+`/`/`/uppercase are not, and hex would need 20 chars for the
same 80 bits. 80 bits → birthday collision at ~2^40 objects against a fleet of tens.
22 chars leaves headroom because GCP auto-names the boot disk after the instance.

`config.go` parses the YAML with `yaml.UnmarshalStrict` and validates at load
(non-empty + unique names, known type, type block present, gcp requires `project`).
**Fail fast from `main`** — never degrade.

---

## Step 3 — Kubernetes pod provider (`internal/provider/kubernetes/`)

**Revised after Step 3 was first built as an in-memory mock provider** (state-machine
simulation + a hand-rolled CONNECT proxy on a shared, refcounted local listener). The
user found that too far from the real system to build confidence in, and didn't need
tests to be fast enough to justify the complexity it cost — a real `kind`-cluster
verification pass "once in a while" is an acceptable trade for tests that actually
look like the final product. Full narrative of the reversal is in
`docs/plans-executions/2026-08-07-1747-proxy-operator.md`; this section describes the
replacement, which is what actually gets built for local dev/CI going forward. No
in-memory provider remains in the tree — GCP is now the only other provider, per the
spec's original two-provider scope.

**What it does:** `Create` creates a `corev1.Pod` running a proxy container in the
same cluster (and same namespace as the owning Proxy CR — `req.Namespace`); `Get`
reads the Pod's phase/IP; `Delete` deletes it (tolerating NotFound); `ListByTag` lists
Pods by the standard `LabelManaged`/`LabelUID` labels, unscoped by namespace (the
operator's RBAC needs cluster-scoped Pod permissions — see RBAC note below).

**Proxy software: `ubuntu/squid`** (Canonical's actively maintained LTS image on
Docker Hub, verified before picking it — 50M+ pulls, updated the same day this
decision was made), not a hand-rolled proxy. It's a public image, so `kind` nodes
pull it directly; no build/load step needed for the quickstart. Squid's config
(`http_port <req.Port>`, permissive ACL) is generated in Go and injected via an env
var the container's command writes to `/etc/squid/squid.conf` before exec'ing squid —
no separate ConfigMap object, so there's still only one Kubernetes object per proxy
instance to create, track, and clean up.

**providerID format: `<namespace>/<podName>`** (parseable with
`k8s.io/client-go/tools/cache.SplitMetaNamespaceKey`), so `Get`/`Delete` are
self-contained without needing to re-derive the namespace — the same reasoning as the
GCP provider's zone-qualified providerID in Step 8.

**Pod naming:** reuses `provider.NameFromUID` unchanged — the same deterministic name
satisfies Kubernetes Pod naming rules (`^[a-z0-9]([-a-z0-9]*[a-z0-9])?$`, ≤253 chars)
with room to spare.

**State mapping:** Pod phase `Pending`, or `Running` with no PodIP yet → Provisioning
(never publish an empty IP); `Running` with a PodIP → Running; `Succeeded`/`Failed`/
`Unknown` → Terminated (the reconciler treats Stopped and Terminated identically —
delete and recreate, cattle not pets — so collapsing three failure-ish phases into one
is enough).

**Client:** built internally via `ctrl.GetConfig()` (auto-detects in-cluster config,
falls back to the local kubeconfig otherwise), not threaded through the registry
`Constructor` signature — this is what makes `make run` against a local `kind`
cluster and running in-cluster use the exact same code path with no provider-specific
wiring in `cmd/main.go`.

**Testing, given envtest can't schedule real Pods** (no kubelet — a Pod created
against `envtest`'s API server just sits `Pending` forever): `internal/provider/kubernetes`
itself is unit-tested against `sigs.k8s.io/controller-runtime/pkg/client/fake` — real
Pod objects, real client interface, fully exercises Create/Get/Delete/ListByTag
logic and the Pod-construction function in isolation, just without a kubelet actually
starting a container. Real end-to-end proof (does a probe actually tunnel through a
real Squid pod) only happens against a real `kind` cluster, in the Verification
section — which is exactly what the user asked for. Step 4's reconciler tests use a
small `Provider`-interface stub defined directly in the controller test file (a
handful of lines, not a package) for exercising the state-machine's branching logic —
categorically simpler than what mock.Provider was, since it has no config format, no
fault-injection surface, and exists only inside test code.

---

## Step 4 — Reconciler (`internal/controller/`)

A **state machine**: every reconcile derives one action from (spec, status, provider
`Get`). Intervals are struct fields, never consts, so envtest can shrink them to
milliseconds.

```go
func (r *ProxyReconciler) Reconcile(ctx, req) (res ctrl.Result, err error) {
    // Get; base := p.DeepCopy()
    // defer patchStatusIfChanged(ctx, base, &p)   // one status write per reconcile, max
    switch {
    case !p.DeletionTimestamp.IsZero():        return r.reconcileDelete(ctx, &p)
    case p.Spec.Mode == v1alpha1.ModeExternal: return r.reconcileExternal(ctx, &p)
    default:                                   return r.reconcileManaged(ctx, &p)
    }
}
```
`patchStatusIfChanged` sets `observedGeneration` and `phase = computePhase(&p)`, then
issues nothing when `equality.Semantic.DeepEqual(base.Status, p.Status)`. `computePhase`
is pure and is the primary table-driven unit-test target.

### Action table (Managed) — `A` = hash annotation, `H` = computed hash

| deletionTS | ID | A vs H | provider.Get | action | result |
|---|---|---|---|---|---|
| no | — | — | — | finalizer absent → add it | `{}` (the Update re-triggers) |
| no | `""` | any | — | resolve cloud-init, `Create`, set ID + `A=H` | `RequeueAfter: ProvisioningPoll` |
| no | set | `==` | Provisioning | clear `ip`, Provisioned=False/Provisioning | `RequeueAfter: ProvisioningPoll` |
| no | set | `==` | Running | set `ip`, Provisioned=True/Created | `RequeueAfter: DriftPoll` |
| no | set | `==` | Stopped/Terminated | `Delete(ID)` — cattle, not pets | `RequeueAfter: DeletionPoll` |
| no | set | `==` | NotFound | clear ID+ip → next pass creates | `Requeue: true` |
| no | set | `!=`, `A != ""` | any | **replace**: `Delete(ID)`, Provisioned=False/Replacing | `RequeueAfter: DeletionPoll` |
| no | set | `!=`, `A == ""` | any | **adopt**: set `A=H`, no replacement | `Requeue: true` |
| no | set | `!=` | NotFound | clear ID+ip (status), then set `A=H` (metadata) | `Requeue: true` |
| yes | `""` | — | — | remove finalizer (orphan GC reaps any stray VM) | `{}` |
| yes | set | — | NotFound | remove finalizer | `{}` |
| yes | set | — | anything else | `Delete(ID)`, phase=Deleting | `RequeueAfter: DeletionPoll` |

External: no finalizer, no provider calls, `status.ip = spec.endpoint.host`,
Provisioned=True/ExternalEndpoint, health drives the rest.

**The trap the spec doesn't mention:** the instance name derives from the *CR UID*,
which does **not** change on a spec edit — so the replacement VM has the *same* name
as the one being deleted. Recreating immediately hits `409 alreadyExists` against a
still-deleting instance. Hence replacement **polls to `NotFound` before recreating**
(rows 7 → 9 → 2). Do not add the hash to the instance name; that breaks the spec's
naming contract and only buys blue/green, a non-goal.

**Crash-safety:** ID is never lost destructively. Even if status is wiped entirely, the
create branch calls `Create`, which finds the existing VM by deterministic name and
returns its ID. That's what makes deterministic naming load-bearing rather than
cosmetic.

**Adopt-on-empty-annotation is required** — otherwise the first deploy of an operator
version whose hash-input struct gained a field mass-replaces the whole fleet.

**Spec hash:** SHA-256 over canonical JSON of an explicit `{placement, cloudInit
(resolved content), port}` struct — explicit, not `ProxySpec` wholesale, to bound
upgrade churn. Because it covers *resolved* Secret content, rotating the Secret must
re-trigger: `Watches(&corev1.Secret{}, EnqueueRequestsFromMapFunc(proxiesForSecret))`
with the cache restricted to Secrets labelled `crawl.example.com/cloud-init=true`.
~30 lines, and it's the difference between immutable replacement working and being
silently stale.

**Requeue per class:** `ErrTransient`/unclassified → return the error (workqueue
backoff). `ErrQuotaExceeded` → condition + `RequeueAfter: 5m`, return **nil** (keeps it
off the backoff curve and out of the error log). `ErrPermanent` → phase `Failed`,
condition, return nil. `ErrNotFound` → never an error, a state-machine input.

Conditions via `apimeta.SetStatusCondition` — note it does **not** populate
`ObservedGeneration`, so pass it explicitly or every condition reports generation 0.
`MaxConcurrentReconciles: 3`.

---

## Step 5 — Health engine (`internal/health/`)

**Delivery: push transitions to the reconciler via channel + `source.Channel` (option
a).** Rationale for `docs/architecture.md` → Decisions: `status.phase` is derived from
*both* provisioning and health. Under direct-patch, two writers each compute `phase`
from half the picture and race on the same subresource — a lost-update/flapping bug.
Option (a) keeps exactly one writer of `.status`, makes "write only on transition" fall
out for free (the engine only *emits* on transition), and costs one channel plus a
read-only `Snapshot()` method. The engine owns health *state*; the reconciler owns
health *representation*.

Channel typed `event.TypedGenericEvent[client.Object]` so the untyped
`&handler.EnqueueRequestForObject{}` satisfies it. Non-blocking send with `default:` —
a wedged reconciler must never stall the probe loop; on drop, don't advance the
"reported" markers, so the next probe retries.

**Threading:** one scheduler goroutine on a 1 s ticker + a fixed pool of 8 workers fed
by a buffered channel. Each tick lists from the cache and enqueues proxies whose
`nextDue <= now` and that aren't in flight. At tens of proxies a per-second list-scan is
free and a timer wheel is unjustified complexity. Startup jitter seeds
`nextDue = now + rand(0, interval)` so a restart doesn't fire every probe at once.

**Probe client** — fresh transport per probe, `defer CloseIdleConnections()`:
```go
Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), DisableKeepAlives: true,
  ForceAttemptHTTP2: false, TLSHandshakeTimeout: timeout,
  ResponseHeaderTimeout: timeout, DialContext: (&net.Dialer{Timeout: timeout}).DialContext }
```
`DisableKeepAlives: true` is **load-bearing** — otherwise `net/http` caches the
established CONNECT tunnel and later probes never re-exercise CONNECT, which is exactly
the failure the spec wants caught.

**CONNECT semantics:** for the default `https://` probe URL the transport sends
`CONNECT host:443` then TLS-handshakes through the tunnel. A proxy that accepts TCP but
can't egress returns non-200 to CONNECT, and `client.Do` returns an **error**, not a
response. So the success predicate is `err == nil && slices.Contains(expected,
resp.StatusCode)` — both halves. Latency is wall time around `Do`, last-value.

**Threshold state:** `map[NamespacedName]*state` under a mutex. Entries are pruned each
tick against the cache list (no leak), and `state.uid` is compared to the CR's UID so a
delete+recreate of the same name doesn't inherit stale fail counters. On leader
handover, state is empty: seed `healthy` from the CR's existing `Healthy` condition so a
healthy proxy doesn't flap to Unknown, but leave counters at zero so a real transition
still needs a full `failureThreshold` run. Documented as a Decision.

**Suppression:** emit only on (a) first-ever result, (b) a threshold-crossing flip, or
(c) `|new − reported| > max(20ms, 0.5×reported)` **and** `now − lastReported > 60s`.
The spec's bare ">50% latency bucket change" is undefined at 0 and makes a proxy
jittering 40↔61 ms write status forever; the absolute floor plus rate limit is what
actually delivers "no unbounded status churn". Flagged in Decisions. Metrics are
observed on *every* probe — that's the right home for high-frequency signal.

Note for the README: transition-only writes mean `status.lastHealthCheckTime` is stale
by construction. It means "time of the last *status-affecting* probe"; true probe
recency lives in metrics.

---

## Step 6 — Lease store (`internal/lease/`)

**`Acquire` takes the candidate *set*, not a chosen proxy** — selection and insertion
must happen under one lock, or two concurrent requests both see "3 of 5 used" and
overcommit.

```go
Acquire(ctx, AcquireRequest{Candidates []Candidate; Target string; TTL time.Duration}) (*Lease, AcquireStats, error)
Release / Report / ActiveCount / Counts / ExpireLoop
```

One `sync.Mutex` for the whole store (tens of proxies, human-rate QPS; sharding is
premature), injectable clock, `byID` + `byProxy` + `cooldown[{proxy,target}]` maps.
Selection is a linear scan + `slices.SortFunc` on `(activeLeases asc, latency asc,
name asc)` — explicitly not a heap, and the third key makes it deterministic and
testable. `AcquireStats{Considered, AtCapacity, InCooldown}` feeds the 409 body.

**Expired-lease retention:** entries stay marked `expired` for the cooldown window
after TTL. `Acquire`/`ActiveCount` ignore them; `Report` still resolves them. Without
this, a report arriving just after the TTL lapses is silently dropped — exactly when a
proxy is being rate-limited, which is when the cooldown matters most.

---

## Step 7 — Discovery API (`internal/discovery/`)

**`NeedLeaderElection() = false`**, and ship `replicas: 1`. Verified in
controller-runtime's runnable ordering: caches start and sync *before* non-leader-
election runnables, so cache reads are safe. If it *were* leader-elected, non-leader
pods would refuse connections while still being Service endpoints. The 1-replica
constraint comes from lease state being per-process, which the spec already accepts —
both facts go in the README caveats.

stdlib `http.ServeMux` using its method+wildcard patterns (`"GET /path"`,
`"/{id}"` + `r.PathValue`) — a stdlib feature available since Go 1.22, used
here so no third-party router is needed; the project itself stays on the
pinned Go 1.26:
`GET /v1/proxies`, `POST /v1/leases`, `DELETE /v1/leases/{id}`,
`POST /v1/leases/{id}/report`, plus unauthenticated `GET /healthz`.
Middleware outermost-first: recover → request-log → `MaxBytesReader(64KiB)` → bearer
auth. Empty `DISCOVERY_TOKEN` passes through **with a loud startup Warn** — in-cluster
that's a silent security hole otherwise. Token compared with
`subtle.ConstantTimeCompare`.

Shapes: list returns `{"proxies":[…],"count":N}`, empty is 200 not 404; lease grant is
201 `{leaseID, proxy, expiresAt, ttlSeconds}`; no match is 409
`{"error":"no_match","message":…,"considered":7,"atCapacity":2,"inCooldown":2,"unhealthy":3}`;
DELETE is always 204; report is 204, 400 on an unknown result value, 404 on a genuinely
unknown lease. All errors share `{"error":"<machine_code>","message":"<human>"}`.
Timeouts on `http.Server`, graceful `Shutdown` with 10 s grace on ctx cancel.

---

## Step 8 — GCP provider (`internal/provider/gcp/`)

Only `compute.NewInstancesRESTClient` (ADC), only Insert/Get/Delete/AggregatedList.
`Operation.Wait` is **never called** — `Create` returns as soon as the operation is
submitted, and `409 alreadyExists` is treated as success, which is what makes a repeat
call after a crash correct.

**providerID = `zones/<zone>/instances/<name>`** — zone-qualified so `Get`/`Delete` are
self-contained. The spec's `Get(ctx, providerID)` carries no zone, and re-reading
`spec.placement.zone` is wrong precisely when a zone edit is the replacement being
processed.

**Test seam is deliberately not an SDK mirror.** `compute.InstancesScopedListPairIterator`
has an unexported `nextFunc`, so a fake cannot construct one — the interface flattens
`AggregatedList` to a slice and returns operations as just their name. The primary unit
test needs no fake at all: `buildInsertRequest` is pure, asserted field-by-field
(machine-type URL, boot disk, `AccessConfigs[0] = {Name:"External NAT",
Type:"ONE_TO_ONE_NAT"}`, `Metadata.Items[user-data]`, GC labels, network tag).

`AggregatedList` needs `ReturnPartialSuccess: true` — otherwise one unreachable zone
fails the entire GC sweep. `RUNNING` **without** a NatIP maps to `Provisioning`, not
Running, so we never publish an empty IP. Error mapping: 404→NotFound;
429 / 403+`quotaExceeded`→Quota; 400/401/403-other→Permanent; 5xx/408/net→Transient;
unknown→Transient.

---

## Step 9 — Orphan GC + metrics

**GC** (`internal/gc/`) — `NeedLeaderElection() = true` (destructive, single-writer),
10 min ticker, first sweep one interval after start. Per provider `ListByTag`; on error
log and continue to the next provider, never abort the sweep. Kill an instance only if
it has our UID label, is older than `MinAge` (10 min), and its UID matches no CR.
Log every kill at Warn with provider, providerID, UID.

**A CR with a `deletionTimestamp` still counts as live** — that's what the spec's
"check deletionTimestamp semantics carefully" points at. Its finalizer owns the
deletion; GC racing it double-deletes. A UID is orphan-eligible only once the object is
fully gone.

**Namespace-scope guard:** if the cache is namespace-restricted but Proxies exist
elsewhere, GC would delete live VMs. It refuses to start unless an explicit
`--gc-allow-namespaced` flag is set.

**Metrics** (`internal/metrics/`) registered by an explicit `Register(...)` called from
`main` (no `init()`, per house rules; also lets tests use a fresh registry).
`proxy_operator_proxies{phase}` and `proxy_operator_leases_active` are **custom
Collectors** that read at scrape time — a reconcile-incremented gauge inevitably drifts
and leaks a series on delete. Per-proxy histogram/counter labels **must** be deleted
from the vec when the health engine GCs a state entry, or series leak forever.
`proxy_operator_provider_requests_total{provider,op,result}` comes from a
`provider.WithMetrics(name, p)` decorator — zero-cost instrumentation for the next five
providers, and the one place `Class()` is called for observability.

---

## Step 10 — Wiring, config, docs

`cmd/main.go`: flags `--providers-config` (required), `--discovery-addr`,
`--proxy-namespace`, `--health-workers`, `--gc-interval`, `--gc-min-age`,
`--lease-cooldown`, `--max-lease-ttl`. Order: load provider config (fail fast) → build
registry → manager → `mgr.Add` health engine, GC, lease expiry loop, discovery server →
`SetupWithManager`. Contexts from `ctrl.SetupSignalHandler()` throughout.

RBAC markers: proxies CRUD + status + finalizers, secrets get/list/watch, events
create/patch, and (for the kubernetes-pod provider) pods get/list/watch/create/delete
— cluster-scoped, since `ListByTag` enumerates across namespaces. `config/`: providers
ConfigMap mount, `DISCOVERY_TOKEN` from a Secret, containerPort 8090 + Service.
Samples: `proxy_kubernetes.yaml`, `proxy_gcp.yaml`, `proxy_external.yaml`,
`providers-config.yaml`.

`docs/architecture.md`: components table, ASCII data-flow diagram, and a **Decisions**
section covering the channel-vs-patch choice, replacement-polls-to-NotFound, base32
naming, discovery without leader election, single-mutex lease store, the mock→
kubernetes-pod-provider revision (why, and why `ubuntu/squid` over a hand-rolled
proxy), leader-handover health seeding, the latency-suppression refinement, the
`lastHealthCheckTime` semantics, and the logr-not-slog deviation (CLAUDE.md says
`slog`, but `log.FromContext(ctx)` returns logr inside controller paths — noted, not
silently ignored).

`README.md`: 60-second architecture summary; copy-pasteable kind quickstart; GCP setup
(ADC, `roles/compute.instanceAdmin.v1`, plus `roles/iam.serviceAccountUser` if
attaching a service account); the **immutable-replacement caveat** (changing a proxy
changes its IP) and **lease-loss-on-restart caveat**, both prominent; a version-pins
note confirming no substitutions were needed. Then a `CHANGELOG.md` entry with a real
`date "+%Y-%m-%d %H:%M %Z"` timestamp.

---

## Step 11 — Tests

**envtest** (`internal/controller/`), a small in-test stub `Provider` (not the real
kubernetes-pod provider — envtest has no kubelet, so a real Pod never leaves Pending)
plus a fake `HealthSnapshotter`, intervals shrunk to 50–200 ms, whole suite behind
`testing.Short()`: Managed→Ready; spec change → old stub instance gone and
providerID changed; delete → finalizer runs and instance removed; External → Ready
on first health pass, no finalizer; injected quota → `Provisioned=False/QuotaExceeded`
and phase *not* Failed; permanent error → Failed and no further provider calls; adopt
(strip annotation → restored, providerID unchanged); and the **CEL cases only a real
API server can test** — mode/provider mutation rejected, Managed-without-provider,
External-without-endpoint, cloudInit both/neither, and `healthCheck` omitted → nested
defaults materialized (the `default={}` assertion).

**Action-table unit tests** — the highest-value tests in the repo: `fake` client with
`WithStatusSubresource`, calling `Reconcile` directly, table-driven over every row of
the Step 4 table, asserting the returned `ctrl.Result`. (Documented caveat: the fake
client runs neither CEL nor defaulting — that's what the envtest CEL cases cover.)

**Units:** name derivation (idempotency, `^proxy-[a-z2-7]{16}$`, 10k-UID distinctness);
`Class()` mapping + `errors.Is`/`errors.As` through the multi-unwrap; config loading;
kubernetes-pod provider Create/Get/Delete/ListByTag against
`sigs.k8s.io/controller-runtime/pkg/client/fake` (real Pod objects, real client
interface, no kubelet needed for this level) plus pure tests of the generated Squid
config and Pod spec; `buildInsertRequest` field-by-field + GCP error classification +
RUNNING-without-IP; `computePhase` truth table; `SpecHash` stability *and*
sensitivity; lease store (capacity, `MaxLeases=0`, least-loaded with latency
tie-break, cooldown with/without target, report on an expired-but-retained lease,
concurrent acquire under `-race` never exceeding `MaxLeases`); discovery handlers
over `httptest` + fake reader + real store; health thresholds against a real
CONNECT-capable `httptest` proxy stub.

Everything runs with `-race`.

---

## Verification

```bash
go vet ./... && make test && make build      # unit + envtest, -race

kind create cluster --name proxy-operator-demo
make install
make run-dev &                                # --providers-config hack/providers-dev.yaml
kubectl apply -f config/samples/proxy_kubernetes.yaml
kubectl get px -w                            # expect Ready with an IP (a real squid Pod)
curl -s 'localhost:8090/v1/proxies?healthy=true' | jq
curl -s -XPOST localhost:8090/v1/leases -d '{"selector":{"geo":"eu"},"ttlSeconds":300}' | jq
curl -s -XPOST localhost:8090/v1/leases/<id>/report -d '{"result":"rate_limited","target":"example.com"}'
curl -si -XDELETE localhost:8090/v1/leases/<id>   # 204, and 204 again
kubectl delete -f config/samples/proxy_kubernetes.yaml  # finalizer runs, Pod is deleted
kind delete cluster --name proxy-operator-demo
```

Success bar: a competent SRE clones the repo, follows the README, and holds a lease on a
healthy proxy — a real Squid pod running in their own `kind` cluster — in under 10
minutes.

Then commit on `feat/proxy-operator`, push with `-u`, open the MR with
`tea pr create --base main --head feat/proxy-operator`, print the URL. No merging or
branch deletion from the CLI.