# Plan: proxy-operator — Kubernetes operator for crawling-proxy VMs
**Created:** 2026-08-07 17:47

## Context

The crawling department runs a small fleet (tens) of HTTP proxy VMs across cloud
providers to dodge rate limiting. This builds a production-quality **prototype**
operator making each proxy VM a first-class Kubernetes object: GitOps-managed,
actively health-checked *through the proxy*, and discoverable by crawler clients via
an HTTP list/lease API.

The repo is empty — `go.mod`, `CLAUDE.md`, `CHANGELOG.md`, `.claude/`, and the spec at
[docs/prompts/__initial-prompt.md](../prompts/__initial-prompt.md). Zero commits.
All greenfield. Governing principle: **proxies are immutable cattle** — any meaningful
spec change deletes and recreates the VM. No in-place update logic.

### Decisions locked with the user

| Question | Decision |
|---|---|
| Module path | Keep `gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator` (spec's `github.com/CHANGEME/...` was a placeholder) |
| API group | `crawl.example.com`, `v1alpha1`, kind `Proxy`, **namespaced** |
| Git flow | Scaffold commit, then work on `feat/proxy-operator`, MR via `tea`, no merge/delete from CLI |
| kubebuilder | `go install sigs.k8s.io/kubebuilder/v4/cmd/kubebuilder@v4.15.0` |
| Mock health | Mock provider runs a **real in-process HTTP CONNECT proxy** per instance, so healthchecks genuinely pass and the kind demo is truly end-to-end |
| Verification | vet + unit + envtest, then a throwaway kind cluster running the README quickstart, then delete it |

### Verified environment — no version substitutions needed

Go 1.26.4 · kubebuilder v4.15.0 · controller-runtime v0.24.1 · k8s.io/* v0.36.3 ·
controller-tools v0.21.0 · compute v1.65.0 · kind v0.32.0 · kubectl v1.36.1 · docker
29.6.2 · gcloud present. envtest bundles exist for k8s **1.36.0 and 1.36.2** on
darwin/arm64 — the scaffold Makefile derives the *minor* (`1.36`) from `k8s.io/api`
and resolves the latest patch; do not "fix" it to 1.36.3, which has no bundle.

Every pin in the spec is satisfiable. The README will say so explicitly rather than
omitting the substitutions section.

### Milestone order — each ends green on `go build ./... && go vet ./...`

1. Scaffold + pins → 2. `api/v1alpha1` + CEL → 3. `internal/provider` + mock →
4. reconciler + envtest → 5. health engine → 6. lease + discovery → 7. GCP provider →
8. orphan GC + metrics → 9. `cmd/main.go` wiring + `config/` → 10. docs + kind run.

---

## Step 0 — Branch and scaffold

```bash
git checkout -b feat/proxy-operator          # main is unborn; branch starts empty
go install sigs.k8s.io/kubebuilder/v4/cmd/kubebuilder@v4.15.0
kubebuilder init --domain example.com \
  --repo gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator --plugins go/v4
kubebuilder create api --group crawl --version v1alpha1 --kind Proxy --resource --controller
make manifests generate
```

**Trap:** `--domain crawl.example.com --group crawl` yields group
`crawl.crawl.example.com`. It must be `--domain example.com --group crawl`.

If `init` refuses the non-empty directory, scaffold into an empty temp dir with the
identical `--repo` and copy the tree in — don't fight the emptiness check. Copy this
plan to `docs/plans/2026-08-07-1747-proxy-operator.md` per CLAUDE.md. **Commit the
untouched scaffold on its own** so every later diff is reviewable.

Post-scaffold hand-edits: `CONTROLLER_TOOLS_VERSION ?= v0.21.0` in the Makefile (CEL
emission at the 1.36 API level); add a `run-mock` target; delete the scaffolded
`.github/workflows/` (the remote is Gitea).

---

## Step 1 — API types (`api/v1alpha1/proxy_types.go`)

Structs exactly as the spec dictates, with these **four corrections that are silent
bugs otherwise**:

- **`MaxLeases *int32`**, not `int32`. With a value type + `omitempty` + `default=5`,
  an explicit `0` is dropped on any Go round-trip and re-defaulted to 5 — "0 =
  unleasable" becomes unreachable.
- **`HealthCheck *HealthCheckSpec` needs `+kubebuilder:default={}`.** Structural
  defaulting only descends into values that exist; without it a nil `healthCheck` gets
  *none* of its nested defaults and the spec's "all defaulted" quietly fails.
- `+kubebuilder:validation:MinLength=1` on `Provider` and `CloudInit.Inline`, so an
  explicit `""` fails OpenAPI validation and the CEL `has()` rules stay simple.
- Conditions get `+listType=map +listMapKey=type`.

**CEL at the `ProxySpec` struct level** — cross-field rules cannot live on a field:

```go
// +kubebuilder:validation:XValidation:rule="self.mode == oldSelf.mode",message="mode is immutable"
// +kubebuilder:validation:XValidation:rule="has(self.provider) == has(oldSelf.provider) && (!has(self.provider) || self.provider == oldSelf.provider)",message="provider is immutable"
// +kubebuilder:validation:XValidation:rule="self.mode != 'Managed' || has(self.provider)",message="provider is required when mode is Managed"
// +kubebuilder:validation:XValidation:rule="self.mode != 'External' || !has(self.provider)",message="provider must not be set when mode is External"
// +kubebuilder:validation:XValidation:rule="self.mode != 'External' || has(self.endpoint)",message="endpoint is required when mode is External"
// +kubebuilder:validation:XValidation:rule="self.mode != 'Managed' || !has(self.endpoint)",message="endpoint must not be set when mode is Managed"
```
plus on `CloudInitSpec`:
`rule="has(self.inline) != has(self.secretRef)"` (CEL `!=` on booleans is XOR).

Semantics that matter: a rule mentioning `oldSelf` is **skipped on CREATE**, so
immutability and required-iff must be *separate markers* — `&&`-ing them together
would skip the required-iff check on create. The immutability rule uses the
`has(self.x) == has(oldSelf.x) && ...` form because `Provider` is optional and a
field-level rule wouldn't fire when the field is absent on either side.

Defaults: `port=3128`, `maxLeases=5`, probeURL `https://www.gstatic.com/generate_204`,
interval 30s, timeout 5s, failureThreshold 3, successThreshold 1,
expectedStatusCodes `{200,204}`. Printer columns Mode/Provider/Phase/IP/Healthy/Age,
`shortName=px`, `+kubebuilder:subresource:status`.

Pure helpers in `helpers.go` (unit-tested): `EffectivePort()`, `EffectiveHost()`,
`HealthCheckOrDefault()`, `MaxLeasesOrDefault()`. **Port lives in two places**
(`spec.port` for Managed, `spec.endpoint.port` for External) — one helper, used by
health, discovery, and the hash.

---

## Step 2 — Provider contract (`internal/provider/`)

```
internal/provider/{provider,errors,name,config,metrics}.go
internal/provider/registry/registry.go   # type→constructor — SEPARATE package
internal/provider/{mock,gcp}/
```

**Import-cycle trap:** a registry inside `internal/provider` would have to import
`internal/provider/mock`, which imports `internal/provider`. `init()` self-registration
is banned by CLAUDE.md, so the registry goes in its own leaf-importing package.

`Instance` needs **two fields the spec omits**, or orphan GC is unimplementable:
`UID string` (from the label, for the liveness match) and `CreatedAt time.Time` (for
the "skip < 10 min" rule).

**Error taxonomy** — the key trick is multi-value unwrap:
```go
type Error struct{ Class error; Op, Provider, ID string; Err error }
func (e *Error) Unwrap() []error { return []error{e.Class, e.Err} }
func Class(err error) error  // returns the sentinel; unclassified → ErrTransient
```
So `errors.Is(err, ErrQuotaExceeded)` **and** `errors.As(err, &googleapiErr)` both work
on the same value. `Class()` defaulting to `ErrTransient` matters: retrying is always
safer than latching `Failed`.

**Deterministic naming** — SHA-256, first 10 bytes, RFC 4648 base32 lowercased,
unpadded → 16 chars, `proxy-` + that = 22 total:
```go
func NameFromUID(uid types.UID) string
```
GCP requires `^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$` ≤63. base32 lowercases to `[a-z2-7]`
— all legal; base64's `+`/`/`/uppercase are not, and hex would need 20 chars for the
same 80 bits. 80 bits → birthday collision at ~2^40 objects against a fleet of tens.
22 chars leaves headroom because GCP auto-names the boot disk after the instance.

`config.go` parses the YAML with `yaml.UnmarshalStrict` and validates at load
(non-empty + unique names, known type, type block present, gcp requires `project`).
**Fail fast from `main`** — never degrade.

---

## Step 3 — Mock provider (`internal/provider/mock/`)

**State is a pure function of an injectable clock — no background timers.** `Get`/
`ListByTag` derive state from `createdAt`/`deletedAt` vs `now()`:
`< provisionDelay` → Provisioning; else Running; `deletedAt` set and `< deleteDelay` →
Terminated; beyond that → purged, `ErrNotFound`. Deterministic under a fake clock,
correct under the real one, and no goroutine lifecycle to leak.

**Real proxy listener (the user's decision):** when a record first reports `Running`,
lazily start an `http.Server` on `127.0.0.1:0` implementing HTTP `CONNECT` tunnelling
(hijack + bidirectional `io.Copy`) plus plain-HTTP forwarding, and report `127.0.0.1`
plus the real listener port. `Delete` shuts it down. This is what makes the health
engine's probe a genuine CONNECT through a real proxy, so the kind quickstart actually
reaches Ready and leasable. One `http.Server` per instance, bounded by fleet size.

Fault injection: config-driven `failNextCreates`/`failWith` for the demo, plus
`InjectCreateFailures(n int, class error)` for tests. `Create` is idempotent by name.

---

## Step 4 — Reconciler (`internal/controller/`)

A **state machine**: every reconcile derives one action from (spec, status, provider
`Get`). Intervals are struct fields, never consts, so envtest can shrink them to
milliseconds.

```go
func (r *ProxyReconciler) Reconcile(ctx, req) (res ctrl.Result, err error) {
    // Get; base := p.DeepCopy()
    // defer patchStatusIfChanged(ctx, base, &p)   // one status write per reconcile, max
    switch {
    case !p.DeletionTimestamp.IsZero():        return r.reconcileDelete(ctx, &p)
    case p.Spec.Mode == v1alpha1.ModeExternal: return r.reconcileExternal(ctx, &p)
    default:                                   return r.reconcileManaged(ctx, &p)
    }
}
```
`patchStatusIfChanged` sets `observedGeneration` and `phase = computePhase(&p)`, then
issues nothing when `equality.Semantic.DeepEqual(base.Status, p.Status)`. `computePhase`
is pure and is the primary table-driven unit-test target.

### Action table (Managed) — `A` = hash annotation, `H` = computed hash

| deletionTS | ID | A vs H | provider.Get | action | result |
|---|---|---|---|---|---|
| no | — | — | — | finalizer absent → add it | `{}` (the Update re-triggers) |
| no | `""` | any | — | resolve cloud-init, `Create`, set ID + `A=H` | `RequeueAfter: ProvisioningPoll` |
| no | set | `==` | Provisioning | clear `ip`, Provisioned=False/Provisioning | `RequeueAfter: ProvisioningPoll` |
| no | set | `==` | Running | set `ip`, Provisioned=True/Created | `RequeueAfter: DriftPoll` |
| no | set | `==` | Stopped/Terminated | `Delete(ID)` — cattle, not pets | `RequeueAfter: DeletionPoll` |
| no | set | `==` | NotFound | clear ID+ip → next pass creates | `Requeue: true` |
| no | set | `!=`, `A != ""` | any | **replace**: `Delete(ID)`, Provisioned=False/Replacing | `RequeueAfter: DeletionPoll` |
| no | set | `!=`, `A == ""` | any | **adopt**: set `A=H`, no replacement | `Requeue: true` |
| no | set | `!=` | NotFound | clear ID+ip (status), then set `A=H` (metadata) | `Requeue: true` |
| yes | `""` | — | — | remove finalizer (orphan GC reaps any stray VM) | `{}` |
| yes | set | — | NotFound | remove finalizer | `{}` |
| yes | set | — | anything else | `Delete(ID)`, phase=Deleting | `RequeueAfter: DeletionPoll` |

External: no finalizer, no provider calls, `status.ip = spec.endpoint.host`,
Provisioned=True/ExternalEndpoint, health drives the rest.

**The trap the spec doesn't mention:** the instance name derives from the *CR UID*,
which does **not** change on a spec edit — so the replacement VM has the *same* name
as the one being deleted. Recreating immediately hits `409 alreadyExists` against a
still-deleting instance. Hence replacement **polls to `NotFound` before recreating**
(rows 7 → 9 → 2). Do not add the hash to the instance name; that breaks the spec's
naming contract and only buys blue/green, a non-goal.

**Crash-safety:** ID is never lost destructively. Even if status is wiped entirely, the
create branch calls `Create`, which finds the existing VM by deterministic name and
returns its ID. That's what makes deterministic naming load-bearing rather than
cosmetic.

**Adopt-on-empty-annotation is required** — otherwise the first deploy of an operator
version whose hash-input struct gained a field mass-replaces the whole fleet.

**Spec hash:** SHA-256 over canonical JSON of an explicit `{placement, cloudInit
(resolved content), port}` struct — explicit, not `ProxySpec` wholesale, to bound
upgrade churn. Because it covers *resolved* Secret content, rotating the Secret must
re-trigger: `Watches(&corev1.Secret{}, EnqueueRequestsFromMapFunc(proxiesForSecret))`
with the cache restricted to Secrets labelled `crawl.example.com/cloud-init=true`.
~30 lines, and it's the difference between immutable replacement working and being
silently stale.

**Requeue per class:** `ErrTransient`/unclassified → return the error (workqueue
backoff). `ErrQuotaExceeded` → condition + `RequeueAfter: 5m`, return **nil** (keeps it
off the backoff curve and out of the error log). `ErrPermanent` → phase `Failed`,
condition, return nil. `ErrNotFound` → never an error, a state-machine input.

Conditions via `apimeta.SetStatusCondition` — note it does **not** populate
`ObservedGeneration`, so pass it explicitly or every condition reports generation 0.
`MaxConcurrentReconciles: 3`.

---

## Step 5 — Health engine (`internal/health/`)

**Delivery: push transitions to the reconciler via channel + `source.Channel` (option
a).** Rationale for `docs/architecture.md` → Decisions: `status.phase` is derived from
*both* provisioning and health. Under direct-patch, two writers each compute `phase`
from half the picture and race on the same subresource — a lost-update/flapping bug.
Option (a) keeps exactly one writer of `.status`, makes "write only on transition" fall
out for free (the engine only *emits* on transition), and costs one channel plus a
read-only `Snapshot()` method. The engine owns health *state*; the reconciler owns
health *representation*.

Channel typed `event.TypedGenericEvent[client.Object]` so the untyped
`&handler.EnqueueRequestForObject{}` satisfies it. Non-blocking send with `default:` —
a wedged reconciler must never stall the probe loop; on drop, don't advance the
"reported" markers, so the next probe retries.

**Threading:** one scheduler goroutine on a 1 s ticker + a fixed pool of 8 workers fed
by a buffered channel. Each tick lists from the cache and enqueues proxies whose
`nextDue <= now` and that aren't in flight. At tens of proxies a per-second list-scan is
free and a timer wheel is unjustified complexity. Startup jitter seeds
`nextDue = now + rand(0, interval)` so a restart doesn't fire every probe at once.

**Probe client** — fresh transport per probe, `defer CloseIdleConnections()`:
```go
Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), DisableKeepAlives: true,
  ForceAttemptHTTP2: false, TLSHandshakeTimeout: timeout,
  ResponseHeaderTimeout: timeout, DialContext: (&net.Dialer{Timeout: timeout}).DialContext }
```
`DisableKeepAlives: true` is **load-bearing** — otherwise `net/http` caches the
established CONNECT tunnel and later probes never re-exercise CONNECT, which is exactly
the failure the spec wants caught.

**CONNECT semantics:** for the default `https://` probe URL the transport sends
`CONNECT host:443` then TLS-handshakes through the tunnel. A proxy that accepts TCP but
can't egress returns non-200 to CONNECT, and `client.Do` returns an **error**, not a
response. So the success predicate is `err == nil && slices.Contains(expected,
resp.StatusCode)` — both halves. Latency is wall time around `Do`, last-value.

**Threshold state:** `map[NamespacedName]*state` under a mutex. Entries are pruned each
tick against the cache list (no leak), and `state.uid` is compared to the CR's UID so a
delete+recreate of the same name doesn't inherit stale fail counters. On leader
handover, state is empty: seed `healthy` from the CR's existing `Healthy` condition so a
healthy proxy doesn't flap to Unknown, but leave counters at zero so a real transition
still needs a full `failureThreshold` run. Documented as a Decision.

**Suppression:** emit only on (a) first-ever result, (b) a threshold-crossing flip, or
(c) `|new − reported| > max(20ms, 0.5×reported)` **and** `now − lastReported > 60s`.
The spec's bare ">50% latency bucket change" is undefined at 0 and makes a proxy
jittering 40↔61 ms write status forever; the absolute floor plus rate limit is what
actually delivers "no unbounded status churn". Flagged in Decisions. Metrics are
observed on *every* probe — that's the right home for high-frequency signal.

Note for the README: transition-only writes mean `status.lastHealthCheckTime` is stale
by construction. It means "time of the last *status-affecting* probe"; true probe
recency lives in metrics.

---

## Step 6 — Lease store (`internal/lease/`)

**`Acquire` takes the candidate *set*, not a chosen proxy** — selection and insertion
must happen under one lock, or two concurrent requests both see "3 of 5 used" and
overcommit.

```go
Acquire(ctx, AcquireRequest{Candidates []Candidate; Target string; TTL time.Duration}) (*Lease, AcquireStats, error)
Release / Report / ActiveCount / Counts / ExpireLoop
```

One `sync.Mutex` for the whole store (tens of proxies, human-rate QPS; sharding is
premature), injectable clock, `byID` + `byProxy` + `cooldown[{proxy,target}]` maps.
Selection is a linear scan + `slices.SortFunc` on `(activeLeases asc, latency asc,
name asc)` — explicitly not a heap, and the third key makes it deterministic and
testable. `AcquireStats{Considered, AtCapacity, InCooldown}` feeds the 409 body.

**Expired-lease retention:** entries stay marked `expired` for the cooldown window
after TTL. `Acquire`/`ActiveCount` ignore them; `Report` still resolves them. Without
this, a report arriving just after the TTL lapses is silently dropped — exactly when a
proxy is being rate-limited, which is when the cooldown matters most.

---

## Step 7 — Discovery API (`internal/discovery/`)

**`NeedLeaderElection() = false`**, and ship `replicas: 1`. Verified in
controller-runtime's runnable ordering: caches start and sync *before* non-leader-
election runnables, so cache reads are safe. If it *were* leader-elected, non-leader
pods would refuse connections while still being Service endpoints. The 1-replica
constraint comes from lease state being per-process, which the spec already accepts —
both facts go in the README caveats.

stdlib `http.ServeMux` with Go 1.22 method+wildcard patterns:
`GET /v1/proxies`, `POST /v1/leases`, `DELETE /v1/leases/{id}`,
`POST /v1/leases/{id}/report`, plus unauthenticated `GET /healthz`.
Middleware outermost-first: recover → request-log → `MaxBytesReader(64KiB)` → bearer
auth. Empty `DISCOVERY_TOKEN` passes through **with a loud startup Warn** — in-cluster
that's a silent security hole otherwise. Token compared with
`subtle.ConstantTimeCompare`.

Shapes: list returns `{"proxies":[…],"count":N}`, empty is 200 not 404; lease grant is
201 `{leaseID, proxy, expiresAt, ttlSeconds}`; no match is 409
`{"error":"no_match","message":…,"considered":7,"atCapacity":2,"inCooldown":2,"unhealthy":3}`;
DELETE is always 204; report is 204, 400 on an unknown result value, 404 on a genuinely
unknown lease. All errors share `{"error":"<machine_code>","message":"<human>"}`.
Timeouts on `http.Server`, graceful `Shutdown` with 10 s grace on ctx cancel.

---

## Step 8 — GCP provider (`internal/provider/gcp/`)

Only `compute.NewInstancesRESTClient` (ADC), only Insert/Get/Delete/AggregatedList.
`Operation.Wait` is **never called** — `Create` returns as soon as the operation is
submitted, and `409 alreadyExists` is treated as success, which is what makes a repeat
call after a crash correct.

**providerID = `zones/<zone>/instances/<name>`** — zone-qualified so `Get`/`Delete` are
self-contained. The spec's `Get(ctx, providerID)` carries no zone, and re-reading
`spec.placement.zone` is wrong precisely when a zone edit is the replacement being
processed.

**Test seam is deliberately not an SDK mirror.** `compute.InstancesScopedListPairIterator`
has an unexported `nextFunc`, so a fake cannot construct one — the interface flattens
`AggregatedList` to a slice and returns operations as just their name. The primary unit
test needs no fake at all: `buildInsertRequest` is pure, asserted field-by-field
(machine-type URL, boot disk, `AccessConfigs[0] = {Name:"External NAT",
Type:"ONE_TO_ONE_NAT"}`, `Metadata.Items[user-data]`, GC labels, network tag).

`AggregatedList` needs `ReturnPartialSuccess: true` — otherwise one unreachable zone
fails the entire GC sweep. `RUNNING` **without** a NatIP maps to `Provisioning`, not
Running, so we never publish an empty IP. Error mapping: 404→NotFound;
429 / 403+`quotaExceeded`→Quota; 400/401/403-other→Permanent; 5xx/408/net→Transient;
unknown→Transient.

---

## Step 9 — Orphan GC + metrics

**GC** (`internal/gc/`) — `NeedLeaderElection() = true` (destructive, single-writer),
10 min ticker, first sweep one interval after start. Per provider `ListByTag`; on error
log and continue to the next provider, never abort the sweep. Kill an instance only if
it has our UID label, is older than `MinAge` (10 min), and its UID matches no CR.
Log every kill at Warn with provider, providerID, UID.

**A CR with a `deletionTimestamp` still counts as live** — that's what the spec's
"check deletionTimestamp semantics carefully" points at. Its finalizer owns the
deletion; GC racing it double-deletes. A UID is orphan-eligible only once the object is
fully gone.

**Namespace-scope guard:** if the cache is namespace-restricted but Proxies exist
elsewhere, GC would delete live VMs. It refuses to start unless an explicit
`--gc-allow-namespaced` flag is set.

**Metrics** (`internal/metrics/`) registered by an explicit `Register(...)` called from
`main` (no `init()`, per house rules; also lets tests use a fresh registry).
`proxy_operator_proxies{phase}` and `proxy_operator_leases_active` are **custom
Collectors** that read at scrape time — a reconcile-incremented gauge inevitably drifts
and leaks a series on delete. Per-proxy histogram/counter labels **must** be deleted
from the vec when the health engine GCs a state entry, or series leak forever.
`proxy_operator_provider_requests_total{provider,op,result}` comes from a
`provider.WithMetrics(name, p)` decorator — zero-cost instrumentation for the next five
providers, and the one place `Class()` is called for observability.

---

## Step 10 — Wiring, config, docs

`cmd/main.go`: flags `--providers-config` (required), `--discovery-addr`,
`--proxy-namespace`, `--health-workers`, `--gc-interval`, `--gc-min-age`,
`--lease-cooldown`, `--max-lease-ttl`. Order: load provider config (fail fast) → build
registry → manager → `mgr.Add` health engine, GC, lease expiry loop, discovery server →
`SetupWithManager`. Contexts from `ctrl.SetupSignalHandler()` throughout.

RBAC markers: proxies CRUD + status + finalizers, secrets get/list/watch, events
create/patch. `config/`: providers ConfigMap mount, `DISCOVERY_TOKEN` from a Secret,
containerPort 8090 + Service. Samples: `proxy_mock.yaml`, `proxy_gcp.yaml`,
`proxy_external.yaml`, `providers-config.yaml`.

`docs/architecture.md`: components table, ASCII data-flow diagram, and a **Decisions**
section covering the channel-vs-patch choice, replacement-polls-to-NotFound, base32
naming, discovery without leader election, single-mutex lease store, mock-runs-a-real-
proxy, leader-handover health seeding, the latency-suppression refinement, the
`lastHealthCheckTime` semantics, and the logr-not-slog deviation (CLAUDE.md says
`slog`, but `log.FromContext(ctx)` returns logr inside controller paths — noted, not
silently ignored).

`README.md`: 60-second architecture summary; copy-pasteable kind quickstart; GCP setup
(ADC, `roles/compute.instanceAdmin.v1`, plus `roles/iam.serviceAccountUser` if
attaching a service account); the **immutable-replacement caveat** (changing a proxy
changes its IP) and **lease-loss-on-restart caveat**, both prominent; a version-pins
note confirming no substitutions were needed. Then a `CHANGELOG.md` entry with a real
`date "+%Y-%m-%d %H:%M %Z"` timestamp.

---

## Step 11 — Tests

**envtest** (`internal/controller/`), mock provider + a fake `HealthSnapshotter`,
intervals shrunk to 50–200 ms, whole suite behind `testing.Short()`:
Managed→Ready; spec change → old mock instance gone and providerID changed; delete →
finalizer runs and instance removed; External → Ready on first health pass, no
finalizer; injected quota → `Provisioned=False/QuotaExceeded` and phase *not* Failed;
permanent error → Failed and no further provider calls; adopt (strip annotation →
restored, providerID unchanged); and the **CEL cases only a real API server can test** —
mode/provider mutation rejected, Managed-without-provider, External-without-endpoint,
cloudInit both/neither, and `healthCheck` omitted → nested defaults materialized (the
`default={}` assertion).

**Action-table unit tests** — the highest-value tests in the repo: `fake` client with
`WithStatusSubresource`, calling `Reconcile` directly, table-driven over every row of
the Step 4 table, asserting the returned `ctrl.Result`. (Documented caveat: the fake
client runs neither CEL nor defaulting — that's what the envtest CEL cases cover.)

**Units:** name derivation (idempotency, `^proxy-[a-z2-7]{16}$`, 10k-UID distinctness);
`Class()` mapping + `errors.Is`/`errors.As` through the multi-unwrap; config loading;
mock state at the delay boundary with a fake clock; `buildInsertRequest` field-by-field
+ GCP error classification + RUNNING-without-IP; `computePhase` truth table; `SpecHash`
stability *and* sensitivity; lease store (capacity, `MaxLeases=0`, least-loaded with
latency tie-break, cooldown with/without target, report on an expired-but-retained
lease, concurrent acquire under `-race` never exceeding `MaxLeases`); discovery
handlers over `httptest` + fake reader + real store; health thresholds against a real
CONNECT-capable `httptest` proxy stub.

Everything runs with `-race`.

---

## Verification

```bash
go vet ./... && make test && make build      # unit + envtest, -race

kind create cluster --name proxy-operator-demo
make install
make run-mock &                              # --providers-config hack/providers-mock.yaml
kubectl apply -f config/samples/proxy_mock.yaml
kubectl get px -w                            # expect Ready with an IP
curl -s 'localhost:8090/v1/proxies?healthy=true' | jq
curl -s -XPOST localhost:8090/v1/leases -d '{"selector":{"geo":"eu"},"ttlSeconds":300}' | jq
curl -s -XPOST localhost:8090/v1/leases/<id>/report -d '{"result":"rate_limited","target":"example.com"}'
curl -si -XDELETE localhost:8090/v1/leases/<id>   # 204, and 204 again
kubectl delete -f config/samples/proxy_mock.yaml  # finalizer runs, object goes
kind delete cluster --name proxy-operator-demo
```

Success bar: a competent SRE clones the repo, follows the README, and holds a lease on a
healthy mock proxy in under 10 minutes.

Then commit on `feat/proxy-operator`, push with `-u`, open the MR with
`tea pr create --base main --head feat/proxy-operator`, print the URL. No merging or
branch deletion from the CLI.