egress-proxies-operator/docs/plans/2026-08-07-1747-proxy-operator.md

Plan: proxy-operator — Kubernetes operator for crawling-proxy VMs

Created: 2026-08-07 17:47

Context

The crawling department runs a small fleet (tens) of HTTP proxy VMs across cloud providers to dodge rate limiting. This builds a production-quality prototype operator making each proxy VM a first-class Kubernetes object: GitOps-managed, actively health-checked through the proxy, and discoverable by crawler clients via an HTTP list/lease API.

The repo is empty — go.mod, CLAUDE.md, CHANGELOG.md, .claude/, and the spec at docs/prompts/__initial-prompt.md. Zero commits. All greenfield. Governing principle: proxies are immutable cattle — any meaningful spec change deletes and recreates the VM. No in-place update logic.

Decisions locked with the user

Question	Decision
Module path	Keep gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator (spec's github.com/CHANGEME/... was a placeholder)
API group	crawl.example.com, v1alpha1, kind Proxy, namespaced
Git flow	Scaffold commit, then work on feat/proxy-operator, MR via tea, no merge/delete from CLI
kubebuilder	go install sigs.k8s.io/kubebuilder/v4/cmd/kubebuilder@v4.15.0
Local/CI provider	Kubernetes pod provider, not an in-memory mock: creates real ubuntu/squid pods in-cluster. Revised mid-build — see Step 3
Verification	vet + unit + envtest, then a throwaway kind cluster running the README quickstart, then delete it

Verified environment — no version substitutions needed

Go 1.26.4 · kubebuilder v4.15.0 · controller-runtime v0.24.1 · k8s.io/* v0.36.3 · controller-tools v0.21.0 · compute v1.65.0 · kind v0.32.0 · kubectl v1.36.1 · docker 29.6.2 · gcloud present. envtest bundles exist for k8s 1.36.0 and 1.36.2 on darwin/arm64 — the scaffold Makefile derives the minor (1.36) from k8s.io/api and resolves the latest patch; do not "fix" it to 1.36.3, which has no bundle.

Every pin in the spec is satisfiable. The README will say so explicitly rather than omitting the substitutions section.

Milestone order — each ends green on go build ./... && go vet ./...

1. Scaffold + pins → 2. api/v1alpha1 + CEL → 3. internal/provider + kubernetes-pod →
2. reconciler + envtest → 5. health engine → 6. lease + discovery → 7. GCP provider →
3. orphan GC + metrics → 9. cmd/main.go wiring + config/ → 10. docs + kind run.

---

Step 0 — Branch and scaffold

git checkout -b feat/proxy-operator          # main is unborn; branch starts empty
go install sigs.k8s.io/kubebuilder/v4/cmd/kubebuilder@v4.15.0
kubebuilder init --domain example.com \
  --repo gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator --plugins go/v4
kubebuilder create api --group crawl --version v1alpha1 --kind Proxy --resource --controller
make manifests generate

Trap: --domain crawl.example.com --group crawl yields group crawl.crawl.example.com. It must be --domain example.com --group crawl.

If init refuses the non-empty directory, scaffold into an empty temp dir with the identical --repo and copy the tree in — don't fight the emptiness check. Copy this plan to docs/plans/2026-08-07-1747-proxy-operator.md per CLAUDE.md. Commit the untouched scaffold on its own so every later diff is reviewable.

Post-scaffold hand-edits: CONTROLLER_TOOLS_VERSION ?= v0.21.0 in the Makefile (CEL emission at the 1.36 API level); add a run-dev target wired to a sample --providers-config (Step 10); delete the scaffolded .github/workflows/ (the remote is Gitea).

---

Step 1 — API types (api/v1alpha1/proxy_types.go)

Structs exactly as the spec dictates, with these four corrections that are silent bugs otherwise:

• MaxLeases *int32, not int32. With a value type + omitempty + default=5, an explicit 0 is dropped on any Go round-trip and re-defaulted to 5 — "0 = unleasable" becomes unreachable.
• HealthCheck *HealthCheckSpec needs +kubebuilder:default={}. Structural defaulting only descends into values that exist; without it a nil healthCheck gets none of its nested defaults and the spec's "all defaulted" quietly fails.
• +kubebuilder:validation:MinLength=1 on Provider and CloudInit.Inline, so an explicit "" fails OpenAPI validation and the CEL has() rules stay simple.
• Conditions get +listType=map +listMapKey=type.

CEL at the ProxySpec struct level — cross-field rules cannot live on a field:

// +kubebuilder:validation:XValidation:rule="self.mode == oldSelf.mode",message="mode is immutable"
// +kubebuilder:validation:XValidation:rule="has(self.provider) == has(oldSelf.provider) && (!has(self.provider) || self.provider == oldSelf.provider)",message="provider is immutable"
// +kubebuilder:validation:XValidation:rule="self.mode != 'Managed' || has(self.provider)",message="provider is required when mode is Managed"
// +kubebuilder:validation:XValidation:rule="self.mode != 'External' || !has(self.provider)",message="provider must not be set when mode is External"
// +kubebuilder:validation:XValidation:rule="self.mode != 'External' || has(self.endpoint)",message="endpoint is required when mode is External"
// +kubebuilder:validation:XValidation:rule="self.mode != 'Managed' || !has(self.endpoint)",message="endpoint must not be set when mode is Managed"

plus on CloudInitSpec: rule="has(self.inline) != has(self.secretRef)" (CEL != on booleans is XOR).

Semantics that matter: a rule mentioning oldSelf is skipped on CREATE, so immutability and required-iff must be separate markers — &&-ing them together would skip the required-iff check on create. The immutability rule uses the has(self.x) == has(oldSelf.x) && ... form because Provider is optional and a field-level rule wouldn't fire when the field is absent on either side.

Defaults: port=3128, maxLeases=5, probeURL https://www.gstatic.com/generate_204, interval 30s, timeout 5s, failureThreshold 3, successThreshold 1, expectedStatusCodes {200,204}. Printer columns Mode/Provider/Phase/IP/Healthy/Age, shortName=px, +kubebuilder:subresource:status.

Pure helpers in helpers.go (unit-tested): EffectivePort(), EffectiveHost(), HealthCheckOrDefault(), MaxLeasesOrDefault(). Port lives in two places (spec.port for Managed, spec.endpoint.port for External) — one helper, used by health, discovery, and the hash.

---

Step 2 — Provider contract (internal/provider/)

internal/provider/{provider,errors,name,config,metrics}.go
internal/provider/registry/registry.go   # type→constructor — SEPARATE package
internal/provider/{kubernetes,gcp}/

Import-cycle trap: a registry inside internal/provider would have to import internal/provider/kubernetes (or .../gcp), both of which import internal/provider for the interface. init() self-registration is banned by CLAUDE.md, so the registry goes in its own leaf-importing package, and its Build function takes the type→constructor map as a parameter instead — see Step 2's execution log entry for why this ended up better than a package-level map even beyond avoiding the cycle.

Instance needs two fields the spec omits, or orphan GC is unimplementable: UID string (from the label, for the liveness match) and CreatedAt time.Time (for the "skip < 10 min" rule).

Error taxonomy — the key trick is multi-value unwrap:

type Error struct{ Class error; Op, Provider, ID string; Err error }
func (e *Error) Unwrap() []error { return []error{e.Class, e.Err} }
func Class(err error) error  // returns the sentinel; unclassified → ErrTransient

So errors.Is(err, ErrQuotaExceeded) and errors.As(err, &googleapiErr) both work on the same value. Class() defaulting to ErrTransient matters: retrying is always safer than latching Failed.

Deterministic naming — SHA-256, first 10 bytes, RFC 4648 base32 lowercased, unpadded → 16 chars, proxy- + that = 22 total:

func NameFromUID(uid types.UID) string

GCP requires ^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$ ≤63. base32 lowercases to [a-z2-7] — all legal; base64's +///uppercase are not, and hex would need 20 chars for the same 80 bits. 80 bits → birthday collision at ~2^40 objects against a fleet of tens. 22 chars leaves headroom because GCP auto-names the boot disk after the instance.

config.go parses the YAML with yaml.UnmarshalStrict and validates at load (non-empty + unique names, known type, type block present, gcp requires project). Fail fast from main — never degrade.

---

Step 3 — Kubernetes pod provider (internal/provider/kubernetes/)

Revised after Step 3 was first built as an in-memory mock provider (state-machine simulation + a hand-rolled CONNECT proxy on a shared, refcounted local listener). The user found that too far from the real system to build confidence in, and didn't need tests to be fast enough to justify the complexity it cost — a real kind-cluster verification pass "once in a while" is an acceptable trade for tests that actually look like the final product. Full narrative of the reversal is in docs/plans-executions/2026-08-07-1747-proxy-operator.md; this section describes the replacement, which is what actually gets built for local dev/CI going forward. No in-memory provider remains in the tree — GCP is now the only other provider, per the spec's original two-provider scope.

What it does: Create creates a corev1.Pod running a proxy container in the same cluster (and same namespace as the owning Proxy CR — req.Namespace); Get reads the Pod's phase/IP; Delete deletes it (tolerating NotFound); ListByTag lists Pods by the standard LabelManaged/LabelUID labels, unscoped by namespace (the operator's RBAC needs cluster-scoped Pod permissions — see RBAC note below).

Proxy software: ubuntu/squid (Canonical's actively maintained LTS image on Docker Hub, verified before picking it — 50M+ pulls, updated the same day this decision was made), not a hand-rolled proxy. It's a public image, so kind nodes pull it directly; no build/load step needed for the quickstart. Squid's config (http_port <req.Port>, permissive ACL) is generated in Go and injected via an env var the container's command writes to /etc/squid/squid.conf before exec'ing squid — no separate ConfigMap object, so there's still only one Kubernetes object per proxy instance to create, track, and clean up.

providerID format: <namespace>/<podName> (parseable with k8s.io/client-go/tools/cache.SplitMetaNamespaceKey), so Get/Delete are self-contained without needing to re-derive the namespace — the same reasoning as the GCP provider's zone-qualified providerID in Step 8.

Pod naming: reuses provider.NameFromUID unchanged — the same deterministic name satisfies Kubernetes Pod naming rules (^[a-z0-9]([-a-z0-9]*[a-z0-9])?$, ≤253 chars) with room to spare.

State mapping: Pod phase Pending, or Running with no PodIP yet → Provisioning (never publish an empty IP); Running with a PodIP → Running; Succeeded/Failed/ Unknown → Terminated (the reconciler treats Stopped and Terminated identically — delete and recreate, cattle not pets — so collapsing three failure-ish phases into one is enough).

Client: built internally via ctrl.GetConfig() (auto-detects in-cluster config, falls back to the local kubeconfig otherwise), not threaded through the registry Constructor signature — this is what makes make run against a local kind cluster and running in-cluster use the exact same code path with no provider-specific wiring in cmd/main.go.

Testing, given envtest can't schedule real Pods (no kubelet — a Pod created against envtest's API server just sits Pending forever): internal/provider/kubernetes itself is unit-tested against sigs.k8s.io/controller-runtime/pkg/client/fake — real Pod objects, real client interface, fully exercises Create/Get/Delete/ListByTag logic and the Pod-construction function in isolation, just without a kubelet actually starting a container. Real end-to-end proof (does a probe actually tunnel through a real Squid pod) only happens against a real kind cluster, in the Verification section — which is exactly what the user asked for. Step 4's reconciler tests use a small Provider-interface stub defined directly in the controller test file (a handful of lines, not a package) for exercising the state-machine's branching logic — categorically simpler than what mock.Provider was, since it has no config format, no fault-injection surface, and exists only inside test code.

---

Step 4 — Reconciler (internal/controller/)

A state machine: every reconcile derives one action from (spec, status, provider Get). Intervals are struct fields, never consts, so envtest can shrink them to milliseconds.

func (r *ProxyReconciler) Reconcile(ctx, req) (res ctrl.Result, err error) {
    // Get; base := p.DeepCopy()
    // defer patchStatusIfChanged(ctx, base, &p)   // one status write per reconcile, max
    switch {
    case !p.DeletionTimestamp.IsZero():        return r.reconcileDelete(ctx, &p)
    case p.Spec.Mode == v1alpha1.ModeExternal: return r.reconcileExternal(ctx, &p)
    default:                                   return r.reconcileManaged(ctx, &p)
    }
}

patchStatusIfChanged sets observedGeneration and phase = computePhase(&p), then issues nothing when equality.Semantic.DeepEqual(base.Status, p.Status). computePhase is pure and is the primary table-driven unit-test target.

Action table (Managed) — A = hash annotation, H = computed hash

deletionTS	ID	A vs H	provider.Get	action	result
no	—	—	—	finalizer absent → add it	{} (the Update re-triggers)
no	""	any	—	resolve cloud-init, Create, set ID + A=H	RequeueAfter: ProvisioningPoll
no	set	==	Provisioning	clear ip, Provisioned=False/Provisioning	RequeueAfter: ProvisioningPoll
no	set	==	Running	set ip, Provisioned=True/Created	RequeueAfter: DriftPoll
no	set	==	Stopped/Terminated	Delete(ID) — cattle, not pets	RequeueAfter: DeletionPoll
no	set	==	NotFound	clear ID+ip → next pass creates	Requeue: true
no	set	!=, A != ""	any	replace: Delete(ID), Provisioned=False/Replacing	RequeueAfter: DeletionPoll
no	set	!=, A == ""	any	adopt: set A=H, no replacement	Requeue: true
no	set	!=	NotFound	clear ID+ip (status), then set A=H (metadata)	Requeue: true
yes	""	—	—	remove finalizer (orphan GC reaps any stray VM)	{}
yes	set	—	NotFound	remove finalizer	{}
yes	set	—	anything else	Delete(ID), phase=Deleting	RequeueAfter: DeletionPoll

External: no finalizer, no provider calls, status.ip = spec.endpoint.host, Provisioned=True/ExternalEndpoint, health drives the rest.

The trap the spec doesn't mention: the instance name derives from the CR UID, which does not change on a spec edit — so the replacement VM has the same name as the one being deleted. Recreating immediately hits 409 alreadyExists against a still-deleting instance. Hence replacement polls to NotFound before recreating (rows 7 → 9 → 2). Do not add the hash to the instance name; that breaks the spec's naming contract and only buys blue/green, a non-goal.

Crash-safety: ID is never lost destructively. Even if status is wiped entirely, the create branch calls Create, which finds the existing VM by deterministic name and returns its ID. That's what makes deterministic naming load-bearing rather than cosmetic.

Adopt-on-empty-annotation is required — otherwise the first deploy of an operator version whose hash-input struct gained a field mass-replaces the whole fleet.

Spec hash: SHA-256 over canonical JSON of an explicit {placement, cloudInit (resolved content), port} struct — explicit, not ProxySpec wholesale, to bound upgrade churn. Because it covers resolved Secret content, rotating the Secret must re-trigger: Watches(&corev1.Secret{}, EnqueueRequestsFromMapFunc(proxiesForSecret)) with the cache restricted to Secrets labelled crawl.example.com/cloud-init=true. ~30 lines, and it's the difference between immutable replacement working and being silently stale.

Requeue per class: ErrTransient/unclassified → return the error (workqueue backoff). ErrQuotaExceeded → condition + RequeueAfter: 5m, return nil (keeps it off the backoff curve and out of the error log). ErrPermanent → phase Failed, condition, return nil. ErrNotFound → never an error, a state-machine input.

Conditions via apimeta.SetStatusCondition — note it does not populate ObservedGeneration, so pass it explicitly or every condition reports generation 0. MaxConcurrentReconciles: 3.

---

Step 5 — Health engine (internal/health/)

Delivery: push transitions to the reconciler via channel + source.Channel (option a). Rationale for docs/architecture.md → Decisions: status.phase is derived from both provisioning and health. Under direct-patch, two writers each compute phase from half the picture and race on the same subresource — a lost-update/flapping bug. Option (a) keeps exactly one writer of .status, makes "write only on transition" fall out for free (the engine only emits on transition), and costs one channel plus a read-only Snapshot() method. The engine owns health state; the reconciler owns health representation.

Channel typed event.TypedGenericEvent[client.Object] so the untyped &handler.EnqueueRequestForObject{} satisfies it. Non-blocking send with default: — a wedged reconciler must never stall the probe loop; on drop, don't advance the "reported" markers, so the next probe retries.

Threading: one scheduler goroutine on a 1 s ticker + a fixed pool of 8 workers fed by a buffered channel. Each tick lists from the cache and enqueues proxies whose nextDue <= now and that aren't in flight. At tens of proxies a per-second list-scan is free and a timer wheel is unjustified complexity. Startup jitter seeds nextDue = now + rand(0, interval) so a restart doesn't fire every probe at once.

Probe client — fresh transport per probe, defer CloseIdleConnections():

Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), DisableKeepAlives: true,
  ForceAttemptHTTP2: false, TLSHandshakeTimeout: timeout,
  ResponseHeaderTimeout: timeout, DialContext: (&net.Dialer{Timeout: timeout}).DialContext }

DisableKeepAlives: true is load-bearing — otherwise net/http caches the established CONNECT tunnel and later probes never re-exercise CONNECT, which is exactly the failure the spec wants caught.

CONNECT semantics: for the default https:// probe URL the transport sends CONNECT host:443 then TLS-handshakes through the tunnel. A proxy that accepts TCP but can't egress returns non-200 to CONNECT, and client.Do returns an error, not a response. So the success predicate is err == nil && slices.Contains(expected, resp.StatusCode) — both halves. Latency is wall time around Do, last-value.

Threshold state: map[NamespacedName]*state under a mutex. Entries are pruned each tick against the cache list (no leak), and state.uid is compared to the CR's UID so a delete+recreate of the same name doesn't inherit stale fail counters. On leader handover, state is empty: seed healthy from the CR's existing Healthy condition so a healthy proxy doesn't flap to Unknown, but leave counters at zero so a real transition still needs a full failureThreshold run. Documented as a Decision.

Suppression: emit only on (a) first-ever result, (b) a threshold-crossing flip, or (c) |new − reported| > max(20ms, 0.5×reported) and now − lastReported > 60s. The spec's bare ">50% latency bucket change" is undefined at 0 and makes a proxy jittering 40↔61 ms write status forever; the absolute floor plus rate limit is what actually delivers "no unbounded status churn". Flagged in Decisions. Metrics are observed on every probe — that's the right home for high-frequency signal.

Note for the README: transition-only writes mean status.lastHealthCheckTime is stale by construction. It means "time of the last status-affecting probe"; true probe recency lives in metrics.

---

Step 6 — Lease store (internal/lease/)

Acquire takes the candidate set, not a chosen proxy — selection and insertion must happen under one lock, or two concurrent requests both see "3 of 5 used" and overcommit.

Acquire(ctx, AcquireRequest{Candidates []Candidate; Target string; TTL time.Duration}) (*Lease, AcquireStats, error)
Release / Report / ActiveCount / Counts / ExpireLoop

One sync.Mutex for the whole store (tens of proxies, human-rate QPS; sharding is premature), injectable clock, byID + byProxy + cooldown[{proxy,target}] maps. Selection is a linear scan + slices.SortFunc on (activeLeases asc, latency asc, name asc) — explicitly not a heap, and the third key makes it deterministic and testable. AcquireStats{Considered, AtCapacity, InCooldown} feeds the 409 body.

Expired-lease retention: entries stay marked expired for the cooldown window after TTL. Acquire/ActiveCount ignore them; Report still resolves them. Without this, a report arriving just after the TTL lapses is silently dropped — exactly when a proxy is being rate-limited, which is when the cooldown matters most.

---

Step 7 — Discovery API (internal/discovery/)

NeedLeaderElection() = false, and ship replicas: 1. Verified in controller-runtime's runnable ordering: caches start and sync before non-leader- election runnables, so cache reads are safe. If it were leader-elected, non-leader pods would refuse connections while still being Service endpoints. The 1-replica constraint comes from lease state being per-process, which the spec already accepts — both facts go in the README caveats.

stdlib http.ServeMux with Go 1.22 method+wildcard patterns: GET /v1/proxies, POST /v1/leases, DELETE /v1/leases/{id}, POST /v1/leases/{id}/report, plus unauthenticated GET /healthz. Middleware outermost-first: recover → request-log → MaxBytesReader(64KiB) → bearer auth. Empty DISCOVERY_TOKEN passes through with a loud startup Warn — in-cluster that's a silent security hole otherwise. Token compared with subtle.ConstantTimeCompare.

Shapes: list returns {"proxies":[…],"count":N}, empty is 200 not 404; lease grant is 201 {leaseID, proxy, expiresAt, ttlSeconds}; no match is 409 {"error":"no_match","message":…,"considered":7,"atCapacity":2,"inCooldown":2,"unhealthy":3}; DELETE is always 204; report is 204, 400 on an unknown result value, 404 on a genuinely unknown lease. All errors share {"error":"<machine_code>","message":"<human>"}. Timeouts on http.Server, graceful Shutdown with 10 s grace on ctx cancel.

---

Step 8 — GCP provider (internal/provider/gcp/)

Only compute.NewInstancesRESTClient (ADC), only Insert/Get/Delete/AggregatedList. Operation.Wait is never called — Create returns as soon as the operation is submitted, and 409 alreadyExists is treated as success, which is what makes a repeat call after a crash correct.

providerID = zones/<zone>/instances/<name> — zone-qualified so Get/Delete are self-contained. The spec's Get(ctx, providerID) carries no zone, and re-reading spec.placement.zone is wrong precisely when a zone edit is the replacement being processed.

Test seam is deliberately not an SDK mirror. compute.InstancesScopedListPairIterator has an unexported nextFunc, so a fake cannot construct one — the interface flattens AggregatedList to a slice and returns operations as just their name. The primary unit test needs no fake at all: buildInsertRequest is pure, asserted field-by-field (machine-type URL, boot disk, AccessConfigs[0] = {Name:"External NAT", Type:"ONE_TO_ONE_NAT"}, Metadata.Items[user-data], GC labels, network tag).

AggregatedList needs ReturnPartialSuccess: true — otherwise one unreachable zone fails the entire GC sweep. RUNNING without a NatIP maps to Provisioning, not Running, so we never publish an empty IP. Error mapping: 404→NotFound; 429 / 403+quotaExceeded→Quota; 400/401/403-other→Permanent; 5xx/408/net→Transient; unknown→Transient.

---

Step 9 — Orphan GC + metrics

GC (internal/gc/) — NeedLeaderElection() = true (destructive, single-writer), 10 min ticker, first sweep one interval after start. Per provider ListByTag; on error log and continue to the next provider, never abort the sweep. Kill an instance only if it has our UID label, is older than MinAge (10 min), and its UID matches no CR. Log every kill at Warn with provider, providerID, UID.

A CR with a deletionTimestamp still counts as live — that's what the spec's "check deletionTimestamp semantics carefully" points at. Its finalizer owns the deletion; GC racing it double-deletes. A UID is orphan-eligible only once the object is fully gone.

Namespace-scope guard: if the cache is namespace-restricted but Proxies exist elsewhere, GC would delete live VMs. It refuses to start unless an explicit --gc-allow-namespaced flag is set.

Metrics (internal/metrics/) registered by an explicit Register(...) called from main (no init(), per house rules; also lets tests use a fresh registry). proxy_operator_proxies{phase} and proxy_operator_leases_active are custom Collectors that read at scrape time — a reconcile-incremented gauge inevitably drifts and leaks a series on delete. Per-proxy histogram/counter labels must be deleted from the vec when the health engine GCs a state entry, or series leak forever. proxy_operator_provider_requests_total{provider,op,result} comes from a provider.WithMetrics(name, p) decorator — zero-cost instrumentation for the next five providers, and the one place Class() is called for observability.

---

Step 10 — Wiring, config, docs

cmd/main.go: flags --providers-config (required), --discovery-addr, --proxy-namespace, --health-workers, --gc-interval, --gc-min-age, --lease-cooldown, --max-lease-ttl. Order: load provider config (fail fast) → build registry → manager → mgr.Add health engine, GC, lease expiry loop, discovery server → SetupWithManager. Contexts from ctrl.SetupSignalHandler() throughout.

RBAC markers: proxies CRUD + status + finalizers, secrets get/list/watch, events create/patch, and (for the kubernetes-pod provider) pods get/list/watch/create/delete — cluster-scoped, since ListByTag enumerates across namespaces. config/: providers ConfigMap mount, DISCOVERY_TOKEN from a Secret, containerPort 8090 + Service. Samples: proxy_kubernetes.yaml, proxy_gcp.yaml, proxy_external.yaml, providers-config.yaml.

docs/architecture.md: components table, ASCII data-flow diagram, and a Decisions section covering the channel-vs-patch choice, replacement-polls-to-NotFound, base32 naming, discovery without leader election, single-mutex lease store, the mock→ kubernetes-pod-provider revision (why, and why ubuntu/squid over a hand-rolled proxy), leader-handover health seeding, the latency-suppression refinement, the lastHealthCheckTime semantics, and the logr-not-slog deviation (CLAUDE.md says slog, but log.FromContext(ctx) returns logr inside controller paths — noted, not silently ignored).

README.md: 60-second architecture summary; copy-pasteable kind quickstart; GCP setup (ADC, roles/compute.instanceAdmin.v1, plus roles/iam.serviceAccountUser if attaching a service account); the immutable-replacement caveat (changing a proxy changes its IP) and lease-loss-on-restart caveat, both prominent; a version-pins note confirming no substitutions were needed. Then a CHANGELOG.md entry with a real date "+%Y-%m-%d %H:%M %Z" timestamp.

---

Step 11 — Tests

envtest (internal/controller/), a small in-test stub Provider (not the real kubernetes-pod provider — envtest has no kubelet, so a real Pod never leaves Pending) plus a fake HealthSnapshotter, intervals shrunk to 50–200 ms, whole suite behind testing.Short(): Managed→Ready; spec change → old stub instance gone and providerID changed; delete → finalizer runs and instance removed; External → Ready on first health pass, no finalizer; injected quota → Provisioned=False/QuotaExceeded and phase not Failed; permanent error → Failed and no further provider calls; adopt (strip annotation → restored, providerID unchanged); and the CEL cases only a real API server can test — mode/provider mutation rejected, Managed-without-provider, External-without-endpoint, cloudInit both/neither, and healthCheck omitted → nested defaults materialized (the default={} assertion).

Action-table unit tests — the highest-value tests in the repo: fake client with WithStatusSubresource, calling Reconcile directly, table-driven over every row of the Step 4 table, asserting the returned ctrl.Result. (Documented caveat: the fake client runs neither CEL nor defaulting — that's what the envtest CEL cases cover.)

Units: name derivation (idempotency, ^proxy-[a-z2-7]{16}$, 10k-UID distinctness); Class() mapping + errors.Is/errors.As through the multi-unwrap; config loading; kubernetes-pod provider Create/Get/Delete/ListByTag against sigs.k8s.io/controller-runtime/pkg/client/fake (real Pod objects, real client interface, no kubelet needed for this level) plus pure tests of the generated Squid config and Pod spec; buildInsertRequest field-by-field + GCP error classification + RUNNING-without-IP; computePhase truth table; SpecHash stability and sensitivity; lease store (capacity, MaxLeases=0, least-loaded with latency tie-break, cooldown with/without target, report on an expired-but-retained lease, concurrent acquire under -race never exceeding MaxLeases); discovery handlers over httptest + fake reader + real store; health thresholds against a real CONNECT-capable httptest proxy stub.

Everything runs with -race.

---

Verification

go vet ./... && make test && make build      # unit + envtest, -race

kind create cluster --name proxy-operator-demo
make install
make run-dev &                                # --providers-config hack/providers-dev.yaml
kubectl apply -f config/samples/proxy_kubernetes.yaml
kubectl get px -w                            # expect Ready with an IP (a real squid Pod)
curl -s 'localhost:8090/v1/proxies?healthy=true' | jq
curl -s -XPOST localhost:8090/v1/leases -d '{"selector":{"geo":"eu"},"ttlSeconds":300}' | jq
curl -s -XPOST localhost:8090/v1/leases/<id>/report -d '{"result":"rate_limited","target":"example.com"}'
curl -si -XDELETE localhost:8090/v1/leases/<id>   # 204, and 204 again
kubectl delete -f config/samples/proxy_kubernetes.yaml  # finalizer runs, Pod is deleted
kind delete cluster --name proxy-operator-demo

Success bar: a competent SRE clones the repo, follows the README, and holds a lease on a healthy proxy — a real Squid pod running in their own kind cluster — in under 10 minutes.

Then commit on feat/proxy-operator, push with -u, open the MR with tea pr create --base main --head feat/proxy-operator, print the URL. No merging or branch deletion from the CLI.