kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Scaffold proxy-operator with kubebuilder v4.15.0 (go/v4)

Browse Source 
Bootstraps the empty repo into a kubebuilder go/v4 project: group
crawl.example.com, version v1alpha1, kind Proxy (namespaced). Keeps the
existing module path and preserves the repo's Go/testing/changelog
conventions from CLAUDE.md untouched.

Drops the scaffolded GitHub Actions workflows since the remote is Gitea,
not GitHub. Everything else is default kubebuilder output, unmodified,
so later diffs stay reviewable against a known baseline.

Full implementation plan: docs/plans/2026-08-07-1747-proxy-operator.md

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Jan Novak
2026-08-07 20:17:45 +02:00
commit 076bc66ebe
61 changed files with 4854 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

0
docs/plans/.gitkeep Normal file
View File

504
docs/plans/2026-08-07-1747-proxy-operator.md Normal file
View File

@@ -0,0 +1,504 @@
# Plan: proxy-operator — Kubernetes operator for crawling-proxy VMs
**Created:** 2026-08-07 17:47
## Context
The crawling department runs a small fleet (tens) of HTTP proxy VMs across cloud
providers to dodge rate limiting. This builds a production-quality **prototype**
operator making each proxy VM a first-class Kubernetes object: GitOps-managed,
actively health-checked *through the proxy*, and discoverable by crawler clients via
an HTTP list/lease API.
The repo is empty — `go.mod`, `CLAUDE.md`, `CHANGELOG.md`, `.claude/`, and the spec at
[docs/prompts/__initial-prompt.md](../prompts/__initial-prompt.md). Zero commits.
All greenfield. Governing principle: **proxies are immutable cattle** — any meaningful
spec change deletes and recreates the VM. No in-place update logic.
### Decisions locked with the user
| Question | Decision |
|---|---|
| Module path | Keep `gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator` (spec's `github.com/CHANGEME/...` was a placeholder) |
| API group | `crawl.example.com`, `v1alpha1`, kind `Proxy`, **namespaced** |
| Git flow | Scaffold commit, then work on `feat/proxy-operator`, MR via `tea`, no merge/delete from CLI |
| kubebuilder | `go install sigs.k8s.io/kubebuilder/v4/cmd/kubebuilder@v4.15.0` |
| Mock health | Mock provider runs a **real in-process HTTP CONNECT proxy** per instance, so healthchecks genuinely pass and the kind demo is truly end-to-end |
| Verification | vet + unit + envtest, then a throwaway kind cluster running the README quickstart, then delete it |
### Verified environment — no version substitutions needed
Go 1.26.4 · kubebuilder v4.15.0 · controller-runtime v0.24.1 · k8s.io/* v0.36.3 ·
controller-tools v0.21.0 · compute v1.65.0 · kind v0.32.0 · kubectl v1.36.1 · docker
29.6.2 · gcloud present. envtest bundles exist for k8s **1.36.0 and 1.36.2** on
darwin/arm64 — the scaffold Makefile derives the *minor* (`1.36`) from `k8s.io/api`
and resolves the latest patch; do not "fix" it to 1.36.3, which has no bundle.
Every pin in the spec is satisfiable. The README will say so explicitly rather than
omitting the substitutions section.
### Milestone order — each ends green on `go build ./... && go vet ./...`
1. Scaffold + pins → 2. `api/v1alpha1` + CEL → 3. `internal/provider` + mock →
4. reconciler + envtest → 5. health engine → 6. lease + discovery → 7. GCP provider →
8. orphan GC + metrics → 9. `cmd/main.go` wiring + `config/` → 10. docs + kind run.
---
## Step 0 — Branch and scaffold
```bash
git checkout -b feat/proxy-operator # main is unborn; branch starts empty
go install sigs.k8s.io/kubebuilder/v4/cmd/kubebuilder@v4.15.0
kubebuilder init --domain example.com \
--repo gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator --plugins go/v4
kubebuilder create api --group crawl --version v1alpha1 --kind Proxy --resource --controller
make manifests generate
```
**Trap:** `--domain crawl.example.com --group crawl` yields group
`crawl.crawl.example.com`. It must be `--domain example.com --group crawl`.
If `init` refuses the non-empty directory, scaffold into an empty temp dir with the
identical `--repo` and copy the tree in — don't fight the emptiness check. Copy this
plan to `docs/plans/2026-08-07-1747-proxy-operator.md` per CLAUDE.md. **Commit the
untouched scaffold on its own** so every later diff is reviewable.
Post-scaffold hand-edits: `CONTROLLER_TOOLS_VERSION ?= v0.21.0` in the Makefile (CEL
emission at the 1.36 API level); add a `run-mock` target; delete the scaffolded
`.github/workflows/` (the remote is Gitea).
---
## Step 1 — API types (`api/v1alpha1/proxy_types.go`)
Structs exactly as the spec dictates, with these **four corrections that are silent
bugs otherwise**:
- **`MaxLeases *int32`**, not `int32`. With a value type + `omitempty` + `default=5`,
an explicit `0` is dropped on any Go round-trip and re-defaulted to 5 — "0 =
unleasable" becomes unreachable.
- **`HealthCheck *HealthCheckSpec` needs `+kubebuilder:default={}`.** Structural
defaulting only descends into values that exist; without it a nil `healthCheck` gets
*none* of its nested defaults and the spec's "all defaulted" quietly fails.
- `+kubebuilder:validation:MinLength=1` on `Provider` and `CloudInit.Inline`, so an
explicit `""` fails OpenAPI validation and the CEL `has()` rules stay simple.
- Conditions get `+listType=map +listMapKey=type`.
**CEL at the `ProxySpec` struct level** — cross-field rules cannot live on a field:
```go
// +kubebuilder:validation:XValidation:rule="self.mode == oldSelf.mode",message="mode is immutable"
// +kubebuilder:validation:XValidation:rule="has(self.provider) == has(oldSelf.provider) && (!has(self.provider) || self.provider == oldSelf.provider)",message="provider is immutable"
// +kubebuilder:validation:XValidation:rule="self.mode != 'Managed' || has(self.provider)",message="provider is required when mode is Managed"
// +kubebuilder:validation:XValidation:rule="self.mode != 'External' || !has(self.provider)",message="provider must not be set when mode is External"
// +kubebuilder:validation:XValidation:rule="self.mode != 'External' || has(self.endpoint)",message="endpoint is required when mode is External"
// +kubebuilder:validation:XValidation:rule="self.mode != 'Managed' || !has(self.endpoint)",message="endpoint must not be set when mode is Managed"
```
plus on `CloudInitSpec`:
`rule="has(self.inline) != has(self.secretRef)"` (CEL `!=` on booleans is XOR).
Semantics that matter: a rule mentioning `oldSelf` is **skipped on CREATE**, so
immutability and required-iff must be *separate markers* — `&&`-ing them together
would skip the required-iff check on create. The immutability rule uses the
`has(self.x) == has(oldSelf.x) && ...` form because `Provider` is optional and a
field-level rule wouldn't fire when the field is absent on either side.
Defaults: `port=3128`, `maxLeases=5`, probeURL `https://www.gstatic.com/generate_204`,
interval 30s, timeout 5s, failureThreshold 3, successThreshold 1,
expectedStatusCodes `{200,204}`. Printer columns Mode/Provider/Phase/IP/Healthy/Age,
`shortName=px`, `+kubebuilder:subresource:status`.
Pure helpers in `helpers.go` (unit-tested): `EffectivePort()`, `EffectiveHost()`,
`HealthCheckOrDefault()`, `MaxLeasesOrDefault()`. **Port lives in two places**
(`spec.port` for Managed, `spec.endpoint.port` for External) — one helper, used by
health, discovery, and the hash.
---
## Step 2 — Provider contract (`internal/provider/`)
```
internal/provider/{provider,errors,name,config,metrics}.go
internal/provider/registry/registry.go # type→constructor — SEPARATE package
internal/provider/{mock,gcp}/
```
**Import-cycle trap:** a registry inside `internal/provider` would have to import
`internal/provider/mock`, which imports `internal/provider`. `init()` self-registration
is banned by CLAUDE.md, so the registry goes in its own leaf-importing package.
`Instance` needs **two fields the spec omits**, or orphan GC is unimplementable:
`UID string` (from the label, for the liveness match) and `CreatedAt time.Time` (for
the "skip < 10 min" rule).
**Error taxonomy** — the key trick is multi-value unwrap:
```go
type Error struct{ Class error; Op, Provider, ID string; Err error }
func (e *Error) Unwrap() []error { return []error{e.Class, e.Err} }
func Class(err error) error // returns the sentinel; unclassified → ErrTransient
```
So `errors.Is(err, ErrQuotaExceeded)` **and** `errors.As(err, &googleapiErr)` both work
on the same value. `Class()` defaulting to `ErrTransient` matters: retrying is always
safer than latching `Failed`.
**Deterministic naming** — SHA-256, first 10 bytes, RFC 4648 base32 lowercased,
unpadded → 16 chars, `proxy-` + that = 22 total:
```go
func NameFromUID(uid types.UID) string
```
GCP requires `^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$` ≤63. base32 lowercases to `[a-z2-7]`
— all legal; base64's `+`/`/`/uppercase are not, and hex would need 20 chars for the
same 80 bits. 80 bits → birthday collision at ~2^40 objects against a fleet of tens.
22 chars leaves headroom because GCP auto-names the boot disk after the instance.
`config.go` parses the YAML with `yaml.UnmarshalStrict` and validates at load
(non-empty + unique names, known type, type block present, gcp requires `project`).
**Fail fast from `main`** — never degrade.
---
## Step 3 — Mock provider (`internal/provider/mock/`)
**State is a pure function of an injectable clock — no background timers.** `Get`/
`ListByTag` derive state from `createdAt`/`deletedAt` vs `now()`:
`< provisionDelay` → Provisioning; else Running; `deletedAt` set and `< deleteDelay` →
Terminated; beyond that → purged, `ErrNotFound`. Deterministic under a fake clock,
correct under the real one, and no goroutine lifecycle to leak.
**Real proxy listener (the user's decision):** when a record first reports `Running`,
lazily start an `http.Server` on `127.0.0.1:0` implementing HTTP `CONNECT` tunnelling
(hijack + bidirectional `io.Copy`) plus plain-HTTP forwarding, and report `127.0.0.1`
plus the real listener port. `Delete` shuts it down. This is what makes the health
engine's probe a genuine CONNECT through a real proxy, so the kind quickstart actually
reaches Ready and leasable. One `http.Server` per instance, bounded by fleet size.
Fault injection: config-driven `failNextCreates`/`failWith` for the demo, plus
`InjectCreateFailures(n int, class error)` for tests. `Create` is idempotent by name.
---
## Step 4 — Reconciler (`internal/controller/`)
A **state machine**: every reconcile derives one action from (spec, status, provider
`Get`). Intervals are struct fields, never consts, so envtest can shrink them to
milliseconds.
```go
func (r *ProxyReconciler) Reconcile(ctx, req) (res ctrl.Result, err error) {
// Get; base := p.DeepCopy()
// defer patchStatusIfChanged(ctx, base, &p) // one status write per reconcile, max
switch {
case !p.DeletionTimestamp.IsZero(): return r.reconcileDelete(ctx, &p)
case p.Spec.Mode == v1alpha1.ModeExternal: return r.reconcileExternal(ctx, &p)
default: return r.reconcileManaged(ctx, &p)
}
}
```
`patchStatusIfChanged` sets `observedGeneration` and `phase = computePhase(&p)`, then
issues nothing when `equality.Semantic.DeepEqual(base.Status, p.Status)`. `computePhase`
is pure and is the primary table-driven unit-test target.
### Action table (Managed) — `A` = hash annotation, `H` = computed hash
| deletionTS | ID | A vs H | provider.Get | action | result |
|---|---|---|---|---|---|
| no | — | — | — | finalizer absent → add it | `{}` (the Update re-triggers) |
| no | `""` | any | — | resolve cloud-init, `Create`, set ID + `A=H` | `RequeueAfter: ProvisioningPoll` |
| no | set | `==` | Provisioning | clear `ip`, Provisioned=False/Provisioning | `RequeueAfter: ProvisioningPoll` |
| no | set | `==` | Running | set `ip`, Provisioned=True/Created | `RequeueAfter: DriftPoll` |
| no | set | `==` | Stopped/Terminated | `Delete(ID)` — cattle, not pets | `RequeueAfter: DeletionPoll` |
| no | set | `==` | NotFound | clear ID+ip → next pass creates | `Requeue: true` |
| no | set | `!=`, `A != ""` | any | **replace**: `Delete(ID)`, Provisioned=False/Replacing | `RequeueAfter: DeletionPoll` |
| no | set | `!=`, `A == ""` | any | **adopt**: set `A=H`, no replacement | `Requeue: true` |
| no | set | `!=` | NotFound | clear ID+ip (status), then set `A=H` (metadata) | `Requeue: true` |
| yes | `""` | — | — | remove finalizer (orphan GC reaps any stray VM) | `{}` |
| yes | set | — | NotFound | remove finalizer | `{}` |
| yes | set | — | anything else | `Delete(ID)`, phase=Deleting | `RequeueAfter: DeletionPoll` |
External: no finalizer, no provider calls, `status.ip = spec.endpoint.host`,
Provisioned=True/ExternalEndpoint, health drives the rest.
**The trap the spec doesn't mention:** the instance name derives from the *CR UID*,
which does **not** change on a spec edit — so the replacement VM has the *same* name
as the one being deleted. Recreating immediately hits `409 alreadyExists` against a
still-deleting instance. Hence replacement **polls to `NotFound` before recreating**
(rows 7 → 9 → 2). Do not add the hash to the instance name; that breaks the spec's
naming contract and only buys blue/green, a non-goal.
**Crash-safety:** ID is never lost destructively. Even if status is wiped entirely, the
create branch calls `Create`, which finds the existing VM by deterministic name and
returns its ID. That's what makes deterministic naming load-bearing rather than
cosmetic.
**Adopt-on-empty-annotation is required** — otherwise the first deploy of an operator
version whose hash-input struct gained a field mass-replaces the whole fleet.
**Spec hash:** SHA-256 over canonical JSON of an explicit `{placement, cloudInit
(resolved content), port}` struct — explicit, not `ProxySpec` wholesale, to bound
upgrade churn. Because it covers *resolved* Secret content, rotating the Secret must
re-trigger: `Watches(&corev1.Secret{}, EnqueueRequestsFromMapFunc(proxiesForSecret))`
with the cache restricted to Secrets labelled `crawl.example.com/cloud-init=true`.
~30 lines, and it's the difference between immutable replacement working and being
silently stale.
**Requeue per class:** `ErrTransient`/unclassified → return the error (workqueue
backoff). `ErrQuotaExceeded` → condition + `RequeueAfter: 5m`, return **nil** (keeps it
off the backoff curve and out of the error log). `ErrPermanent` → phase `Failed`,
condition, return nil. `ErrNotFound` → never an error, a state-machine input.
Conditions via `apimeta.SetStatusCondition` — note it does **not** populate
`ObservedGeneration`, so pass it explicitly or every condition reports generation 0.
`MaxConcurrentReconciles: 3`.
---
## Step 5 — Health engine (`internal/health/`)
**Delivery: push transitions to the reconciler via channel + `source.Channel` (option
a).** Rationale for `docs/architecture.md` → Decisions: `status.phase` is derived from
*both* provisioning and health. Under direct-patch, two writers each compute `phase`
from half the picture and race on the same subresource — a lost-update/flapping bug.
Option (a) keeps exactly one writer of `.status`, makes "write only on transition" fall
out for free (the engine only *emits* on transition), and costs one channel plus a
read-only `Snapshot()` method. The engine owns health *state*; the reconciler owns
health *representation*.
Channel typed `event.TypedGenericEvent[client.Object]` so the untyped
`&handler.EnqueueRequestForObject{}` satisfies it. Non-blocking send with `default:` —
a wedged reconciler must never stall the probe loop; on drop, don't advance the
"reported" markers, so the next probe retries.
**Threading:** one scheduler goroutine on a 1 s ticker + a fixed pool of 8 workers fed
by a buffered channel. Each tick lists from the cache and enqueues proxies whose
`nextDue <= now` and that aren't in flight. At tens of proxies a per-second list-scan is
free and a timer wheel is unjustified complexity. Startup jitter seeds
`nextDue = now + rand(0, interval)` so a restart doesn't fire every probe at once.
**Probe client** — fresh transport per probe, `defer CloseIdleConnections()`:
```go
Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), DisableKeepAlives: true,
ForceAttemptHTTP2: false, TLSHandshakeTimeout: timeout,
ResponseHeaderTimeout: timeout, DialContext: (&net.Dialer{Timeout: timeout}).DialContext }
```
`DisableKeepAlives: true` is **load-bearing** — otherwise `net/http` caches the
established CONNECT tunnel and later probes never re-exercise CONNECT, which is exactly
the failure the spec wants caught.
**CONNECT semantics:** for the default `https://` probe URL the transport sends
`CONNECT host:443` then TLS-handshakes through the tunnel. A proxy that accepts TCP but
can't egress returns non-200 to CONNECT, and `client.Do` returns an **error**, not a
response. So the success predicate is `err == nil && slices.Contains(expected,
resp.StatusCode)` — both halves. Latency is wall time around `Do`, last-value.
**Threshold state:** `map[NamespacedName]*state` under a mutex. Entries are pruned each
tick against the cache list (no leak), and `state.uid` is compared to the CR's UID so a
delete+recreate of the same name doesn't inherit stale fail counters. On leader
handover, state is empty: seed `healthy` from the CR's existing `Healthy` condition so a
healthy proxy doesn't flap to Unknown, but leave counters at zero so a real transition
still needs a full `failureThreshold` run. Documented as a Decision.
**Suppression:** emit only on (a) first-ever result, (b) a threshold-crossing flip, or
(c) `|new − reported| > max(20ms, 0.5×reported)` **and** `now − lastReported > 60s`.
The spec's bare ">50% latency bucket change" is undefined at 0 and makes a proxy
jittering 40↔61 ms write status forever; the absolute floor plus rate limit is what
actually delivers "no unbounded status churn". Flagged in Decisions. Metrics are
observed on *every* probe — that's the right home for high-frequency signal.
Note for the README: transition-only writes mean `status.lastHealthCheckTime` is stale
by construction. It means "time of the last *status-affecting* probe"; true probe
recency lives in metrics.
---
## Step 6 — Lease store (`internal/lease/`)
**`Acquire` takes the candidate *set*, not a chosen proxy** — selection and insertion
must happen under one lock, or two concurrent requests both see "3 of 5 used" and
overcommit.
```go
Acquire(ctx, AcquireRequest{Candidates []Candidate; Target string; TTL time.Duration}) (*Lease, AcquireStats, error)
Release / Report / ActiveCount / Counts / ExpireLoop
```
One `sync.Mutex` for the whole store (tens of proxies, human-rate QPS; sharding is
premature), injectable clock, `byID` + `byProxy` + `cooldown[{proxy,target}]` maps.
Selection is a linear scan + `slices.SortFunc` on `(activeLeases asc, latency asc,
name asc)` — explicitly not a heap, and the third key makes it deterministic and
testable. `AcquireStats{Considered, AtCapacity, InCooldown}` feeds the 409 body.
**Expired-lease retention:** entries stay marked `expired` for the cooldown window
after TTL. `Acquire`/`ActiveCount` ignore them; `Report` still resolves them. Without
this, a report arriving just after the TTL lapses is silently dropped — exactly when a
proxy is being rate-limited, which is when the cooldown matters most.
---
## Step 7 — Discovery API (`internal/discovery/`)
**`NeedLeaderElection() = false`**, and ship `replicas: 1`. Verified in
controller-runtime's runnable ordering: caches start and sync *before* non-leader-
election runnables, so cache reads are safe. If it *were* leader-elected, non-leader
pods would refuse connections while still being Service endpoints. The 1-replica
constraint comes from lease state being per-process, which the spec already accepts —
both facts go in the README caveats.
stdlib `http.ServeMux` with Go 1.22 method+wildcard patterns:
`GET /v1/proxies`, `POST /v1/leases`, `DELETE /v1/leases/{id}`,
`POST /v1/leases/{id}/report`, plus unauthenticated `GET /healthz`.
Middleware outermost-first: recover → request-log → `MaxBytesReader(64KiB)` → bearer
auth. Empty `DISCOVERY_TOKEN` passes through **with a loud startup Warn** — in-cluster
that's a silent security hole otherwise. Token compared with
`subtle.ConstantTimeCompare`.
Shapes: list returns `{"proxies":[…],"count":N}`, empty is 200 not 404; lease grant is
201 `{leaseID, proxy, expiresAt, ttlSeconds}`; no match is 409
`{"error":"no_match","message":…,"considered":7,"atCapacity":2,"inCooldown":2,"unhealthy":3}`;
DELETE is always 204; report is 204, 400 on an unknown result value, 404 on a genuinely
unknown lease. All errors share `{"error":"<machine_code>","message":"<human>"}`.
Timeouts on `http.Server`, graceful `Shutdown` with 10 s grace on ctx cancel.
---
## Step 8 — GCP provider (`internal/provider/gcp/`)
Only `compute.NewInstancesRESTClient` (ADC), only Insert/Get/Delete/AggregatedList.
`Operation.Wait` is **never called** — `Create` returns as soon as the operation is
submitted, and `409 alreadyExists` is treated as success, which is what makes a repeat
call after a crash correct.
**providerID = `zones/<zone>/instances/<name>`** — zone-qualified so `Get`/`Delete` are
self-contained. The spec's `Get(ctx, providerID)` carries no zone, and re-reading
`spec.placement.zone` is wrong precisely when a zone edit is the replacement being
processed.
**Test seam is deliberately not an SDK mirror.** `compute.InstancesScopedListPairIterator`
has an unexported `nextFunc`, so a fake cannot construct one — the interface flattens
`AggregatedList` to a slice and returns operations as just their name. The primary unit
test needs no fake at all: `buildInsertRequest` is pure, asserted field-by-field
(machine-type URL, boot disk, `AccessConfigs[0] = {Name:"External NAT",
Type:"ONE_TO_ONE_NAT"}`, `Metadata.Items[user-data]`, GC labels, network tag).
`AggregatedList` needs `ReturnPartialSuccess: true` — otherwise one unreachable zone
fails the entire GC sweep. `RUNNING` **without** a NatIP maps to `Provisioning`, not
Running, so we never publish an empty IP. Error mapping: 404→NotFound;
429 / 403+`quotaExceeded`→Quota; 400/401/403-other→Permanent; 5xx/408/net→Transient;
unknown→Transient.
---
## Step 9 — Orphan GC + metrics
**GC** (`internal/gc/`) — `NeedLeaderElection() = true` (destructive, single-writer),
10 min ticker, first sweep one interval after start. Per provider `ListByTag`; on error
log and continue to the next provider, never abort the sweep. Kill an instance only if
it has our UID label, is older than `MinAge` (10 min), and its UID matches no CR.
Log every kill at Warn with provider, providerID, UID.
**A CR with a `deletionTimestamp` still counts as live** — that's what the spec's
"check deletionTimestamp semantics carefully" points at. Its finalizer owns the
deletion; GC racing it double-deletes. A UID is orphan-eligible only once the object is
fully gone.
**Namespace-scope guard:** if the cache is namespace-restricted but Proxies exist
elsewhere, GC would delete live VMs. It refuses to start unless an explicit
`--gc-allow-namespaced` flag is set.
**Metrics** (`internal/metrics/`) registered by an explicit `Register(...)` called from
`main` (no `init()`, per house rules; also lets tests use a fresh registry).
`proxy_operator_proxies{phase}` and `proxy_operator_leases_active` are **custom
Collectors** that read at scrape time — a reconcile-incremented gauge inevitably drifts
and leaks a series on delete. Per-proxy histogram/counter labels **must** be deleted
from the vec when the health engine GCs a state entry, or series leak forever.
`proxy_operator_provider_requests_total{provider,op,result}` comes from a
`provider.WithMetrics(name, p)` decorator — zero-cost instrumentation for the next five
providers, and the one place `Class()` is called for observability.
---
## Step 10 — Wiring, config, docs
`cmd/main.go`: flags `--providers-config` (required), `--discovery-addr`,
`--proxy-namespace`, `--health-workers`, `--gc-interval`, `--gc-min-age`,
`--lease-cooldown`, `--max-lease-ttl`. Order: load provider config (fail fast) → build
registry → manager → `mgr.Add` health engine, GC, lease expiry loop, discovery server →
`SetupWithManager`. Contexts from `ctrl.SetupSignalHandler()` throughout.
RBAC markers: proxies CRUD + status + finalizers, secrets get/list/watch, events
create/patch. `config/`: providers ConfigMap mount, `DISCOVERY_TOKEN` from a Secret,
containerPort 8090 + Service. Samples: `proxy_mock.yaml`, `proxy_gcp.yaml`,
`proxy_external.yaml`, `providers-config.yaml`.
`docs/architecture.md`: components table, ASCII data-flow diagram, and a **Decisions**
section covering the channel-vs-patch choice, replacement-polls-to-NotFound, base32
naming, discovery without leader election, single-mutex lease store, mock-runs-a-real-
proxy, leader-handover health seeding, the latency-suppression refinement, the
`lastHealthCheckTime` semantics, and the logr-not-slog deviation (CLAUDE.md says
`slog`, but `log.FromContext(ctx)` returns logr inside controller paths — noted, not
silently ignored).
`README.md`: 60-second architecture summary; copy-pasteable kind quickstart; GCP setup
(ADC, `roles/compute.instanceAdmin.v1`, plus `roles/iam.serviceAccountUser` if
attaching a service account); the **immutable-replacement caveat** (changing a proxy
changes its IP) and **lease-loss-on-restart caveat**, both prominent; a version-pins
note confirming no substitutions were needed. Then a `CHANGELOG.md` entry with a real
`date "+%Y-%m-%d %H:%M %Z"` timestamp.
---
## Step 11 — Tests
**envtest** (`internal/controller/`), mock provider + a fake `HealthSnapshotter`,
intervals shrunk to 50–200 ms, whole suite behind `testing.Short()`:
Managed→Ready; spec change → old mock instance gone and providerID changed; delete →
finalizer runs and instance removed; External → Ready on first health pass, no
finalizer; injected quota → `Provisioned=False/QuotaExceeded` and phase *not* Failed;
permanent error → Failed and no further provider calls; adopt (strip annotation →
restored, providerID unchanged); and the **CEL cases only a real API server can test** —
mode/provider mutation rejected, Managed-without-provider, External-without-endpoint,
cloudInit both/neither, and `healthCheck` omitted → nested defaults materialized (the
`default={}` assertion).
**Action-table unit tests** — the highest-value tests in the repo: `fake` client with
`WithStatusSubresource`, calling `Reconcile` directly, table-driven over every row of
the Step 4 table, asserting the returned `ctrl.Result`. (Documented caveat: the fake
client runs neither CEL nor defaulting — that's what the envtest CEL cases cover.)
**Units:** name derivation (idempotency, `^proxy-[a-z2-7]{16}$`, 10k-UID distinctness);
`Class()` mapping + `errors.Is`/`errors.As` through the multi-unwrap; config loading;
mock state at the delay boundary with a fake clock; `buildInsertRequest` field-by-field
+ GCP error classification + RUNNING-without-IP; `computePhase` truth table; `SpecHash`
stability *and* sensitivity; lease store (capacity, `MaxLeases=0`, least-loaded with
latency tie-break, cooldown with/without target, report on an expired-but-retained
lease, concurrent acquire under `-race` never exceeding `MaxLeases`); discovery
handlers over `httptest` + fake reader + real store; health thresholds against a real
CONNECT-capable `httptest` proxy stub.
Everything runs with `-race`.
---
## Verification
```bash
go vet ./... && make test && make build # unit + envtest, -race
kind create cluster --name proxy-operator-demo
make install
make run-mock & # --providers-config hack/providers-mock.yaml
kubectl apply -f config/samples/proxy_mock.yaml
kubectl get px -w # expect Ready with an IP
curl -s 'localhost:8090/v1/proxies?healthy=true' | jq
curl -s -XPOST localhost:8090/v1/leases -d '{"selector":{"geo":"eu"},"ttlSeconds":300}' | jq
curl -s -XPOST localhost:8090/v1/leases/<id>/report -d '{"result":"rate_limited","target":"example.com"}'
curl -si -XDELETE localhost:8090/v1/leases/<id> # 204, and 204 again
kubectl delete -f config/samples/proxy_mock.yaml # finalizer runs, object goes
kind delete cluster --name proxy-operator-demo
```
Success bar: a competent SRE clones the repo, follows the README, and holds a lease on a
healthy mock proxy in under 10 minutes.
Then commit on `feat/proxy-operator`, push with `-u`, open the MR with
`tea pr create --base main --head feat/proxy-operator`, print the URL. No merging or
branch deletion from the CLI.