egress-proxies-operator/docs/architecture.md

Architecture

Components

Component	Package	Runs as	Leader-elected	Role
Proxy CRD + helpers	api/v1alpha1	types	—	Proxy spec/status, CEL validation, defaulting, pure helpers
Reconciler	internal/controller	controller	yes (with the manager)	the state machine: provision, replace, delete, represent health
Provider contract	internal/provider	library	—	Provider interface, error taxonomy, deterministic naming, config, metrics decorator
kubernetes provider	internal/provider/kubernetes	library	—	real Squid pods in this cluster (local dev/CI)
gcp provider	internal/provider/gcp	library	—	Compute Engine VMs, four API calls, fire-and-forget ops
Health engine	internal/health	Runnable	yes	through-the-proxy probes, thresholds, transition events
Lease store	internal/lease	Runnable (expiry sweep)	no	in-memory leases + cooldowns, single mutex
Discovery API	internal/discovery	Runnable	no	HTTP list/lease/release/report on :8090
Orphan GC	internal/gc	Runnable	yes	deletes tagged instances whose CR is gone
Metrics	internal/metrics	library	—	explicit registration, scrape-time collectors
Composition root	cmd/main.go	binary	—	flags, provider registry, wires everything onto one manager

Event flow: cluster events → reconciler functions

Which functions run in response to which Kubernetes cluster events, as wired in internal/controller/proxy_controller.go.

1. How cluster events reach the reconciler

KUBERNETES CLUSTER EVENTS                                (wiring: SetupWithManager,
─────────────────────────                                 proxy_controller.go)

 Proxy CR created / spec edited /     Secret created /           health transition
 status patched / delete requested    updated / deleted          (engine, see §6)
              │                              │                          │
   watch: For(&crawlv1alpha1.Proxy{})   watch: Watches(          WatchesRawSource(
              │                         &corev1.Secret{}, ...)   source.Channel(
              │                              │                    r.HealthEvents, ...))
              │                r.proxiesForSecret(ctx, secret)          │
              │                  │ r.List(Proxies in namespace)         │
              │                  │ keeps those whose                    │
              │                  │ spec.cloudInit.secretRef matches     │
              │                  ▼                                      │
              │                [reconcile.Request per matching Proxy]   │
              ▼                              │                          │
        ┌────────────────────────────────────┴──────────────────────────┴──┐
        │                controller-runtime workqueue                      │◄── RequeueAfter
        │  (dedup by namespace/name, rate-limited,                         │    timers
        │   MaxConcurrentReconciles: 3)                                    │◄── error backoff
        └──────────────────────────┬────────────────────────────────────────┘
                                   ▼
                    ProxyReconciler.Reconcile(ctx, req)

The Secret watch makes rotating a cloud-init Secret a first-class event: it re-enqueues every Proxy referencing that Secret, which is how secret rotation triggers VM replacement even though the Proxy spec is untouched.

2. Inside Reconcile — dispatch and the single status write

Reconcile(ctx, req)
  │ r.Get(ctx, req.NamespacedName, &p)          ── fetch the Proxy (NotFound → done)
  │ base := p.DeepCopy()                        ── snapshot for the diff
  │ defer patchStatusIfChanged(ctx, base, &p) ──────────────────────────┐
  │                                                                     │
  ├─ p.DeletionTimestamp set ──► reconcileDelete(ctx, &p)               │
  ├─ p.Spec.Mode == External ──► reconcileExternal(ctx, &p)             │
  └─ otherwise (Managed) ──────► reconcileManaged(ctx, &p)              │
                                                                        ▼
                                     patchStatusIfChanged  (status.go)
                                       │ p.Status.ObservedGeneration = p.Generation
                                       │ p.Status.Phase = computePhase(&p)
                                       │ equality.Semantic.DeepEqual(base, p)?
                                       └─ changed → r.Status().Patch(...)   ◄── the ONLY
                                          unchanged → no API call               status write

3. reconcileManaged — the state machine

reconcileManaged(ctx, p)
  │
  ├─ controllerutil.AddFinalizer? ──► r.Update ──► return {}        (watch event re-triggers)
  ├─ permanent-failure latch (FindStatusCondition == PermanentError
  │    at this generation) ──► return {}                            (silent until spec edit)
  ├─ r.Providers[p.Spec.Provider] missing ──► setProvisioned(PermanentError) → Failed
  │
  ├─ resolveCloudInit(ctx, p) ──► r.Get(Secret) if secretRef        (error → CloudInitError + backoff)
  ├─ hash := specHash(p, cloudInit)                                  (spechash.go)
  │
  ├─ status.providerID == "" ─────────► prov.Create(CreateRequest{Name: NameFromUID(p.UID), ...})
  │                                       │ setSpecHash → r.Update (annotation)
  │                                       └ stage providerID + Provisioned=False/Provisioning
  │                                         ──► RequeueAfter: ProvisioningPoll
  │
  ├─ annotation != hash, annotation == "" ──► adopt: setSpecHash → r.Update
  │                                           ──► RequeueAfter: RequeueNow
  ├─ annotation != hash, annotation != "" ──► replaceInstance:
  │                                             prov.Get ─ NotFound → setSpecHash, clear ID/IP
  │                                             │            ──► RequeueNow (next pass creates)
  │                                             └ exists  → prov.Delete, Provisioned=False/Replacing
  │                                                          ──► RequeueAfter: DeletionPoll
  │
  └─ annotation == hash ──► prov.Get(providerID)
        ├─ ErrNotFound      ──► clear ID/IP            ──► RequeueNow (next pass creates)
        ├─ Provisioning     ──► Provisioned=False      ──► ProvisioningPoll
        ├─ Running          ──► status.ip = inst.IP,
        │                       Provisioned=True/Created,
        │                       applyHealth (see §6)   ──► DriftPoll
        └─ Stopped/Termin.  ──► prov.Delete (cattle)   ──► DeletionPoll

  any provider error ──► providerFailure(p, err) ── provider.Class(err):
        ├─ ErrQuotaExceeded ──► condition QuotaExceeded ──► RequeueAfter: QuotaRetry (nil error)
        ├─ ErrPermanent     ──► condition PermanentError ──► phase Failed, no retry
        └─ ErrTransient     ──► return err ──► workqueue exponential backoff

4. reconcileDelete and reconcileExternal

reconcileDelete(ctx, p)                          reconcileExternal(ctx, p)
  ├─ no finalizer ──► return {}                    │ status.ip = spec.endpoint.host
  ├─ providerID == "" ──► RemoveFinalizer          │ setProvisioned(True/ExternalEndpoint)
  │     → r.Update → object actually deleted       │ applyHealth (see §6)
  ├─ prov.Get → ErrNotFound ──► RemoveFinalizer    └─ return {}   (no finalizer,
  │     → r.Update → object actually deleted                       no provider calls ever)
  └─ exists ──► prov.Delete
        → Provisioned=False/Deleting
        ──► RequeueAfter: DeletionPoll (poll until gone)

5. What provider calls do in the outside world

kubernetes pod provider (internal/provider/kubernetes/)
prov.Create ──► buildPod (pure) ──► client.Create(corev1.Pod)   ─┐ these cause Pod events,
prov.Get    ──► client.Get(Pod) → phase/IP → InstanceState       │ but the operator does NOT
prov.Delete ──► client.Delete(Pod, tolerate NotFound)            │ watch Pods — it observes
prov.ListByTag ─► client.List(Pods by labels, all namespaces)   ─┘ them by polling prov.Get
                                                                    on each RequeueAfter tick

gcp provider (internal/provider/gcp/) — instances.{Insert,Get,Delete,AggregatedList}, nothing else
prov.Create ──► buildInsertRequest (pure) ──► instances.Insert   ─┐ fire-and-forget:
                  409 alreadyExists = success (idempotent retry)  │ Operation.Wait is never
prov.Get    ──► instances.Get → status/NatIP → InstanceState      │ called; readiness is
                  RUNNING without NatIP = still Provisioning      │ discovered by Get polls,
prov.Delete ──► instances.Delete (404 = success)                  │ exactly like the pod
prov.ListByTag ─► AggregatedList(label filter,                   ─┘ provider
                    ReturnPartialSuccess: true)
providerID = zones/<zone>/instances/<name> — zone-qualified, so Get/Delete
stay correct even mid-replacement after a zone edit

The reconciler never watches provider-side resources (Pods or GCP VMs). All instance-state observation is poll-based through the Provider interface, so the same flow works identically for a cloud API that has no watch mechanism at all.

6. Health engine (internal/health/) — probes and transitions

The engine is a leader-elected manager Runnable with its own goroutines, independent of the workqueue. It owns health state; the reconciler owns its representation in status — that split keeps exactly one writer of .status and makes write-only-on-transition fall out for free.

Engine.Start(ctx)                                (engine.go)
  ├─ spawns Workers (8) probe goroutines ◄─┐
  └─ ticker loop (Tick = 1s):              │ jobs channel (non-blocking send;
       tick(ctx, now, jobs)                │ saturated pool → retry next tick)
         │ Reader.List(Proxies)  ── from the manager cache
         │ per proxy: skip if no IP/host or deleting (state pruned →
         │   a replaced instance starts with fresh counters)
         │ newState: seed verdict from an existing Healthy condition
         │   (leader handover), jitter first probe across the interval
         │ due && !inFlight ──► jobs ◄── probe worker picks up
         └ prune states for proxies gone from the cache
                                           │
              probe(ctx, proxyURL, hc, tls)   (probe.go)
                │ fresh transport per probe, DisableKeepAlives=true
                │   (load-bearing: keep-alives would cache the CONNECT
                │    tunnel and later probes would never re-exercise it)
                │ https probe URL ⇒ CONNECT through the proxy + TLS inside
                └ success = err == nil AND expected status code
                                           │
              record(job, result, now)     ── under one mutex
                │ counters: consecOK/consecFail; verdict flips only at
                │   successThreshold / failureThreshold
                │ emit ONLY on: first-ever verdict │ threshold flip │
                │   latency Δ > max(20ms, 50% of reported) rate-limited
                │   to one report per MinReportInterval (60s)
                ▼
        Events chan (buffered 64, non-blocking send;
        on drop the reported markers do NOT advance → next probe retries)
                │
                ▼
        source.Channel → workqueue → Reconcile        (see §1)
                │
                ▼
        r.applyHealth(p)  ── reads Engine.Snapshot(key)   (status.go)
              stages the Healthy condition + latencyMillis +
              lastHealthCheckTime; computePhase turns Provisioned=True
              + Healthy=True/False into phase Ready / Unhealthy

Consequence worth knowing: status.lastHealthCheckTime is the time of the last status-affecting probe, not the most recent probe — suppressed probes deliberately never write status. True probe recency will live in metrics (Step 9).

7. Discovery + lease API (internal/discovery/, internal/lease/)

HTTP-driven, not cluster-event-driven: crawler clients call in; the only Kubernetes interaction is reading Proxies from the manager's cache. The server is a non-leader-elected Runnable (all replicas would serve, but the deployment ships replicas: 1 because lease state is per-process — an operator restart drops all leases and cooldowns, a documented caveat). Client-facing reference with request/response schemas and curl examples: api.md.

crawler client
  │  Authorization: Bearer $DISCOVERY_TOKEN   (empty token = auth disabled, loud startup warning)
  ▼
Server.handler()      middleware, outermost first        (server.go)
  recover → request-log → MaxBytesReader(64KiB) → bearer auth (constant-time; /healthz exempt)
  │
  ├─ GET  /healthz            ──► 200 ok (unauthenticated)
  │
  ├─ GET  /v1/proxies?attr.k=v&healthy=true            (handlers.go)
  │     Reader.List(Proxies) ── manager cache
  │     filter: attributes equality + Healthy condition
  │     + Store.Counts() for activeLeases
  │     ──► 200 {"proxies":[...], "count":N}   (empty list is 200, not 404)
  │
  ├─ POST /v1/leases  {"selector":{...},"ttlSeconds":300,"target":"..."}
  │     Reader.List → filter selector; unhealthy matches counted, not offered
  │     Store.Acquire(healthy candidates, target, ttl)   ── one lock: select+insert
  │     │    selection: fewest active leases, then latency, then name
  │     ├─ granted ──► 201 {leaseID, proxy:{...}, expiresAt, ttlSeconds}
  │     └─ ErrNoMatch ──► 409 {"error":"no_match", considered, atCapacity,
  │                            inCooldown, unhealthy}
  │
  ├─ DELETE /v1/leases/{id} ──► Store.Release ──► always 204 (idempotent)
  │
  └─ POST /v1/leases/{id}/report {"result":"ok|rate_limited|banned","target":"..."}
        Store.Report ── rate_limited/banned ⇒ cooldown[{proxy,target}] for
        │               CooldownWindow (target falls back: report → lease → global)
        ├─ 204 │ 400 invalid_result │ 404 unknown_lease
        └─ an expired lease still resolves for CooldownWindow past its TTL —
           a late report lands exactly when the proxy is being rate-limited

Store.Start(ctx)  ── manager Runnable, NOT leader-elected: sweeps expired
                     leases + cooldowns; correctness never depends on the
                     sweep (every read checks ExpiresAt against the clock)

8. Orphan GC (internal/gc/) — the crash-safety net

Timer-driven, leader-elected (destructive ⇒ single writer). Exists for the one gap the reconciler cannot close alone: a crash after a provider Create but before the status write that records the instance.

Sweeper.Start(ctx)   ── refuses to run when the cache is namespace-
  │                     restricted unless --gc-allow-namespaced is explicit
  │                     (an incomplete live set would "orphan" live VMs)
  └─ every Interval (10m; first sweep a full interval after start):
       sweep(ctx)
         │ Reader.List(Proxies) → live UID set
         │   List fails → skip the whole sweep (never guess)
         │   a CR with deletionTimestamp still counts as LIVE — its
         │   finalizer owns that deletion; GC racing it double-deletes
         └ per provider: ListByTag
             │ error → log, continue with the next provider
             └ delete only when ALL hold:
                 has the proxy-operator-uid label   (ownership proof)
                 older than MinAge (10m)            (not mid-create)
                 UID matches no existing CR         (truly orphaned)
               each kill logged loudly with provider, providerID, UID

9. Metrics (internal/metrics/)

Registered explicitly from cmd/main.go (no init(); tests use fresh registries). Two kinds:

• Scrape-time collectors — proxy_operator_proxies{phase} and proxy_operator_leases_active read the cache / lease store at every scrape; reconcile-incremented gauges inevitably drift and leak series.
• Fed vectors — healthcheck_duration_seconds{proxy} and healthcheck_failures_total{proxy} observe EVERY probe (status writes are transition-only; metrics carry the high-frequency signal), and the health engine deletes a proxy's series when it prunes its state; lease_requests_total{outcome} from the discovery handlers; provider_requests_total{provider,op,result} from the provider.WithMetrics decorator — the one place Class() is called purely for observability.

Each consuming package defines its own small recorder interface (health.ProbeMetrics, discovery.LeaseMetrics, provider.RequestRecorder); metrics.Metrics satisfies all of them structurally, so no package other than cmd/main.go imports the metrics package.

Decisions

Judgment calls the spec left open, and deliberate deviations — recorded so they read as choices, not accidents. Chronological by build step.

• Registry takes its constructor map as a parameter instead of holding a package-level map: avoids the provider⇄registry import cycle and puts the wiring at the composition root, where it is visible.
• Mock provider replaced by the kubernetes-pod provider (user decision, mid-build): a simulated in-memory provider was too far from the real system to build confidence in. Local dev/CI now runs real ubuntu/squid pods (Canonical's actively maintained image, verified 50M+ pulls, pinned tag) in the operator's own cluster. Trade-off accepted: envtest has no kubelet, so end-to-end proof lives in the kind quickstart, and cluster pods share one egress IP — distinct egress paths remain the GCP provider's job.
• RequeueAfter: RequeueNow instead of the plan's Requeue: true: ctrl.Result.Requeue is deprecated in controller-runtime v0.24; a fifth configurable interval (default 1s) keeps identical semantics and stays shrinkable in tests.
• Quota exhaustion is a wait, not a failure: ErrQuotaExceeded sets a condition and requeues slowly (5m) with a nil error — off the backoff curve, out of the error log, and never phase: Failed. Only ErrPermanent latches Failed, keyed to the generation so a spec edit auto-recovers.
• The finalizer path never latches permanent failures: a permanent error during deletion keeps retrying visibly instead — latching there would wedge the object forever with no path out but manual finalizer surgery.
• Health transitions travel reconciler-ward over a channel (source.Channel), not direct status patches: phase derives from both provisioning and health, so two status writers would race and flap. One writer of status; the engine owns health state, the reconciler its representation; write-only-on-transition falls out for free.
• Health state seeds from the existing Healthy condition on leader handover (verdict kept, counters zeroed, first probe jittered), so a healthy fleet doesn't flap to Unknown on restart — but a real transition still needs a full threshold run. A never-probed proxy skips the jitter and probes on the next tick: startup spread matters for restarts, not for a single new proxy.
• Latency suppression is max(20ms, 50%) + a 60s rate limit, and only while the verdict is healthy. The spec's bare ">50% change" is undefined at 0 and lets a proxy jittering 40↔61ms write status forever; the healthy-only guard (found by test) stops a below-threshold success streak from emitting latency updates for a proxy still reported unhealthy. Consequence: status.lastHealthCheckTime means "last status-affecting probe" — true probe recency is in the metrics.
• Deterministic instance names are proxy- + 16 chars of base32(SHA-256(CR UID)): legal for both GCP ([a-z2-7] ⊂ [-a-z0-9], 22 ≤ 63 chars) and Pod names, 80 bits against birthday collisions at a fleet of tens. The replacement VM therefore has the same name as the one being deleted — which is why replacement polls to NotFound before recreating instead of racing a 409.
• banned and rate_limited share one cooldown window: a second duration knob the spec doesn't ask for; the report's semantic difference is preserved in the API but not the store.
• Report targets fall back report → lease → global, so a client that leased with a target can't accidentally poison the proxy's global pool by omitting the target in its report.
• The 409 body's considered counts unhealthy matches too (the store only ever sees healthy candidates): considered = atCapacity + inCooldown + unhealthy + eligible-but-outranked, keeping the numbers additive for a human debugging "why no proxy?".
• TTLs above --max-lease-ttl are a 400, not a silent clamp — a client asking for a week should find out.
• Discovery is not leader-elected and ships replicas: 1: caches start before non-leader-election runnables (verified in controller-runtime's ordering), and a leader-elected server would leave non-leader replicas as broken Service endpoints. One replica because lease state is per-process.
• GCP Create requires zone, machineType, and image and fails ErrPermanent naming the missing field — inventing machine-type defaults would silently create billable VMs of arbitrary shape.
• Unknown GCP instance statuses map to Stopped: the reconciler's answer to Stopped is delete-and-recreate, the always-safe move for cattle when the API grows a new state.
• Kubernetes 403s classify as ErrPermanent even though quota exhaustion also surfaces as 403 (indistinguishable from RBAC denial in apierrors): not hammering an API server that may never allow the request is the safer default; a real ResourceQuota 403 forgoes the gentler quota backoff. Documented at the classification site.
• GC kills log at Info with a WARNING: prefix — logr has no Warn level; the plan's "log at Warn" is met in spirit with provider, providerID, and UID always attached. Same convention as the discovery server's empty-token warning.
• GC trusts only provable orphans: instances without the UID label are never deleted, a CR with a deletionTimestamp still counts as live (its finalizer owns that deletion), and an unreadable Proxy list skips the whole sweep. The namespace guard refuses to sweep a namespace-restricted cache without --gc-allow-namespaced.
• Cloud-init Secrets must carry crawl.example.com/cloud-init: "true": the manager caches only labelled Secrets (the operator holds cluster-wide Secret read RBAC — an unrestricted cache would hold every Secret in scope). Unlabelled referenced Secrets are invisible by construction, surfacing as CloudInitError.
• Events RBAC from the plan is omitted: nothing wires an EventRecorder in the prototype, and granting verbs nothing uses would be RBAC lint noise. Add the marker together with the recorder if events land later.
• logr, not slog, inside controller paths: the repo convention says slog, but log.FromContext(ctx) hands controller-runtime's logr logger to everything running under the manager — fighting that would mean two logging systems in one process. Noted as a deviation rather than silently ignored.