# Architecture ## Components | Component | Package | Runs as | Leader-elected | Role | |---|---|---|---|---| | Proxy CRD + helpers | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers | | Reconciler | `internal/controller` | controller | yes (with the manager) | the state machine: provision, replace, delete, represent health | | Provider contract | `internal/provider` | library | — | `Provider` interface, error taxonomy, deterministic naming, config, metrics decorator | | kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) | | gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops | | Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events | | Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex | | Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` | | Orphan GC | `internal/gc` | Runnable | yes | deletes tagged instances whose CR is gone | | Metrics | `internal/metrics` | library | — | explicit registration, scrape-time collectors | | Composition root | `cmd/main.go` | binary | — | flags, provider registry, wires everything onto one manager | ## Event flow: cluster events → reconciler functions Which functions run in response to which Kubernetes cluster events, as wired in `internal/controller/proxy_controller.go`. ### 1. How cluster events reach the reconciler ```text KUBERNETES CLUSTER EVENTS (wiring: SetupWithManager, ───────────────────────── proxy_controller.go) Proxy CR created / spec edited / Secret created / health transition status patched / delete requested updated / deleted (engine, see §6) │ │ │ watch: For(&crawlv1alpha1.Proxy{}) watch: Watches( WatchesRawSource( │ &corev1.Secret{}, ...) source.Channel( │ │ r.HealthEvents, ...)) │ r.proxiesForSecret(ctx, secret) │ │ │ r.List(Proxies in namespace) │ │ │ keeps those whose │ │ │ spec.cloudInit.secretRef matches │ │ ▼ │ │ [reconcile.Request per matching Proxy] │ ▼ │ │ ┌────────────────────────────────────┴──────────────────────────┴──┐ │ controller-runtime workqueue │◄── RequeueAfter │ (dedup by namespace/name, rate-limited, │ timers │ MaxConcurrentReconciles: 3) │◄── error backoff └──────────────────────────┬────────────────────────────────────────┘ ▼ ProxyReconciler.Reconcile(ctx, req) ``` The Secret watch makes *rotating a cloud-init Secret* a first-class event: it re-enqueues every Proxy referencing that Secret, which is how secret rotation triggers VM replacement even though the Proxy spec is untouched. ### 2. Inside `Reconcile` — dispatch and the single status write ```text Reconcile(ctx, req) │ r.Get(ctx, req.NamespacedName, &p) ── fetch the Proxy (NotFound → done) │ base := p.DeepCopy() ── snapshot for the diff │ defer patchStatusIfChanged(ctx, base, &p) ──────────────────────────┐ │ │ ├─ p.DeletionTimestamp set ──► reconcileDelete(ctx, &p) │ ├─ p.Spec.Mode == External ──► reconcileExternal(ctx, &p) │ └─ otherwise (Managed) ──────► reconcileManaged(ctx, &p) │ ▼ patchStatusIfChanged (status.go) │ p.Status.ObservedGeneration = p.Generation │ p.Status.Phase = computePhase(&p) │ equality.Semantic.DeepEqual(base, p)? └─ changed → r.Status().Patch(...) ◄── the ONLY unchanged → no API call status write ``` ### 3. `reconcileManaged` — the state machine ```text reconcileManaged(ctx, p) │ ├─ controllerutil.AddFinalizer? ──► r.Update ──► return {} (watch event re-triggers) ├─ permanent-failure latch (FindStatusCondition == PermanentError │ at this generation) ──► return {} (silent until spec edit) ├─ r.Providers[p.Spec.Provider] missing ──► setProvisioned(PermanentError) → Failed │ ├─ resolveCloudInit(ctx, p) ──► r.Get(Secret) if secretRef (error → CloudInitError + backoff) ├─ hash := specHash(p, cloudInit) (spechash.go) │ ├─ status.providerID == "" ─────────► prov.Create(CreateRequest{Name: NameFromUID(p.UID), ...}) │ │ setSpecHash → r.Update (annotation) │ └ stage providerID + Provisioned=False/Provisioning │ ──► RequeueAfter: ProvisioningPoll │ ├─ annotation != hash, annotation == "" ──► adopt: setSpecHash → r.Update │ ──► RequeueAfter: RequeueNow ├─ annotation != hash, annotation != "" ──► replaceInstance: │ prov.Get ─ NotFound → setSpecHash, clear ID/IP │ │ ──► RequeueNow (next pass creates) │ └ exists → prov.Delete, Provisioned=False/Replacing │ ──► RequeueAfter: DeletionPoll │ └─ annotation == hash ──► prov.Get(providerID) ├─ ErrNotFound ──► clear ID/IP ──► RequeueNow (next pass creates) ├─ Provisioning ──► Provisioned=False ──► ProvisioningPoll ├─ Running ──► status.ip = inst.IP, │ Provisioned=True/Created, │ applyHealth (see §6) ──► DriftPoll └─ Stopped/Termin. ──► prov.Delete (cattle) ──► DeletionPoll any provider error ──► providerFailure(p, err) ── provider.Class(err): ├─ ErrQuotaExceeded ──► condition QuotaExceeded ──► RequeueAfter: QuotaRetry (nil error) ├─ ErrPermanent ──► condition PermanentError ──► phase Failed, no retry └─ ErrTransient ──► return err ──► workqueue exponential backoff ``` ### 4. `reconcileDelete` and `reconcileExternal` ```text reconcileDelete(ctx, p) reconcileExternal(ctx, p) ├─ no finalizer ──► return {} │ status.ip = spec.endpoint.host ├─ providerID == "" ──► RemoveFinalizer │ setProvisioned(True/ExternalEndpoint) │ → r.Update → object actually deleted │ applyHealth (see §6) ├─ prov.Get → ErrNotFound ──► RemoveFinalizer └─ return {} (no finalizer, │ → r.Update → object actually deleted no provider calls ever) └─ exists ──► prov.Delete → Provisioned=False/Deleting ──► RequeueAfter: DeletionPoll (poll until gone) ``` ### 5. What provider calls do in the outside world ```text kubernetes pod provider (internal/provider/kubernetes/) prov.Create ──► buildPod (pure) ──► client.Create(corev1.Pod) ─┐ these cause Pod events, prov.Get ──► client.Get(Pod) → phase/IP → InstanceState │ but the operator does NOT prov.Delete ──► client.Delete(Pod, tolerate NotFound) │ watch Pods — it observes prov.ListByTag ─► client.List(Pods by labels, all namespaces) ─┘ them by polling prov.Get on each RequeueAfter tick gcp provider (internal/provider/gcp/) — instances.{Insert,Get,Delete,AggregatedList}, nothing else prov.Create ──► buildInsertRequest (pure) ──► instances.Insert ─┐ fire-and-forget: 409 alreadyExists = success (idempotent retry) │ Operation.Wait is never prov.Get ──► instances.Get → status/NatIP → InstanceState │ called; readiness is RUNNING without NatIP = still Provisioning │ discovered by Get polls, prov.Delete ──► instances.Delete (404 = success) │ exactly like the pod prov.ListByTag ─► AggregatedList(label filter, ─┘ provider ReturnPartialSuccess: true) providerID = zones//instances/ — zone-qualified, so Get/Delete stay correct even mid-replacement after a zone edit ``` The reconciler never watches provider-side resources (Pods or GCP VMs). All instance-state observation is poll-based through the `Provider` interface, so the same flow works identically for a cloud API that has no watch mechanism at all. ### 6. Health engine (`internal/health/`) — probes and transitions The engine is a leader-elected manager Runnable with its own goroutines, independent of the workqueue. It owns health *state*; the reconciler owns its *representation* in status — that split keeps exactly one writer of `.status` and makes write-only-on-transition fall out for free. ```text Engine.Start(ctx) (engine.go) ├─ spawns Workers (8) probe goroutines ◄─┐ └─ ticker loop (Tick = 1s): │ jobs channel (non-blocking send; tick(ctx, now, jobs) │ saturated pool → retry next tick) │ Reader.List(Proxies) ── from the manager cache │ per proxy: skip if no IP/host or deleting (state pruned → │ a replaced instance starts with fresh counters) │ newState: seed verdict from an existing Healthy condition │ (leader handover), jitter first probe across the interval │ due && !inFlight ──► jobs ◄── probe worker picks up └ prune states for proxies gone from the cache │ probe(ctx, proxyURL, hc, tls) (probe.go) │ fresh transport per probe, DisableKeepAlives=true │ (load-bearing: keep-alives would cache the CONNECT │ tunnel and later probes would never re-exercise it) │ https probe URL ⇒ CONNECT through the proxy + TLS inside └ success = err == nil AND expected status code │ record(job, result, now) ── under one mutex │ counters: consecOK/consecFail; verdict flips only at │ successThreshold / failureThreshold │ emit ONLY on: first-ever verdict │ threshold flip │ │ latency Δ > max(20ms, 50% of reported) rate-limited │ to one report per MinReportInterval (60s) ▼ Events chan (buffered 64, non-blocking send; on drop the reported markers do NOT advance → next probe retries) │ ▼ source.Channel → workqueue → Reconcile (see §1) │ ▼ r.applyHealth(p) ── reads Engine.Snapshot(key) (status.go) stages the Healthy condition + latencyMillis + lastHealthCheckTime; computePhase turns Provisioned=True + Healthy=True/False into phase Ready / Unhealthy ``` Consequence worth knowing: `status.lastHealthCheckTime` is the time of the last *status-affecting* probe, not the most recent probe — suppressed probes deliberately never write status. True probe recency will live in metrics (Step 9). ### 7. Discovery + lease API (`internal/discovery/`, `internal/lease/`) HTTP-driven, not cluster-event-driven: crawler clients call in; the only Kubernetes interaction is reading Proxies from the manager's cache. The server is a non-leader-elected Runnable (all replicas would serve, but the deployment ships `replicas: 1` because lease state is per-process — an operator restart drops all leases and cooldowns, a documented caveat). Client-facing reference with request/response schemas and curl examples: [api.md](api.md). ```text crawler client │ Authorization: Bearer $DISCOVERY_TOKEN (empty token = auth disabled, loud startup warning) ▼ Server.handler() middleware, outermost first (server.go) recover → request-log → MaxBytesReader(64KiB) → bearer auth (constant-time; /healthz exempt) │ ├─ GET /healthz ──► 200 ok (unauthenticated) │ ├─ GET /v1/proxies?attr.k=v&healthy=true (handlers.go) │ Reader.List(Proxies) ── manager cache │ filter: attributes equality + Healthy condition │ + Store.Counts() for activeLeases │ ──► 200 {"proxies":[...], "count":N} (empty list is 200, not 404) │ ├─ POST /v1/leases {"selector":{...},"ttlSeconds":300,"target":"..."} │ Reader.List → filter selector; unhealthy matches counted, not offered │ Store.Acquire(healthy candidates, target, ttl) ── one lock: select+insert │ │ selection: fewest active leases, then latency, then name │ ├─ granted ──► 201 {leaseID, proxy:{...}, expiresAt, ttlSeconds} │ └─ ErrNoMatch ──► 409 {"error":"no_match", considered, atCapacity, │ inCooldown, unhealthy} │ ├─ DELETE /v1/leases/{id} ──► Store.Release ──► always 204 (idempotent) │ └─ POST /v1/leases/{id}/report {"result":"ok|rate_limited|banned","target":"..."} Store.Report ── rate_limited/banned ⇒ cooldown[{proxy,target}] for │ CooldownWindow (target falls back: report → lease → global) ├─ 204 │ 400 invalid_result │ 404 unknown_lease └─ an expired lease still resolves for CooldownWindow past its TTL — a late report lands exactly when the proxy is being rate-limited Store.Start(ctx) ── manager Runnable, NOT leader-elected: sweeps expired leases + cooldowns; correctness never depends on the sweep (every read checks ExpiresAt against the clock) ``` ### 8. Orphan GC (`internal/gc/`) — the crash-safety net Timer-driven, leader-elected (destructive ⇒ single writer). Exists for the one gap the reconciler cannot close alone: a crash after a provider Create but before the status write that records the instance. ```text Sweeper.Start(ctx) ── refuses to run when the cache is namespace- │ restricted unless --gc-allow-namespaced is explicit │ (an incomplete live set would "orphan" live VMs) └─ every Interval (10m; first sweep a full interval after start): sweep(ctx) │ Reader.List(Proxies) → live UID set │ List fails → skip the whole sweep (never guess) │ a CR with deletionTimestamp still counts as LIVE — its │ finalizer owns that deletion; GC racing it double-deletes └ per provider: ListByTag │ error → log, continue with the next provider └ delete only when ALL hold: has the proxy-operator-uid label (ownership proof) older than MinAge (10m) (not mid-create) UID matches no existing CR (truly orphaned) each kill logged loudly with provider, providerID, UID ``` ### 9. Metrics (`internal/metrics/`) Registered explicitly from `cmd/main.go` (no `init()`; tests use fresh registries). Two kinds: - **Scrape-time collectors** — `proxy_operator_proxies{phase}` and `proxy_operator_leases_active` read the cache / lease store at every scrape; reconcile-incremented gauges inevitably drift and leak series. - **Fed vectors** — `healthcheck_duration_seconds{proxy}` and `healthcheck_failures_total{proxy}` observe EVERY probe (status writes are transition-only; metrics carry the high-frequency signal), and the health engine deletes a proxy's series when it prunes its state; `lease_requests_total{outcome}` from the discovery handlers; `provider_requests_total{provider,op,result}` from the `provider.WithMetrics` decorator — the one place `Class()` is called purely for observability. Each consuming package defines its own small recorder interface (`health.ProbeMetrics`, `discovery.LeaseMetrics`, `provider.RequestRecorder`); `metrics.Metrics` satisfies all of them structurally, so no package other than `cmd/main.go` imports the metrics package. ## Decisions Judgment calls the spec left open, and deliberate deviations — recorded so they read as choices, not accidents. Chronological by build step. - **Registry takes its constructor map as a parameter** instead of holding a package-level map: avoids the provider⇄registry import cycle and puts the wiring at the composition root, where it is visible. - **Mock provider replaced by the kubernetes-pod provider** (user decision, mid-build): a simulated in-memory provider was too far from the real system to build confidence in. Local dev/CI now runs real `ubuntu/squid` pods (Canonical's actively maintained image, verified 50M+ pulls, pinned tag) in the operator's own cluster. Trade-off accepted: envtest has no kubelet, so end-to-end proof lives in the kind quickstart, and cluster pods share one egress IP — distinct egress paths remain the GCP provider's job. - **`RequeueAfter: RequeueNow` instead of the plan's `Requeue: true`:** `ctrl.Result.Requeue` is deprecated in controller-runtime v0.24; a fifth configurable interval (default 1s) keeps identical semantics and stays shrinkable in tests. - **Quota exhaustion is a wait, not a failure:** `ErrQuotaExceeded` sets a condition and requeues slowly (5m) with a nil error — off the backoff curve, out of the error log, and never `phase: Failed`. Only `ErrPermanent` latches Failed, keyed to the generation so a spec edit auto-recovers. - **The finalizer path never latches permanent failures:** a permanent error during deletion keeps retrying visibly instead — latching there would wedge the object forever with no path out but manual finalizer surgery. - **Health transitions travel reconciler-ward over a channel** (`source.Channel`), not direct status patches: `phase` derives from both provisioning and health, so two status writers would race and flap. One writer of status; the engine owns health *state*, the reconciler its *representation*; write-only-on-transition falls out for free. - **Health state seeds from the existing Healthy condition on leader handover** (verdict kept, counters zeroed, first probe jittered), so a healthy fleet doesn't flap to Unknown on restart — but a real transition still needs a full threshold run. A never-probed proxy skips the jitter and probes on the next tick: startup spread matters for restarts, not for a single new proxy. - **Latency suppression is `max(20ms, 50%)` + a 60s rate limit, and only while the verdict is healthy.** The spec's bare ">50% change" is undefined at 0 and lets a proxy jittering 40↔61ms write status forever; the healthy-only guard (found by test) stops a below-threshold success streak from emitting latency updates for a proxy still reported unhealthy. Consequence: `status.lastHealthCheckTime` means "last status-affecting probe" — true probe recency is in the metrics. - **Deterministic instance names** are `proxy-` + 16 chars of base32(SHA-256(CR UID)): legal for both GCP (`[a-z2-7]` ⊂ `[-a-z0-9]`, 22 ≤ 63 chars) and Pod names, 80 bits against birthday collisions at a fleet of tens. The replacement VM therefore has the *same name* as the one being deleted — which is why replacement polls to NotFound before recreating instead of racing a 409. - **`banned` and `rate_limited` share one cooldown window:** a second duration knob the spec doesn't ask for; the report's semantic difference is preserved in the API but not the store. - **Report targets fall back report → lease → global**, so a client that leased with a target can't accidentally poison the proxy's global pool by omitting the target in its report. - **The 409 body's `considered` counts unhealthy matches too** (the store only ever sees healthy candidates): `considered = atCapacity + inCooldown + unhealthy + eligible-but-outranked`, keeping the numbers additive for a human debugging "why no proxy?". - **TTLs above `--max-lease-ttl` are a 400, not a silent clamp** — a client asking for a week should find out. - **Discovery is not leader-elected and ships `replicas: 1`:** caches start before non-leader-election runnables (verified in controller-runtime's ordering), and a leader-elected server would leave non-leader replicas as broken Service endpoints. One replica because lease state is per-process. - **GCP `Create` requires zone, machineType, and image** and fails `ErrPermanent` naming the missing field — inventing machine-type defaults would silently create billable VMs of arbitrary shape. - **Unknown GCP instance statuses map to `Stopped`:** the reconciler's answer to Stopped is delete-and-recreate, the always-safe move for cattle when the API grows a new state. - **Kubernetes 403s classify as `ErrPermanent`** even though quota exhaustion also surfaces as 403 (indistinguishable from RBAC denial in `apierrors`): not hammering an API server that may never allow the request is the safer default; a real ResourceQuota 403 forgoes the gentler quota backoff. Documented at the classification site. - **GC kills log at Info with a `WARNING:` prefix** — logr has no Warn level; the plan's "log at Warn" is met in spirit with provider, providerID, and UID always attached. Same convention as the discovery server's empty-token warning. - **GC trusts only provable orphans:** instances without the UID label are never deleted, a CR with a deletionTimestamp still counts as live (its finalizer owns that deletion), and an unreadable Proxy list skips the whole sweep. The namespace guard refuses to sweep a namespace-restricted cache without `--gc-allow-namespaced`. - **Cloud-init Secrets must carry `crawl.example.com/cloud-init: "true"`:** the manager caches only labelled Secrets (the operator holds cluster-wide Secret read RBAC — an unrestricted cache would hold every Secret in scope). Unlabelled referenced Secrets are invisible by construction, surfacing as `CloudInitError`. - **Events RBAC from the plan is omitted:** nothing wires an EventRecorder in the prototype, and granting verbs nothing uses would be RBAC lint noise. Add the marker together with the recorder if events land later. - **logr, not slog, inside controller paths:** the repo convention says `slog`, but `log.FromContext(ctx)` hands controller-runtime's logr logger to everything running under the manager — fighting that would mean two logging systems in one process. Noted as a deviation rather than silently ignored.