14 KiB
14 KiB
Architecture
Status: the operator is built through Step 8 (GCP provider) of docs/plans/2026-08-07-1747-proxy-operator.md. This document currently covers the event/reconcile flow and the HTTP-driven lease/discovery path; the components table and the Decisions section arrive with Step 10, and the orphan-GC flow lands with Step 9.
Event flow: cluster events → reconciler functions
Which functions run in response to which Kubernetes cluster events, as
wired in internal/controller/proxy_controller.go.
1. How cluster events reach the reconciler
KUBERNETES CLUSTER EVENTS (wiring: SetupWithManager,
───────────────────────── proxy_controller.go)
Proxy CR created / spec edited / Secret created / health transition
status patched / delete requested updated / deleted (engine, see §6)
│ │ │
watch: For(&crawlv1alpha1.Proxy{}) watch: Watches( WatchesRawSource(
│ &corev1.Secret{}, ...) source.Channel(
│ │ r.HealthEvents, ...))
│ r.proxiesForSecret(ctx, secret) │
│ │ r.List(Proxies in namespace) │
│ │ keeps those whose │
│ │ spec.cloudInit.secretRef matches │
│ ▼ │
│ [reconcile.Request per matching Proxy] │
▼ │ │
┌────────────────────────────────────┴──────────────────────────┴──┐
│ controller-runtime workqueue │◄── RequeueAfter
│ (dedup by namespace/name, rate-limited, │ timers
│ MaxConcurrentReconciles: 3) │◄── error backoff
└──────────────────────────┬────────────────────────────────────────┘
▼
ProxyReconciler.Reconcile(ctx, req)
The Secret watch makes rotating a cloud-init Secret a first-class event: it re-enqueues every Proxy referencing that Secret, which is how secret rotation triggers VM replacement even though the Proxy spec is untouched.
2. Inside Reconcile — dispatch and the single status write
Reconcile(ctx, req)
│ r.Get(ctx, req.NamespacedName, &p) ── fetch the Proxy (NotFound → done)
│ base := p.DeepCopy() ── snapshot for the diff
│ defer patchStatusIfChanged(ctx, base, &p) ──────────────────────────┐
│ │
├─ p.DeletionTimestamp set ──► reconcileDelete(ctx, &p) │
├─ p.Spec.Mode == External ──► reconcileExternal(ctx, &p) │
└─ otherwise (Managed) ──────► reconcileManaged(ctx, &p) │
▼
patchStatusIfChanged (status.go)
│ p.Status.ObservedGeneration = p.Generation
│ p.Status.Phase = computePhase(&p)
│ equality.Semantic.DeepEqual(base, p)?
└─ changed → r.Status().Patch(...) ◄── the ONLY
unchanged → no API call status write
3. reconcileManaged — the state machine
reconcileManaged(ctx, p)
│
├─ controllerutil.AddFinalizer? ──► r.Update ──► return {} (watch event re-triggers)
├─ permanent-failure latch (FindStatusCondition == PermanentError
│ at this generation) ──► return {} (silent until spec edit)
├─ r.Providers[p.Spec.Provider] missing ──► setProvisioned(PermanentError) → Failed
│
├─ resolveCloudInit(ctx, p) ──► r.Get(Secret) if secretRef (error → CloudInitError + backoff)
├─ hash := specHash(p, cloudInit) (spechash.go)
│
├─ status.providerID == "" ─────────► prov.Create(CreateRequest{Name: NameFromUID(p.UID), ...})
│ │ setSpecHash → r.Update (annotation)
│ └ stage providerID + Provisioned=False/Provisioning
│ ──► RequeueAfter: ProvisioningPoll
│
├─ annotation != hash, annotation == "" ──► adopt: setSpecHash → r.Update
│ ──► RequeueAfter: RequeueNow
├─ annotation != hash, annotation != "" ──► replaceInstance:
│ prov.Get ─ NotFound → setSpecHash, clear ID/IP
│ │ ──► RequeueNow (next pass creates)
│ └ exists → prov.Delete, Provisioned=False/Replacing
│ ──► RequeueAfter: DeletionPoll
│
└─ annotation == hash ──► prov.Get(providerID)
├─ ErrNotFound ──► clear ID/IP ──► RequeueNow (next pass creates)
├─ Provisioning ──► Provisioned=False ──► ProvisioningPoll
├─ Running ──► status.ip = inst.IP,
│ Provisioned=True/Created,
│ applyHealth (see §6) ──► DriftPoll
└─ Stopped/Termin. ──► prov.Delete (cattle) ──► DeletionPoll
any provider error ──► providerFailure(p, err) ── provider.Class(err):
├─ ErrQuotaExceeded ──► condition QuotaExceeded ──► RequeueAfter: QuotaRetry (nil error)
├─ ErrPermanent ──► condition PermanentError ──► phase Failed, no retry
└─ ErrTransient ──► return err ──► workqueue exponential backoff
4. reconcileDelete and reconcileExternal
reconcileDelete(ctx, p) reconcileExternal(ctx, p)
├─ no finalizer ──► return {} │ status.ip = spec.endpoint.host
├─ providerID == "" ──► RemoveFinalizer │ setProvisioned(True/ExternalEndpoint)
│ → r.Update → object actually deleted │ applyHealth (see §6)
├─ prov.Get → ErrNotFound ──► RemoveFinalizer └─ return {} (no finalizer,
│ → r.Update → object actually deleted no provider calls ever)
└─ exists ──► prov.Delete
→ Provisioned=False/Deleting
──► RequeueAfter: DeletionPoll (poll until gone)
5. What provider calls do in the outside world
kubernetes pod provider (internal/provider/kubernetes/)
prov.Create ──► buildPod (pure) ──► client.Create(corev1.Pod) ─┐ these cause Pod events,
prov.Get ──► client.Get(Pod) → phase/IP → InstanceState │ but the operator does NOT
prov.Delete ──► client.Delete(Pod, tolerate NotFound) │ watch Pods — it observes
prov.ListByTag ─► client.List(Pods by labels, all namespaces) ─┘ them by polling prov.Get
on each RequeueAfter tick
gcp provider (internal/provider/gcp/) — instances.{Insert,Get,Delete,AggregatedList}, nothing else
prov.Create ──► buildInsertRequest (pure) ──► instances.Insert ─┐ fire-and-forget:
409 alreadyExists = success (idempotent retry) │ Operation.Wait is never
prov.Get ──► instances.Get → status/NatIP → InstanceState │ called; readiness is
RUNNING without NatIP = still Provisioning │ discovered by Get polls,
prov.Delete ──► instances.Delete (404 = success) │ exactly like the pod
prov.ListByTag ─► AggregatedList(label filter, ─┘ provider
ReturnPartialSuccess: true)
providerID = zones/<zone>/instances/<name> — zone-qualified, so Get/Delete
stay correct even mid-replacement after a zone edit
The reconciler never watches provider-side resources (Pods or GCP VMs).
All instance-state observation is poll-based through the Provider
interface, so the same flow works identically for a cloud API that has no
watch mechanism at all.
6. Health engine (internal/health/) — probes and transitions
The engine is a leader-elected manager Runnable with its own goroutines,
independent of the workqueue. It owns health state; the reconciler owns
its representation in status — that split keeps exactly one writer of
.status and makes write-only-on-transition fall out for free.
Engine.Start(ctx) (engine.go)
├─ spawns Workers (8) probe goroutines ◄─┐
└─ ticker loop (Tick = 1s): │ jobs channel (non-blocking send;
tick(ctx, now, jobs) │ saturated pool → retry next tick)
│ Reader.List(Proxies) ── from the manager cache
│ per proxy: skip if no IP/host or deleting (state pruned →
│ a replaced instance starts with fresh counters)
│ newState: seed verdict from an existing Healthy condition
│ (leader handover), jitter first probe across the interval
│ due && !inFlight ──► jobs ◄── probe worker picks up
└ prune states for proxies gone from the cache
│
probe(ctx, proxyURL, hc, tls) (probe.go)
│ fresh transport per probe, DisableKeepAlives=true
│ (load-bearing: keep-alives would cache the CONNECT
│ tunnel and later probes would never re-exercise it)
│ https probe URL ⇒ CONNECT through the proxy + TLS inside
└ success = err == nil AND expected status code
│
record(job, result, now) ── under one mutex
│ counters: consecOK/consecFail; verdict flips only at
│ successThreshold / failureThreshold
│ emit ONLY on: first-ever verdict │ threshold flip │
│ latency Δ > max(20ms, 50% of reported) rate-limited
│ to one report per MinReportInterval (60s)
▼
Events chan (buffered 64, non-blocking send;
on drop the reported markers do NOT advance → next probe retries)
│
▼
source.Channel → workqueue → Reconcile (see §1)
│
▼
r.applyHealth(p) ── reads Engine.Snapshot(key) (status.go)
stages the Healthy condition + latencyMillis +
lastHealthCheckTime; computePhase turns Provisioned=True
+ Healthy=True/False into phase Ready / Unhealthy
Consequence worth knowing: status.lastHealthCheckTime is the time of the
last status-affecting probe, not the most recent probe — suppressed
probes deliberately never write status. True probe recency will live in
metrics (Step 9).
7. Discovery + lease API (internal/discovery/, internal/lease/)
HTTP-driven, not cluster-event-driven: crawler clients call in; the only
Kubernetes interaction is reading Proxies from the manager's cache. The
server is a non-leader-elected Runnable (all replicas would serve, but the
deployment ships replicas: 1 because lease state is per-process — an
operator restart drops all leases and cooldowns, a documented caveat).
crawler client
│ Authorization: Bearer $DISCOVERY_TOKEN (empty token = auth disabled, loud startup warning)
▼
Server.handler() middleware, outermost first (server.go)
recover → request-log → MaxBytesReader(64KiB) → bearer auth (constant-time; /healthz exempt)
│
├─ GET /healthz ──► 200 ok (unauthenticated)
│
├─ GET /v1/proxies?attr.k=v&healthy=true (handlers.go)
│ Reader.List(Proxies) ── manager cache
│ filter: attributes equality + Healthy condition
│ + Store.Counts() for activeLeases
│ ──► 200 {"proxies":[...], "count":N} (empty list is 200, not 404)
│
├─ POST /v1/leases {"selector":{...},"ttlSeconds":300,"target":"..."}
│ Reader.List → filter selector; unhealthy matches counted, not offered
│ Store.Acquire(healthy candidates, target, ttl) ── one lock: select+insert
│ │ selection: fewest active leases, then latency, then name
│ ├─ granted ──► 201 {leaseID, proxy:{...}, expiresAt, ttlSeconds}
│ └─ ErrNoMatch ──► 409 {"error":"no_match", considered, atCapacity,
│ inCooldown, unhealthy}
│
├─ DELETE /v1/leases/{id} ──► Store.Release ──► always 204 (idempotent)
│
└─ POST /v1/leases/{id}/report {"result":"ok|rate_limited|banned","target":"..."}
Store.Report ── rate_limited/banned ⇒ cooldown[{proxy,target}] for
│ CooldownWindow (target falls back: report → lease → global)
├─ 204 │ 400 invalid_result │ 404 unknown_lease
└─ an expired lease still resolves for CooldownWindow past its TTL —
a late report lands exactly when the proxy is being rate-limited
Store.Start(ctx) ── manager Runnable, NOT leader-elected: sweeps expired
leases + cooldowns; correctness never depends on the
sweep (every read checks ExpiresAt against the clock)