# Architecture > **Status:** the operator is built through Step 6 (lease store) of > [docs/plans/2026-08-07-1747-proxy-operator.md](plans/2026-08-07-1747-proxy-operator.md). > This document currently covers the event/reconcile flow; the components > table and the Decisions section arrive with Step 10. The lease store > (`internal/lease/`) is HTTP-driven, not cluster-event-driven, so its > diagram lands together with the discovery API in Step 7; the orphan-GC > flow lands with Step 9. ## Event flow: cluster events → reconciler functions Which functions run in response to which Kubernetes cluster events, as wired in `internal/controller/proxy_controller.go`. ### 1. How cluster events reach the reconciler ```text KUBERNETES CLUSTER EVENTS (wiring: SetupWithManager, ───────────────────────── proxy_controller.go) Proxy CR created / spec edited / Secret created / health transition status patched / delete requested updated / deleted (engine, see §6) │ │ │ watch: For(&crawlv1alpha1.Proxy{}) watch: Watches( WatchesRawSource( │ &corev1.Secret{}, ...) source.Channel( │ │ r.HealthEvents, ...)) │ r.proxiesForSecret(ctx, secret) │ │ │ r.List(Proxies in namespace) │ │ │ keeps those whose │ │ │ spec.cloudInit.secretRef matches │ │ ▼ │ │ [reconcile.Request per matching Proxy] │ ▼ │ │ ┌────────────────────────────────────┴──────────────────────────┴──┐ │ controller-runtime workqueue │◄── RequeueAfter │ (dedup by namespace/name, rate-limited, │ timers │ MaxConcurrentReconciles: 3) │◄── error backoff └──────────────────────────┬────────────────────────────────────────┘ ▼ ProxyReconciler.Reconcile(ctx, req) ``` The Secret watch makes *rotating a cloud-init Secret* a first-class event: it re-enqueues every Proxy referencing that Secret, which is how secret rotation triggers VM replacement even though the Proxy spec is untouched. ### 2. Inside `Reconcile` — dispatch and the single status write ```text Reconcile(ctx, req) │ r.Get(ctx, req.NamespacedName, &p) ── fetch the Proxy (NotFound → done) │ base := p.DeepCopy() ── snapshot for the diff │ defer patchStatusIfChanged(ctx, base, &p) ──────────────────────────┐ │ │ ├─ p.DeletionTimestamp set ──► reconcileDelete(ctx, &p) │ ├─ p.Spec.Mode == External ──► reconcileExternal(ctx, &p) │ └─ otherwise (Managed) ──────► reconcileManaged(ctx, &p) │ ▼ patchStatusIfChanged (status.go) │ p.Status.ObservedGeneration = p.Generation │ p.Status.Phase = computePhase(&p) │ equality.Semantic.DeepEqual(base, p)? └─ changed → r.Status().Patch(...) ◄── the ONLY unchanged → no API call status write ``` ### 3. `reconcileManaged` — the state machine ```text reconcileManaged(ctx, p) │ ├─ controllerutil.AddFinalizer? ──► r.Update ──► return {} (watch event re-triggers) ├─ permanent-failure latch (FindStatusCondition == PermanentError │ at this generation) ──► return {} (silent until spec edit) ├─ r.Providers[p.Spec.Provider] missing ──► setProvisioned(PermanentError) → Failed │ ├─ resolveCloudInit(ctx, p) ──► r.Get(Secret) if secretRef (error → CloudInitError + backoff) ├─ hash := specHash(p, cloudInit) (spechash.go) │ ├─ status.providerID == "" ─────────► prov.Create(CreateRequest{Name: NameFromUID(p.UID), ...}) │ │ setSpecHash → r.Update (annotation) │ └ stage providerID + Provisioned=False/Provisioning │ ──► RequeueAfter: ProvisioningPoll │ ├─ annotation != hash, annotation == "" ──► adopt: setSpecHash → r.Update │ ──► RequeueAfter: RequeueNow ├─ annotation != hash, annotation != "" ──► replaceInstance: │ prov.Get ─ NotFound → setSpecHash, clear ID/IP │ │ ──► RequeueNow (next pass creates) │ └ exists → prov.Delete, Provisioned=False/Replacing │ ──► RequeueAfter: DeletionPoll │ └─ annotation == hash ──► prov.Get(providerID) ├─ ErrNotFound ──► clear ID/IP ──► RequeueNow (next pass creates) ├─ Provisioning ──► Provisioned=False ──► ProvisioningPoll ├─ Running ──► status.ip = inst.IP, │ Provisioned=True/Created, │ applyHealth (see §6) ──► DriftPoll └─ Stopped/Termin. ──► prov.Delete (cattle) ──► DeletionPoll any provider error ──► providerFailure(p, err) ── provider.Class(err): ├─ ErrQuotaExceeded ──► condition QuotaExceeded ──► RequeueAfter: QuotaRetry (nil error) ├─ ErrPermanent ──► condition PermanentError ──► phase Failed, no retry └─ ErrTransient ──► return err ──► workqueue exponential backoff ``` ### 4. `reconcileDelete` and `reconcileExternal` ```text reconcileDelete(ctx, p) reconcileExternal(ctx, p) ├─ no finalizer ──► return {} │ status.ip = spec.endpoint.host ├─ providerID == "" ──► RemoveFinalizer │ setProvisioned(True/ExternalEndpoint) │ → r.Update → object actually deleted │ applyHealth (see §6) ├─ prov.Get → ErrNotFound ──► RemoveFinalizer └─ return {} (no finalizer, │ → r.Update → object actually deleted no provider calls ever) └─ exists ──► prov.Delete → Provisioned=False/Deleting ──► RequeueAfter: DeletionPoll (poll until gone) ``` ### 5. What provider calls do back in the cluster (kubernetes pod provider) ```text prov.Create ──► buildPod (pure) ──► client.Create(corev1.Pod) ─┐ these cause Pod events, prov.Get ──► client.Get(Pod) → phase/IP → InstanceState │ but the operator does NOT prov.Delete ──► client.Delete(Pod, tolerate NotFound) │ watch Pods — it observes prov.ListByTag ─► client.List(Pods by labels, all namespaces) ─┘ them by polling prov.Get on each RequeueAfter tick ``` The reconciler never watches provider-side resources (Pods now, GCP VMs later). All instance-state observation is poll-based through the `Provider` interface, so the same flow works identically for a cloud API that has no watch mechanism at all. ### 6. Health engine (`internal/health/`) — probes and transitions The engine is a leader-elected manager Runnable with its own goroutines, independent of the workqueue. It owns health *state*; the reconciler owns its *representation* in status — that split keeps exactly one writer of `.status` and makes write-only-on-transition fall out for free. ```text Engine.Start(ctx) (engine.go) ├─ spawns Workers (8) probe goroutines ◄─┐ └─ ticker loop (Tick = 1s): │ jobs channel (non-blocking send; tick(ctx, now, jobs) │ saturated pool → retry next tick) │ Reader.List(Proxies) ── from the manager cache │ per proxy: skip if no IP/host or deleting (state pruned → │ a replaced instance starts with fresh counters) │ newState: seed verdict from an existing Healthy condition │ (leader handover), jitter first probe across the interval │ due && !inFlight ──► jobs ◄── probe worker picks up └ prune states for proxies gone from the cache │ probe(ctx, proxyURL, hc, tls) (probe.go) │ fresh transport per probe, DisableKeepAlives=true │ (load-bearing: keep-alives would cache the CONNECT │ tunnel and later probes would never re-exercise it) │ https probe URL ⇒ CONNECT through the proxy + TLS inside └ success = err == nil AND expected status code │ record(job, result, now) ── under one mutex │ counters: consecOK/consecFail; verdict flips only at │ successThreshold / failureThreshold │ emit ONLY on: first-ever verdict │ threshold flip │ │ latency Δ > max(20ms, 50% of reported) rate-limited │ to one report per MinReportInterval (60s) ▼ Events chan (buffered 64, non-blocking send; on drop the reported markers do NOT advance → next probe retries) │ ▼ source.Channel → workqueue → Reconcile (see §1) │ ▼ r.applyHealth(p) ── reads Engine.Snapshot(key) (status.go) stages the Healthy condition + latencyMillis + lastHealthCheckTime; computePhase turns Provisioned=True + Healthy=True/False into phase Ready / Unhealthy ``` Consequence worth knowing: `status.lastHealthCheckTime` is the time of the last *status-affecting* probe, not the most recent probe — suppressed probes deliberately never write status. True probe recency will live in metrics (Step 9).