186 lines
11 KiB
Markdown
186 lines
11 KiB
Markdown
# Architecture
|
|
|
|
> **Status:** the operator is built through Step 5 (health engine) of
|
|
> [docs/plans/2026-08-07-1747-proxy-operator.md](plans/2026-08-07-1747-proxy-operator.md).
|
|
> This document currently covers the event/reconcile flow; the components
|
|
> table and the Decisions section arrive with Step 10, and the diagrams
|
|
> below grow as the lease store, discovery API, and orphan GC land.
|
|
|
|
## Event flow: cluster events → reconciler functions
|
|
|
|
Which functions run in response to which Kubernetes cluster events, as
|
|
wired in `internal/controller/proxy_controller.go`.
|
|
|
|
### 1. How cluster events reach the reconciler
|
|
|
|
```text
|
|
KUBERNETES CLUSTER EVENTS (wiring: SetupWithManager,
|
|
───────────────────────── proxy_controller.go)
|
|
|
|
Proxy CR created / spec edited / Secret created / health transition
|
|
status patched / delete requested updated / deleted (engine, see §6)
|
|
│ │ │
|
|
watch: For(&crawlv1alpha1.Proxy{}) watch: Watches( WatchesRawSource(
|
|
│ &corev1.Secret{}, ...) source.Channel(
|
|
│ │ r.HealthEvents, ...))
|
|
│ r.proxiesForSecret(ctx, secret) │
|
|
│ │ r.List(Proxies in namespace) │
|
|
│ │ keeps those whose │
|
|
│ │ spec.cloudInit.secretRef matches │
|
|
│ ▼ │
|
|
│ [reconcile.Request per matching Proxy] │
|
|
▼ │ │
|
|
┌────────────────────────────────────┴──────────────────────────┴──┐
|
|
│ controller-runtime workqueue │◄── RequeueAfter
|
|
│ (dedup by namespace/name, rate-limited, │ timers
|
|
│ MaxConcurrentReconciles: 3) │◄── error backoff
|
|
└──────────────────────────┬────────────────────────────────────────┘
|
|
▼
|
|
ProxyReconciler.Reconcile(ctx, req)
|
|
```
|
|
|
|
The Secret watch makes *rotating a cloud-init Secret* a first-class event:
|
|
it re-enqueues every Proxy referencing that Secret, which is how secret
|
|
rotation triggers VM replacement even though the Proxy spec is untouched.
|
|
|
|
### 2. Inside `Reconcile` — dispatch and the single status write
|
|
|
|
```text
|
|
Reconcile(ctx, req)
|
|
│ r.Get(ctx, req.NamespacedName, &p) ── fetch the Proxy (NotFound → done)
|
|
│ base := p.DeepCopy() ── snapshot for the diff
|
|
│ defer patchStatusIfChanged(ctx, base, &p) ──────────────────────────┐
|
|
│ │
|
|
├─ p.DeletionTimestamp set ──► reconcileDelete(ctx, &p) │
|
|
├─ p.Spec.Mode == External ──► reconcileExternal(ctx, &p) │
|
|
└─ otherwise (Managed) ──────► reconcileManaged(ctx, &p) │
|
|
▼
|
|
patchStatusIfChanged (status.go)
|
|
│ p.Status.ObservedGeneration = p.Generation
|
|
│ p.Status.Phase = computePhase(&p)
|
|
│ equality.Semantic.DeepEqual(base, p)?
|
|
└─ changed → r.Status().Patch(...) ◄── the ONLY
|
|
unchanged → no API call status write
|
|
```
|
|
|
|
### 3. `reconcileManaged` — the state machine
|
|
|
|
```text
|
|
reconcileManaged(ctx, p)
|
|
│
|
|
├─ controllerutil.AddFinalizer? ──► r.Update ──► return {} (watch event re-triggers)
|
|
├─ permanent-failure latch (FindStatusCondition == PermanentError
|
|
│ at this generation) ──► return {} (silent until spec edit)
|
|
├─ r.Providers[p.Spec.Provider] missing ──► setProvisioned(PermanentError) → Failed
|
|
│
|
|
├─ resolveCloudInit(ctx, p) ──► r.Get(Secret) if secretRef (error → CloudInitError + backoff)
|
|
├─ hash := specHash(p, cloudInit) (spechash.go)
|
|
│
|
|
├─ status.providerID == "" ─────────► prov.Create(CreateRequest{Name: NameFromUID(p.UID), ...})
|
|
│ │ setSpecHash → r.Update (annotation)
|
|
│ └ stage providerID + Provisioned=False/Provisioning
|
|
│ ──► RequeueAfter: ProvisioningPoll
|
|
│
|
|
├─ annotation != hash, annotation == "" ──► adopt: setSpecHash → r.Update
|
|
│ ──► RequeueAfter: RequeueNow
|
|
├─ annotation != hash, annotation != "" ──► replaceInstance:
|
|
│ prov.Get ─ NotFound → setSpecHash, clear ID/IP
|
|
│ │ ──► RequeueNow (next pass creates)
|
|
│ └ exists → prov.Delete, Provisioned=False/Replacing
|
|
│ ──► RequeueAfter: DeletionPoll
|
|
│
|
|
└─ annotation == hash ──► prov.Get(providerID)
|
|
├─ ErrNotFound ──► clear ID/IP ──► RequeueNow (next pass creates)
|
|
├─ Provisioning ──► Provisioned=False ──► ProvisioningPoll
|
|
├─ Running ──► status.ip = inst.IP,
|
|
│ Provisioned=True/Created,
|
|
│ applyHealth (see §6) ──► DriftPoll
|
|
└─ Stopped/Termin. ──► prov.Delete (cattle) ──► DeletionPoll
|
|
|
|
any provider error ──► providerFailure(p, err) ── provider.Class(err):
|
|
├─ ErrQuotaExceeded ──► condition QuotaExceeded ──► RequeueAfter: QuotaRetry (nil error)
|
|
├─ ErrPermanent ──► condition PermanentError ──► phase Failed, no retry
|
|
└─ ErrTransient ──► return err ──► workqueue exponential backoff
|
|
```
|
|
|
|
### 4. `reconcileDelete` and `reconcileExternal`
|
|
|
|
```text
|
|
reconcileDelete(ctx, p) reconcileExternal(ctx, p)
|
|
├─ no finalizer ──► return {} │ status.ip = spec.endpoint.host
|
|
├─ providerID == "" ──► RemoveFinalizer │ setProvisioned(True/ExternalEndpoint)
|
|
│ → r.Update → object actually deleted │ applyHealth (see §6)
|
|
├─ prov.Get → ErrNotFound ──► RemoveFinalizer └─ return {} (no finalizer,
|
|
│ → r.Update → object actually deleted no provider calls ever)
|
|
└─ exists ──► prov.Delete
|
|
→ Provisioned=False/Deleting
|
|
──► RequeueAfter: DeletionPoll (poll until gone)
|
|
```
|
|
|
|
### 5. What provider calls do back in the cluster (kubernetes pod provider)
|
|
|
|
```text
|
|
prov.Create ──► buildPod (pure) ──► client.Create(corev1.Pod) ─┐ these cause Pod events,
|
|
prov.Get ──► client.Get(Pod) → phase/IP → InstanceState │ but the operator does NOT
|
|
prov.Delete ──► client.Delete(Pod, tolerate NotFound) │ watch Pods — it observes
|
|
prov.ListByTag ─► client.List(Pods by labels, all namespaces) ─┘ them by polling prov.Get
|
|
on each RequeueAfter tick
|
|
```
|
|
|
|
The reconciler never watches provider-side resources (Pods now, GCP VMs
|
|
later). All instance-state observation is poll-based through the
|
|
`Provider` interface, so the same flow works identically for a cloud API
|
|
that has no watch mechanism at all.
|
|
|
|
### 6. Health engine (`internal/health/`) — probes and transitions
|
|
|
|
The engine is a leader-elected manager Runnable with its own goroutines,
|
|
independent of the workqueue. It owns health *state*; the reconciler owns
|
|
its *representation* in status — that split keeps exactly one writer of
|
|
`.status` and makes write-only-on-transition fall out for free.
|
|
|
|
```text
|
|
Engine.Start(ctx) (engine.go)
|
|
├─ spawns Workers (8) probe goroutines ◄─┐
|
|
└─ ticker loop (Tick = 1s): │ jobs channel (non-blocking send;
|
|
tick(ctx, now, jobs) │ saturated pool → retry next tick)
|
|
│ Reader.List(Proxies) ── from the manager cache
|
|
│ per proxy: skip if no IP/host or deleting (state pruned →
|
|
│ a replaced instance starts with fresh counters)
|
|
│ newState: seed verdict from an existing Healthy condition
|
|
│ (leader handover), jitter first probe across the interval
|
|
│ due && !inFlight ──► jobs ◄── probe worker picks up
|
|
└ prune states for proxies gone from the cache
|
|
│
|
|
probe(ctx, proxyURL, hc, tls) (probe.go)
|
|
│ fresh transport per probe, DisableKeepAlives=true
|
|
│ (load-bearing: keep-alives would cache the CONNECT
|
|
│ tunnel and later probes would never re-exercise it)
|
|
│ https probe URL ⇒ CONNECT through the proxy + TLS inside
|
|
└ success = err == nil AND expected status code
|
|
│
|
|
record(job, result, now) ── under one mutex
|
|
│ counters: consecOK/consecFail; verdict flips only at
|
|
│ successThreshold / failureThreshold
|
|
│ emit ONLY on: first-ever verdict │ threshold flip │
|
|
│ latency Δ > max(20ms, 50% of reported) rate-limited
|
|
│ to one report per MinReportInterval (60s)
|
|
▼
|
|
Events chan (buffered 64, non-blocking send;
|
|
on drop the reported markers do NOT advance → next probe retries)
|
|
│
|
|
▼
|
|
source.Channel → workqueue → Reconcile (see §1)
|
|
│
|
|
▼
|
|
r.applyHealth(p) ── reads Engine.Snapshot(key) (status.go)
|
|
stages the Healthy condition + latencyMillis +
|
|
lastHealthCheckTime; computePhase turns Provisioned=True
|
|
+ Healthy=True/False into phase Ready / Unhealthy
|
|
```
|
|
|
|
Consequence worth knowing: `status.lastHealthCheckTime` is the time of the
|
|
last *status-affecting* probe, not the most recent probe — suppressed
|
|
probes deliberately never write status. True probe recency will live in
|
|
metrics (Step 9).
|