Jan Novak
09845e4eaf
Full client-facing reference: auth, all routes with schemas and curl examples, selection/cooldown semantics, caveats. README and architecture.md link to it. Co-Authored-By: Claude <noreply@anthropic.com>
407 lines
24 KiB
Markdown
407 lines
24 KiB
Markdown
# Architecture
|
|
|
|
## Components
|
|
|
|
| Component | Package | Runs as | Leader-elected | Role |
|
|
|---|---|---|---|---|
|
|
| Proxy CRD + helpers | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers |
|
|
| Reconciler | `internal/controller` | controller | yes (with the manager) | the state machine: provision, replace, delete, represent health |
|
|
| Provider contract | `internal/provider` | library | — | `Provider` interface, error taxonomy, deterministic naming, config, metrics decorator |
|
|
| kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) |
|
|
| gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
|
|
| Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events |
|
|
| Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
|
|
| Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
|
|
| Orphan GC | `internal/gc` | Runnable | yes | deletes tagged instances whose CR is gone |
|
|
| Metrics | `internal/metrics` | library | — | explicit registration, scrape-time collectors |
|
|
| Composition root | `cmd/main.go` | binary | — | flags, provider registry, wires everything onto one manager |
|
|
|
|
## Event flow: cluster events → reconciler functions
|
|
|
|
Which functions run in response to which Kubernetes cluster events, as
|
|
wired in `internal/controller/proxy_controller.go`.
|
|
|
|
### 1. How cluster events reach the reconciler
|
|
|
|
```text
|
|
KUBERNETES CLUSTER EVENTS (wiring: SetupWithManager,
|
|
───────────────────────── proxy_controller.go)
|
|
|
|
Proxy CR created / spec edited / Secret created / health transition
|
|
status patched / delete requested updated / deleted (engine, see §6)
|
|
│ │ │
|
|
watch: For(&crawlv1alpha1.Proxy{}) watch: Watches( WatchesRawSource(
|
|
│ &corev1.Secret{}, ...) source.Channel(
|
|
│ │ r.HealthEvents, ...))
|
|
│ r.proxiesForSecret(ctx, secret) │
|
|
│ │ r.List(Proxies in namespace) │
|
|
│ │ keeps those whose │
|
|
│ │ spec.cloudInit.secretRef matches │
|
|
│ ▼ │
|
|
│ [reconcile.Request per matching Proxy] │
|
|
▼ │ │
|
|
┌────────────────────────────────────┴──────────────────────────┴──┐
|
|
│ controller-runtime workqueue │◄── RequeueAfter
|
|
│ (dedup by namespace/name, rate-limited, │ timers
|
|
│ MaxConcurrentReconciles: 3) │◄── error backoff
|
|
└──────────────────────────┬────────────────────────────────────────┘
|
|
▼
|
|
ProxyReconciler.Reconcile(ctx, req)
|
|
```
|
|
|
|
The Secret watch makes *rotating a cloud-init Secret* a first-class event:
|
|
it re-enqueues every Proxy referencing that Secret, which is how secret
|
|
rotation triggers VM replacement even though the Proxy spec is untouched.
|
|
|
|
### 2. Inside `Reconcile` — dispatch and the single status write
|
|
|
|
```text
|
|
Reconcile(ctx, req)
|
|
│ r.Get(ctx, req.NamespacedName, &p) ── fetch the Proxy (NotFound → done)
|
|
│ base := p.DeepCopy() ── snapshot for the diff
|
|
│ defer patchStatusIfChanged(ctx, base, &p) ──────────────────────────┐
|
|
│ │
|
|
├─ p.DeletionTimestamp set ──► reconcileDelete(ctx, &p) │
|
|
├─ p.Spec.Mode == External ──► reconcileExternal(ctx, &p) │
|
|
└─ otherwise (Managed) ──────► reconcileManaged(ctx, &p) │
|
|
▼
|
|
patchStatusIfChanged (status.go)
|
|
│ p.Status.ObservedGeneration = p.Generation
|
|
│ p.Status.Phase = computePhase(&p)
|
|
│ equality.Semantic.DeepEqual(base, p)?
|
|
└─ changed → r.Status().Patch(...) ◄── the ONLY
|
|
unchanged → no API call status write
|
|
```
|
|
|
|
### 3. `reconcileManaged` — the state machine
|
|
|
|
```text
|
|
reconcileManaged(ctx, p)
|
|
│
|
|
├─ controllerutil.AddFinalizer? ──► r.Update ──► return {} (watch event re-triggers)
|
|
├─ permanent-failure latch (FindStatusCondition == PermanentError
|
|
│ at this generation) ──► return {} (silent until spec edit)
|
|
├─ r.Providers[p.Spec.Provider] missing ──► setProvisioned(PermanentError) → Failed
|
|
│
|
|
├─ resolveCloudInit(ctx, p) ──► r.Get(Secret) if secretRef (error → CloudInitError + backoff)
|
|
├─ hash := specHash(p, cloudInit) (spechash.go)
|
|
│
|
|
├─ status.providerID == "" ─────────► prov.Create(CreateRequest{Name: NameFromUID(p.UID), ...})
|
|
│ │ setSpecHash → r.Update (annotation)
|
|
│ └ stage providerID + Provisioned=False/Provisioning
|
|
│ ──► RequeueAfter: ProvisioningPoll
|
|
│
|
|
├─ annotation != hash, annotation == "" ──► adopt: setSpecHash → r.Update
|
|
│ ──► RequeueAfter: RequeueNow
|
|
├─ annotation != hash, annotation != "" ──► replaceInstance:
|
|
│ prov.Get ─ NotFound → setSpecHash, clear ID/IP
|
|
│ │ ──► RequeueNow (next pass creates)
|
|
│ └ exists → prov.Delete, Provisioned=False/Replacing
|
|
│ ──► RequeueAfter: DeletionPoll
|
|
│
|
|
└─ annotation == hash ──► prov.Get(providerID)
|
|
├─ ErrNotFound ──► clear ID/IP ──► RequeueNow (next pass creates)
|
|
├─ Provisioning ──► Provisioned=False ──► ProvisioningPoll
|
|
├─ Running ──► status.ip = inst.IP,
|
|
│ Provisioned=True/Created,
|
|
│ applyHealth (see §6) ──► DriftPoll
|
|
└─ Stopped/Termin. ──► prov.Delete (cattle) ──► DeletionPoll
|
|
|
|
any provider error ──► providerFailure(p, err) ── provider.Class(err):
|
|
├─ ErrQuotaExceeded ──► condition QuotaExceeded ──► RequeueAfter: QuotaRetry (nil error)
|
|
├─ ErrPermanent ──► condition PermanentError ──► phase Failed, no retry
|
|
└─ ErrTransient ──► return err ──► workqueue exponential backoff
|
|
```
|
|
|
|
### 4. `reconcileDelete` and `reconcileExternal`
|
|
|
|
```text
|
|
reconcileDelete(ctx, p) reconcileExternal(ctx, p)
|
|
├─ no finalizer ──► return {} │ status.ip = spec.endpoint.host
|
|
├─ providerID == "" ──► RemoveFinalizer │ setProvisioned(True/ExternalEndpoint)
|
|
│ → r.Update → object actually deleted │ applyHealth (see §6)
|
|
├─ prov.Get → ErrNotFound ──► RemoveFinalizer └─ return {} (no finalizer,
|
|
│ → r.Update → object actually deleted no provider calls ever)
|
|
└─ exists ──► prov.Delete
|
|
→ Provisioned=False/Deleting
|
|
──► RequeueAfter: DeletionPoll (poll until gone)
|
|
```
|
|
|
|
### 5. What provider calls do in the outside world
|
|
|
|
```text
|
|
kubernetes pod provider (internal/provider/kubernetes/)
|
|
prov.Create ──► buildPod (pure) ──► client.Create(corev1.Pod) ─┐ these cause Pod events,
|
|
prov.Get ──► client.Get(Pod) → phase/IP → InstanceState │ but the operator does NOT
|
|
prov.Delete ──► client.Delete(Pod, tolerate NotFound) │ watch Pods — it observes
|
|
prov.ListByTag ─► client.List(Pods by labels, all namespaces) ─┘ them by polling prov.Get
|
|
on each RequeueAfter tick
|
|
|
|
gcp provider (internal/provider/gcp/) — instances.{Insert,Get,Delete,AggregatedList}, nothing else
|
|
prov.Create ──► buildInsertRequest (pure) ──► instances.Insert ─┐ fire-and-forget:
|
|
409 alreadyExists = success (idempotent retry) │ Operation.Wait is never
|
|
prov.Get ──► instances.Get → status/NatIP → InstanceState │ called; readiness is
|
|
RUNNING without NatIP = still Provisioning │ discovered by Get polls,
|
|
prov.Delete ──► instances.Delete (404 = success) │ exactly like the pod
|
|
prov.ListByTag ─► AggregatedList(label filter, ─┘ provider
|
|
ReturnPartialSuccess: true)
|
|
providerID = zones/<zone>/instances/<name> — zone-qualified, so Get/Delete
|
|
stay correct even mid-replacement after a zone edit
|
|
```
|
|
|
|
The reconciler never watches provider-side resources (Pods or GCP VMs).
|
|
All instance-state observation is poll-based through the `Provider`
|
|
interface, so the same flow works identically for a cloud API that has no
|
|
watch mechanism at all.
|
|
|
|
### 6. Health engine (`internal/health/`) — probes and transitions
|
|
|
|
The engine is a leader-elected manager Runnable with its own goroutines,
|
|
independent of the workqueue. It owns health *state*; the reconciler owns
|
|
its *representation* in status — that split keeps exactly one writer of
|
|
`.status` and makes write-only-on-transition fall out for free.
|
|
|
|
```text
|
|
Engine.Start(ctx) (engine.go)
|
|
├─ spawns Workers (8) probe goroutines ◄─┐
|
|
└─ ticker loop (Tick = 1s): │ jobs channel (non-blocking send;
|
|
tick(ctx, now, jobs) │ saturated pool → retry next tick)
|
|
│ Reader.List(Proxies) ── from the manager cache
|
|
│ per proxy: skip if no IP/host or deleting (state pruned →
|
|
│ a replaced instance starts with fresh counters)
|
|
│ newState: seed verdict from an existing Healthy condition
|
|
│ (leader handover), jitter first probe across the interval
|
|
│ due && !inFlight ──► jobs ◄── probe worker picks up
|
|
└ prune states for proxies gone from the cache
|
|
│
|
|
probe(ctx, proxyURL, hc, tls) (probe.go)
|
|
│ fresh transport per probe, DisableKeepAlives=true
|
|
│ (load-bearing: keep-alives would cache the CONNECT
|
|
│ tunnel and later probes would never re-exercise it)
|
|
│ https probe URL ⇒ CONNECT through the proxy + TLS inside
|
|
└ success = err == nil AND expected status code
|
|
│
|
|
record(job, result, now) ── under one mutex
|
|
│ counters: consecOK/consecFail; verdict flips only at
|
|
│ successThreshold / failureThreshold
|
|
│ emit ONLY on: first-ever verdict │ threshold flip │
|
|
│ latency Δ > max(20ms, 50% of reported) rate-limited
|
|
│ to one report per MinReportInterval (60s)
|
|
▼
|
|
Events chan (buffered 64, non-blocking send;
|
|
on drop the reported markers do NOT advance → next probe retries)
|
|
│
|
|
▼
|
|
source.Channel → workqueue → Reconcile (see §1)
|
|
│
|
|
▼
|
|
r.applyHealth(p) ── reads Engine.Snapshot(key) (status.go)
|
|
stages the Healthy condition + latencyMillis +
|
|
lastHealthCheckTime; computePhase turns Provisioned=True
|
|
+ Healthy=True/False into phase Ready / Unhealthy
|
|
```
|
|
|
|
Consequence worth knowing: `status.lastHealthCheckTime` is the time of the
|
|
last *status-affecting* probe, not the most recent probe — suppressed
|
|
probes deliberately never write status. True probe recency will live in
|
|
metrics (Step 9).
|
|
|
|
### 7. Discovery + lease API (`internal/discovery/`, `internal/lease/`)
|
|
|
|
HTTP-driven, not cluster-event-driven: crawler clients call in; the only
|
|
Kubernetes interaction is reading Proxies from the manager's cache. The
|
|
server is a non-leader-elected Runnable (all replicas would serve, but the
|
|
deployment ships `replicas: 1` because lease state is per-process — an
|
|
operator restart drops all leases and cooldowns, a documented caveat).
|
|
Client-facing reference with request/response schemas and curl examples:
|
|
[api.md](api.md).
|
|
|
|
```text
|
|
crawler client
|
|
│ Authorization: Bearer $DISCOVERY_TOKEN (empty token = auth disabled, loud startup warning)
|
|
▼
|
|
Server.handler() middleware, outermost first (server.go)
|
|
recover → request-log → MaxBytesReader(64KiB) → bearer auth (constant-time; /healthz exempt)
|
|
│
|
|
├─ GET /healthz ──► 200 ok (unauthenticated)
|
|
│
|
|
├─ GET /v1/proxies?attr.k=v&healthy=true (handlers.go)
|
|
│ Reader.List(Proxies) ── manager cache
|
|
│ filter: attributes equality + Healthy condition
|
|
│ + Store.Counts() for activeLeases
|
|
│ ──► 200 {"proxies":[...], "count":N} (empty list is 200, not 404)
|
|
│
|
|
├─ POST /v1/leases {"selector":{...},"ttlSeconds":300,"target":"..."}
|
|
│ Reader.List → filter selector; unhealthy matches counted, not offered
|
|
│ Store.Acquire(healthy candidates, target, ttl) ── one lock: select+insert
|
|
│ │ selection: fewest active leases, then latency, then name
|
|
│ ├─ granted ──► 201 {leaseID, proxy:{...}, expiresAt, ttlSeconds}
|
|
│ └─ ErrNoMatch ──► 409 {"error":"no_match", considered, atCapacity,
|
|
│ inCooldown, unhealthy}
|
|
│
|
|
├─ DELETE /v1/leases/{id} ──► Store.Release ──► always 204 (idempotent)
|
|
│
|
|
└─ POST /v1/leases/{id}/report {"result":"ok|rate_limited|banned","target":"..."}
|
|
Store.Report ── rate_limited/banned ⇒ cooldown[{proxy,target}] for
|
|
│ CooldownWindow (target falls back: report → lease → global)
|
|
├─ 204 │ 400 invalid_result │ 404 unknown_lease
|
|
└─ an expired lease still resolves for CooldownWindow past its TTL —
|
|
a late report lands exactly when the proxy is being rate-limited
|
|
|
|
Store.Start(ctx) ── manager Runnable, NOT leader-elected: sweeps expired
|
|
leases + cooldowns; correctness never depends on the
|
|
sweep (every read checks ExpiresAt against the clock)
|
|
```
|
|
|
|
### 8. Orphan GC (`internal/gc/`) — the crash-safety net
|
|
|
|
Timer-driven, leader-elected (destructive ⇒ single writer). Exists for the
|
|
one gap the reconciler cannot close alone: a crash after a provider Create
|
|
but before the status write that records the instance.
|
|
|
|
```text
|
|
Sweeper.Start(ctx) ── refuses to run when the cache is namespace-
|
|
│ restricted unless --gc-allow-namespaced is explicit
|
|
│ (an incomplete live set would "orphan" live VMs)
|
|
└─ every Interval (10m; first sweep a full interval after start):
|
|
sweep(ctx)
|
|
│ Reader.List(Proxies) → live UID set
|
|
│ List fails → skip the whole sweep (never guess)
|
|
│ a CR with deletionTimestamp still counts as LIVE — its
|
|
│ finalizer owns that deletion; GC racing it double-deletes
|
|
└ per provider: ListByTag
|
|
│ error → log, continue with the next provider
|
|
└ delete only when ALL hold:
|
|
has the proxy-operator-uid label (ownership proof)
|
|
older than MinAge (10m) (not mid-create)
|
|
UID matches no existing CR (truly orphaned)
|
|
each kill logged loudly with provider, providerID, UID
|
|
```
|
|
|
|
### 9. Metrics (`internal/metrics/`)
|
|
|
|
Registered explicitly from `cmd/main.go` (no `init()`; tests use fresh
|
|
registries). Two kinds:
|
|
|
|
- **Scrape-time collectors** — `proxy_operator_proxies{phase}` and
|
|
`proxy_operator_leases_active` read the cache / lease store at every
|
|
scrape; reconcile-incremented gauges inevitably drift and leak series.
|
|
- **Fed vectors** — `healthcheck_duration_seconds{proxy}` and
|
|
`healthcheck_failures_total{proxy}` observe EVERY probe (status writes
|
|
are transition-only; metrics carry the high-frequency signal), and the
|
|
health engine deletes a proxy's series when it prunes its state;
|
|
`lease_requests_total{outcome}` from the discovery handlers;
|
|
`provider_requests_total{provider,op,result}` from the
|
|
`provider.WithMetrics` decorator — the one place `Class()` is called
|
|
purely for observability.
|
|
|
|
Each consuming package defines its own small recorder interface
|
|
(`health.ProbeMetrics`, `discovery.LeaseMetrics`, `provider.RequestRecorder`);
|
|
`metrics.Metrics` satisfies all of them structurally, so no package other
|
|
than `cmd/main.go` imports the metrics package.
|
|
|
|
## Decisions
|
|
|
|
Judgment calls the spec left open, and deliberate deviations — recorded so
|
|
they read as choices, not accidents. Chronological by build step.
|
|
|
|
- **Registry takes its constructor map as a parameter** instead of holding
|
|
a package-level map: avoids the provider⇄registry import cycle and puts
|
|
the wiring at the composition root, where it is visible.
|
|
- **Mock provider replaced by the kubernetes-pod provider** (user
|
|
decision, mid-build): a simulated in-memory provider was too far from
|
|
the real system to build confidence in. Local dev/CI now runs real
|
|
`ubuntu/squid` pods (Canonical's actively maintained image, verified
|
|
50M+ pulls, pinned tag) in the operator's own cluster. Trade-off
|
|
accepted: envtest has no kubelet, so end-to-end proof lives in the kind
|
|
quickstart, and cluster pods share one egress IP — distinct egress
|
|
paths remain the GCP provider's job.
|
|
- **`RequeueAfter: RequeueNow` instead of the plan's `Requeue: true`:**
|
|
`ctrl.Result.Requeue` is deprecated in controller-runtime v0.24; a fifth
|
|
configurable interval (default 1s) keeps identical semantics and stays
|
|
shrinkable in tests.
|
|
- **Quota exhaustion is a wait, not a failure:** `ErrQuotaExceeded` sets a
|
|
condition and requeues slowly (5m) with a nil error — off the backoff
|
|
curve, out of the error log, and never `phase: Failed`. Only
|
|
`ErrPermanent` latches Failed, keyed to the generation so a spec edit
|
|
auto-recovers.
|
|
- **The finalizer path never latches permanent failures:** a permanent
|
|
error during deletion keeps retrying visibly instead — latching there
|
|
would wedge the object forever with no path out but manual finalizer
|
|
surgery.
|
|
- **Health transitions travel reconciler-ward over a channel**
|
|
(`source.Channel`), not direct status patches: `phase` derives from both
|
|
provisioning and health, so two status writers would race and flap. One
|
|
writer of status; the engine owns health *state*, the reconciler its
|
|
*representation*; write-only-on-transition falls out for free.
|
|
- **Health state seeds from the existing Healthy condition on leader
|
|
handover** (verdict kept, counters zeroed, first probe jittered), so a
|
|
healthy fleet doesn't flap to Unknown on restart — but a real
|
|
transition still needs a full threshold run. A never-probed proxy skips
|
|
the jitter and probes on the next tick: startup spread matters for
|
|
restarts, not for a single new proxy.
|
|
- **Latency suppression is `max(20ms, 50%)` + a 60s rate limit, and only
|
|
while the verdict is healthy.** The spec's bare ">50% change" is
|
|
undefined at 0 and lets a proxy jittering 40↔61ms write status forever;
|
|
the healthy-only guard (found by test) stops a below-threshold success
|
|
streak from emitting latency updates for a proxy still reported
|
|
unhealthy. Consequence: `status.lastHealthCheckTime` means "last
|
|
status-affecting probe" — true probe recency is in the metrics.
|
|
- **Deterministic instance names** are `proxy-` + 16 chars of
|
|
base32(SHA-256(CR UID)): legal for both GCP (`[a-z2-7]` ⊂ `[-a-z0-9]`,
|
|
22 ≤ 63 chars) and Pod names, 80 bits against birthday collisions at a
|
|
fleet of tens. The replacement VM therefore has the *same name* as the
|
|
one being deleted — which is why replacement polls to NotFound before
|
|
recreating instead of racing a 409.
|
|
- **`banned` and `rate_limited` share one cooldown window:** a second
|
|
duration knob the spec doesn't ask for; the report's semantic
|
|
difference is preserved in the API but not the store.
|
|
- **Report targets fall back report → lease → global**, so a client that
|
|
leased with a target can't accidentally poison the proxy's global pool
|
|
by omitting the target in its report.
|
|
- **The 409 body's `considered` counts unhealthy matches too** (the store
|
|
only ever sees healthy candidates): `considered = atCapacity +
|
|
inCooldown + unhealthy + eligible-but-outranked`, keeping the numbers
|
|
additive for a human debugging "why no proxy?".
|
|
- **TTLs above `--max-lease-ttl` are a 400, not a silent clamp** — a
|
|
client asking for a week should find out.
|
|
- **Discovery is not leader-elected and ships `replicas: 1`:** caches
|
|
start before non-leader-election runnables (verified in
|
|
controller-runtime's ordering), and a leader-elected server would leave
|
|
non-leader replicas as broken Service endpoints. One replica because
|
|
lease state is per-process.
|
|
- **GCP `Create` requires zone, machineType, and image** and fails
|
|
`ErrPermanent` naming the missing field — inventing machine-type
|
|
defaults would silently create billable VMs of arbitrary shape.
|
|
- **Unknown GCP instance statuses map to `Stopped`:** the reconciler's
|
|
answer to Stopped is delete-and-recreate, the always-safe move for
|
|
cattle when the API grows a new state.
|
|
- **Kubernetes 403s classify as `ErrPermanent`** even though quota
|
|
exhaustion also surfaces as 403 (indistinguishable from RBAC denial in
|
|
`apierrors`): not hammering an API server that may never allow the
|
|
request is the safer default; a real ResourceQuota 403 forgoes the
|
|
gentler quota backoff. Documented at the classification site.
|
|
- **GC kills log at Info with a `WARNING:` prefix** — logr has no Warn
|
|
level; the plan's "log at Warn" is met in spirit with provider,
|
|
providerID, and UID always attached. Same convention as the
|
|
discovery server's empty-token warning.
|
|
- **GC trusts only provable orphans:** instances without the UID label
|
|
are never deleted, a CR with a deletionTimestamp still counts as live
|
|
(its finalizer owns that deletion), and an unreadable Proxy list skips
|
|
the whole sweep. The namespace guard refuses to sweep a
|
|
namespace-restricted cache without `--gc-allow-namespaced`.
|
|
- **Cloud-init Secrets must carry `crawl.example.com/cloud-init: "true"`:**
|
|
the manager caches only labelled Secrets (the operator holds
|
|
cluster-wide Secret read RBAC — an unrestricted cache would hold every
|
|
Secret in scope). Unlabelled referenced Secrets are invisible by
|
|
construction, surfacing as `CloudInitError`.
|
|
- **Events RBAC from the plan is omitted:** nothing wires an
|
|
EventRecorder in the prototype, and granting verbs nothing uses would
|
|
be RBAC lint noise. Add the marker together with the recorder if events
|
|
land later.
|
|
- **logr, not slog, inside controller paths:** the repo convention says
|
|
`slog`, but `log.FromContext(ctx)` hands controller-runtime's logr
|
|
logger to everything running under the manager — fighting that would
|
|
mean two logging systems in one process. Noted as a deviation rather
|
|
than silently ignored.
|