Add the health engine: through-proxy probes, thresholds, channel-fed transitions
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1,11 +1,10 @@
|
||||
# Architecture
|
||||
|
||||
> **Status:** the operator is built through Step 4 (reconciler) of
|
||||
> **Status:** the operator is built through Step 5 (health engine) of
|
||||
> [docs/plans/2026-08-07-1747-proxy-operator.md](plans/2026-08-07-1747-proxy-operator.md).
|
||||
> This document currently covers the event/reconcile flow; the components
|
||||
> table and the Decisions section arrive with Step 10, and the diagrams
|
||||
> below grow as the health engine, lease store, discovery API, and orphan
|
||||
> GC land.
|
||||
> below grow as the lease store, discovery API, and orphan GC land.
|
||||
|
||||
## Event flow: cluster events → reconciler functions
|
||||
|
||||
@@ -18,23 +17,24 @@ wired in `internal/controller/proxy_controller.go`.
|
||||
KUBERNETES CLUSTER EVENTS (wiring: SetupWithManager,
|
||||
───────────────────────── proxy_controller.go)
|
||||
|
||||
Proxy CR created / spec edited / Secret created / updated / deleted
|
||||
status patched / delete requested │
|
||||
│ │
|
||||
watch: For(&crawlv1alpha1.Proxy{}) watch: Watches(&corev1.Secret{}, ...)
|
||||
│ │
|
||||
│ r.proxiesForSecret(ctx, secret)
|
||||
│ │ r.List(Proxies in secret's namespace)
|
||||
│ │ keeps those whose
|
||||
│ │ spec.cloudInit.secretRef.name matches
|
||||
│ ▼
|
||||
│ [reconcile.Request per matching Proxy]
|
||||
▼ │
|
||||
┌─────────────────────────────────────────────────┴──┐
|
||||
│ controller-runtime workqueue │◄── RequeueAfter timers
|
||||
│ (dedup by namespace/name, rate-limited, │ (from prior reconciles)
|
||||
│ MaxConcurrentReconciles: 3) │◄── error backoff retries
|
||||
└──────────────────────────┬──────────────────────────┘
|
||||
Proxy CR created / spec edited / Secret created / health transition
|
||||
status patched / delete requested updated / deleted (engine, see §6)
|
||||
│ │ │
|
||||
watch: For(&crawlv1alpha1.Proxy{}) watch: Watches( WatchesRawSource(
|
||||
│ &corev1.Secret{}, ...) source.Channel(
|
||||
│ │ r.HealthEvents, ...))
|
||||
│ r.proxiesForSecret(ctx, secret) │
|
||||
│ │ r.List(Proxies in namespace) │
|
||||
│ │ keeps those whose │
|
||||
│ │ spec.cloudInit.secretRef matches │
|
||||
│ ▼ │
|
||||
│ [reconcile.Request per matching Proxy] │
|
||||
▼ │ │
|
||||
┌────────────────────────────────────┴──────────────────────────┴──┐
|
||||
│ controller-runtime workqueue │◄── RequeueAfter
|
||||
│ (dedup by namespace/name, rate-limited, │ timers
|
||||
│ MaxConcurrentReconciles: 3) │◄── error backoff
|
||||
└──────────────────────────┬────────────────────────────────────────┘
|
||||
▼
|
||||
ProxyReconciler.Reconcile(ctx, req)
|
||||
```
|
||||
@@ -93,7 +93,8 @@ reconcileManaged(ctx, p)
|
||||
├─ ErrNotFound ──► clear ID/IP ──► RequeueNow (next pass creates)
|
||||
├─ Provisioning ──► Provisioned=False ──► ProvisioningPoll
|
||||
├─ Running ──► status.ip = inst.IP,
|
||||
│ Provisioned=True/Created ──► DriftPoll
|
||||
│ Provisioned=True/Created,
|
||||
│ applyHealth (see §6) ──► DriftPoll
|
||||
└─ Stopped/Termin. ──► prov.Delete (cattle) ──► DeletionPoll
|
||||
|
||||
any provider error ──► providerFailure(p, err) ── provider.Class(err):
|
||||
@@ -108,10 +109,10 @@ reconcileManaged(ctx, p)
|
||||
reconcileDelete(ctx, p) reconcileExternal(ctx, p)
|
||||
├─ no finalizer ──► return {} │ status.ip = spec.endpoint.host
|
||||
├─ providerID == "" ──► RemoveFinalizer │ setProvisioned(True/ExternalEndpoint)
|
||||
│ → r.Update → object actually deleted └─ return {} (no finalizer, no
|
||||
├─ prov.Get → ErrNotFound ──► RemoveFinalizer provider calls ever;
|
||||
│ → r.Update → object actually deleted the health engine —
|
||||
└─ exists ──► prov.Delete Step 5 — drives the rest)
|
||||
│ → r.Update → object actually deleted │ applyHealth (see §6)
|
||||
├─ prov.Get → ErrNotFound ──► RemoveFinalizer └─ return {} (no finalizer,
|
||||
│ → r.Update → object actually deleted no provider calls ever)
|
||||
└─ exists ──► prov.Delete
|
||||
→ Provisioned=False/Deleting
|
||||
──► RequeueAfter: DeletionPoll (poll until gone)
|
||||
```
|
||||
@@ -129,4 +130,56 @@ prov.ListByTag ─► client.List(Pods by labels, all namespaces) ─┘ them
|
||||
The reconciler never watches provider-side resources (Pods now, GCP VMs
|
||||
later). All instance-state observation is poll-based through the
|
||||
`Provider` interface, so the same flow works identically for a cloud API
|
||||
that has no watch mechanism at all.
|
||||
that has no watch mechanism at all.
|
||||
|
||||
### 6. Health engine (`internal/health/`) — probes and transitions
|
||||
|
||||
The engine is a leader-elected manager Runnable with its own goroutines,
|
||||
independent of the workqueue. It owns health *state*; the reconciler owns
|
||||
its *representation* in status — that split keeps exactly one writer of
|
||||
`.status` and makes write-only-on-transition fall out for free.
|
||||
|
||||
```text
|
||||
Engine.Start(ctx) (engine.go)
|
||||
├─ spawns Workers (8) probe goroutines ◄─┐
|
||||
└─ ticker loop (Tick = 1s): │ jobs channel (non-blocking send;
|
||||
tick(ctx, now, jobs) │ saturated pool → retry next tick)
|
||||
│ Reader.List(Proxies) ── from the manager cache
|
||||
│ per proxy: skip if no IP/host or deleting (state pruned →
|
||||
│ a replaced instance starts with fresh counters)
|
||||
│ newState: seed verdict from an existing Healthy condition
|
||||
│ (leader handover), jitter first probe across the interval
|
||||
│ due && !inFlight ──► jobs ◄── probe worker picks up
|
||||
└ prune states for proxies gone from the cache
|
||||
│
|
||||
probe(ctx, proxyURL, hc, tls) (probe.go)
|
||||
│ fresh transport per probe, DisableKeepAlives=true
|
||||
│ (load-bearing: keep-alives would cache the CONNECT
|
||||
│ tunnel and later probes would never re-exercise it)
|
||||
│ https probe URL ⇒ CONNECT through the proxy + TLS inside
|
||||
└ success = err == nil AND expected status code
|
||||
│
|
||||
record(job, result, now) ── under one mutex
|
||||
│ counters: consecOK/consecFail; verdict flips only at
|
||||
│ successThreshold / failureThreshold
|
||||
│ emit ONLY on: first-ever verdict │ threshold flip │
|
||||
│ latency Δ > max(20ms, 50% of reported) rate-limited
|
||||
│ to one report per MinReportInterval (60s)
|
||||
▼
|
||||
Events chan (buffered 64, non-blocking send;
|
||||
on drop the reported markers do NOT advance → next probe retries)
|
||||
│
|
||||
▼
|
||||
source.Channel → workqueue → Reconcile (see §1)
|
||||
│
|
||||
▼
|
||||
r.applyHealth(p) ── reads Engine.Snapshot(key) (status.go)
|
||||
stages the Healthy condition + latencyMillis +
|
||||
lastHealthCheckTime; computePhase turns Provisioned=True
|
||||
+ Healthy=True/False into phase Ready / Unhealthy
|
||||
```
|
||||
|
||||
Consequence worth knowing: `status.lastHealthCheckTime` is the time of the
|
||||
last *status-affecting* probe, not the most recent probe — suppressed
|
||||
probes deliberately never write status. True probe recency will live in
|
||||
metrics (Step 9).
|
||||
|
||||
Reference in New Issue
Block a user