Add the health engine: through-proxy probes, thresholds, channel-fed transitions
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1,11 +1,10 @@
|
||||
# Architecture
|
||||
|
||||
> **Status:** the operator is built through Step 4 (reconciler) of
|
||||
> **Status:** the operator is built through Step 5 (health engine) of
|
||||
> [docs/plans/2026-08-07-1747-proxy-operator.md](plans/2026-08-07-1747-proxy-operator.md).
|
||||
> This document currently covers the event/reconcile flow; the components
|
||||
> table and the Decisions section arrive with Step 10, and the diagrams
|
||||
> below grow as the health engine, lease store, discovery API, and orphan
|
||||
> GC land.
|
||||
> below grow as the lease store, discovery API, and orphan GC land.
|
||||
|
||||
## Event flow: cluster events → reconciler functions
|
||||
|
||||
@@ -18,23 +17,24 @@ wired in `internal/controller/proxy_controller.go`.
|
||||
KUBERNETES CLUSTER EVENTS (wiring: SetupWithManager,
|
||||
───────────────────────── proxy_controller.go)
|
||||
|
||||
Proxy CR created / spec edited / Secret created / updated / deleted
|
||||
status patched / delete requested │
|
||||
│ │
|
||||
watch: For(&crawlv1alpha1.Proxy{}) watch: Watches(&corev1.Secret{}, ...)
|
||||
│ │
|
||||
│ r.proxiesForSecret(ctx, secret)
|
||||
│ │ r.List(Proxies in secret's namespace)
|
||||
│ │ keeps those whose
|
||||
│ │ spec.cloudInit.secretRef.name matches
|
||||
│ ▼
|
||||
│ [reconcile.Request per matching Proxy]
|
||||
▼ │
|
||||
┌─────────────────────────────────────────────────┴──┐
|
||||
│ controller-runtime workqueue │◄── RequeueAfter timers
|
||||
│ (dedup by namespace/name, rate-limited, │ (from prior reconciles)
|
||||
│ MaxConcurrentReconciles: 3) │◄── error backoff retries
|
||||
└──────────────────────────┬──────────────────────────┘
|
||||
Proxy CR created / spec edited / Secret created / health transition
|
||||
status patched / delete requested updated / deleted (engine, see §6)
|
||||
│ │ │
|
||||
watch: For(&crawlv1alpha1.Proxy{}) watch: Watches( WatchesRawSource(
|
||||
│ &corev1.Secret{}, ...) source.Channel(
|
||||
│ │ r.HealthEvents, ...))
|
||||
│ r.proxiesForSecret(ctx, secret) │
|
||||
│ │ r.List(Proxies in namespace) │
|
||||
│ │ keeps those whose │
|
||||
│ │ spec.cloudInit.secretRef matches │
|
||||
│ ▼ │
|
||||
│ [reconcile.Request per matching Proxy] │
|
||||
▼ │ │
|
||||
┌────────────────────────────────────┴──────────────────────────┴──┐
|
||||
│ controller-runtime workqueue │◄── RequeueAfter
|
||||
│ (dedup by namespace/name, rate-limited, │ timers
|
||||
│ MaxConcurrentReconciles: 3) │◄── error backoff
|
||||
└──────────────────────────┬────────────────────────────────────────┘
|
||||
▼
|
||||
ProxyReconciler.Reconcile(ctx, req)
|
||||
```
|
||||
@@ -93,7 +93,8 @@ reconcileManaged(ctx, p)
|
||||
├─ ErrNotFound ──► clear ID/IP ──► RequeueNow (next pass creates)
|
||||
├─ Provisioning ──► Provisioned=False ──► ProvisioningPoll
|
||||
├─ Running ──► status.ip = inst.IP,
|
||||
│ Provisioned=True/Created ──► DriftPoll
|
||||
│ Provisioned=True/Created,
|
||||
│ applyHealth (see §6) ──► DriftPoll
|
||||
└─ Stopped/Termin. ──► prov.Delete (cattle) ──► DeletionPoll
|
||||
|
||||
any provider error ──► providerFailure(p, err) ── provider.Class(err):
|
||||
@@ -108,10 +109,10 @@ reconcileManaged(ctx, p)
|
||||
reconcileDelete(ctx, p) reconcileExternal(ctx, p)
|
||||
├─ no finalizer ──► return {} │ status.ip = spec.endpoint.host
|
||||
├─ providerID == "" ──► RemoveFinalizer │ setProvisioned(True/ExternalEndpoint)
|
||||
│ → r.Update → object actually deleted └─ return {} (no finalizer, no
|
||||
├─ prov.Get → ErrNotFound ──► RemoveFinalizer provider calls ever;
|
||||
│ → r.Update → object actually deleted the health engine —
|
||||
└─ exists ──► prov.Delete Step 5 — drives the rest)
|
||||
│ → r.Update → object actually deleted │ applyHealth (see §6)
|
||||
├─ prov.Get → ErrNotFound ──► RemoveFinalizer └─ return {} (no finalizer,
|
||||
│ → r.Update → object actually deleted no provider calls ever)
|
||||
└─ exists ──► prov.Delete
|
||||
→ Provisioned=False/Deleting
|
||||
──► RequeueAfter: DeletionPoll (poll until gone)
|
||||
```
|
||||
@@ -129,4 +130,56 @@ prov.ListByTag ─► client.List(Pods by labels, all namespaces) ─┘ them
|
||||
The reconciler never watches provider-side resources (Pods now, GCP VMs
|
||||
later). All instance-state observation is poll-based through the
|
||||
`Provider` interface, so the same flow works identically for a cloud API
|
||||
that has no watch mechanism at all.
|
||||
that has no watch mechanism at all.
|
||||
|
||||
### 6. Health engine (`internal/health/`) — probes and transitions
|
||||
|
||||
The engine is a leader-elected manager Runnable with its own goroutines,
|
||||
independent of the workqueue. It owns health *state*; the reconciler owns
|
||||
its *representation* in status — that split keeps exactly one writer of
|
||||
`.status` and makes write-only-on-transition fall out for free.
|
||||
|
||||
```text
|
||||
Engine.Start(ctx) (engine.go)
|
||||
├─ spawns Workers (8) probe goroutines ◄─┐
|
||||
└─ ticker loop (Tick = 1s): │ jobs channel (non-blocking send;
|
||||
tick(ctx, now, jobs) │ saturated pool → retry next tick)
|
||||
│ Reader.List(Proxies) ── from the manager cache
|
||||
│ per proxy: skip if no IP/host or deleting (state pruned →
|
||||
│ a replaced instance starts with fresh counters)
|
||||
│ newState: seed verdict from an existing Healthy condition
|
||||
│ (leader handover), jitter first probe across the interval
|
||||
│ due && !inFlight ──► jobs ◄── probe worker picks up
|
||||
└ prune states for proxies gone from the cache
|
||||
│
|
||||
probe(ctx, proxyURL, hc, tls) (probe.go)
|
||||
│ fresh transport per probe, DisableKeepAlives=true
|
||||
│ (load-bearing: keep-alives would cache the CONNECT
|
||||
│ tunnel and later probes would never re-exercise it)
|
||||
│ https probe URL ⇒ CONNECT through the proxy + TLS inside
|
||||
└ success = err == nil AND expected status code
|
||||
│
|
||||
record(job, result, now) ── under one mutex
|
||||
│ counters: consecOK/consecFail; verdict flips only at
|
||||
│ successThreshold / failureThreshold
|
||||
│ emit ONLY on: first-ever verdict │ threshold flip │
|
||||
│ latency Δ > max(20ms, 50% of reported) rate-limited
|
||||
│ to one report per MinReportInterval (60s)
|
||||
▼
|
||||
Events chan (buffered 64, non-blocking send;
|
||||
on drop the reported markers do NOT advance → next probe retries)
|
||||
│
|
||||
▼
|
||||
source.Channel → workqueue → Reconcile (see §1)
|
||||
│
|
||||
▼
|
||||
r.applyHealth(p) ── reads Engine.Snapshot(key) (status.go)
|
||||
stages the Healthy condition + latencyMillis +
|
||||
lastHealthCheckTime; computePhase turns Provisioned=True
|
||||
+ Healthy=True/False into phase Ready / Unhealthy
|
||||
```
|
||||
|
||||
Consequence worth knowing: `status.lastHealthCheckTime` is the time of the
|
||||
last *status-affecting* probe, not the most recent probe — suppressed
|
||||
probes deliberately never write status. True probe recency will live in
|
||||
metrics (Step 9).
|
||||
|
||||
@@ -9,7 +9,7 @@ Pairs with [docs/plans/2026-08-07-1747-proxy-operator.md](../plans/2026-08-07-17
|
||||
- [x] Step 2 — Provider contract (`internal/provider/`)
|
||||
- [x] Step 3 — Kubernetes pod provider (`internal/provider/kubernetes/`; first built as an in-memory mock, then replaced — see the two Step 3 sections below)
|
||||
- [x] Step 4 — Reconciler (`internal/controller/`)
|
||||
- [ ] Step 5 — Health engine (`internal/health/`)
|
||||
- [x] Step 5 — Health engine (`internal/health/`)
|
||||
- [ ] Step 6 — Lease store (`internal/lease/`)
|
||||
- [ ] Step 7 — Discovery API (`internal/discovery/`)
|
||||
- [ ] Step 8 — GCP provider (`internal/provider/gcp/`)
|
||||
@@ -626,3 +626,71 @@ deterministic and fast, at the cost of not exercising watch-driven
|
||||
requeues; Step 11's manager-driven cases cover that. The Secret watch is
|
||||
wired in `SetupWithManager` but the label-restricted Secret cache it
|
||||
assumes arrives with `cmd/main.go` in Step 10.
|
||||
|
||||
## Step 5 — Health engine (`internal/health/`)
|
||||
|
||||
Implemented the engine per the plan's design: `probe.go` (through-the-proxy
|
||||
probe with the plan's exact transport — fresh per probe,
|
||||
`DisableKeepAlives: true` so every probe re-exercises CONNECT) and
|
||||
`engine.go` (leader-elected manager Runnable: 1 s scheduler tick + a pool
|
||||
of 8 workers, per-proxy threshold state under one mutex, transition-only
|
||||
emission over a buffered `chan event.GenericEvent`). The reconciler side
|
||||
landed in the same step: a `HealthSnapshotter` interface + `applyHealth`
|
||||
staging the Healthy condition/latency/lastHealthCheckTime from
|
||||
`Engine.Snapshot`, and a conditional
|
||||
`WatchesRawSource(source.Channel(...))` in `SetupWithManager`. Everything
|
||||
tolerates nil (engine unwired) until `cmd/main.go` connects the two in
|
||||
Step 10.
|
||||
|
||||
Deviations and judgment calls beyond the plan text:
|
||||
|
||||
- **First-probe scheduling is split by whether a verdict was seeded.** The
|
||||
plan's startup jitter (`nextDue = now + rand(0, interval)`) applies only
|
||||
to proxies whose state was seeded from an existing Healthy condition —
|
||||
the restart case it exists for. A never-probed proxy is probed on the
|
||||
next tick instead; making a brand-new proxy wait up to a full interval
|
||||
for its first verdict would be pure lag with no thundering-herd benefit.
|
||||
- **The latency-change emission rule only applies while the verdict is
|
||||
healthy.** Caught by the first test run, not foreseen: a success streak
|
||||
still below `successThreshold` (verdict unhealthy, reported unhealthy)
|
||||
satisfied the plan's rule (c) — latency delta vs a stale reported value,
|
||||
rate window open — and emitted a pointless latency-only update for a
|
||||
proxy still reported as unhealthy. Guarded with `res.ok && *st.healthy`.
|
||||
- **`ProbeTLSConfig` field added to the engine** (nil = system roots). The
|
||||
probe function needs a CA override to be testable against
|
||||
`httptest.NewTLSServer`, and the same knob is genuinely useful for
|
||||
probing targets signed by a private CA. Not a test-only backdoor.
|
||||
- **State pruning doubles as replacement hygiene:** any proxy with no
|
||||
probeable host (provisioning, mid-replacement, deleting) has its state
|
||||
dropped each tick, so a replacement instance always starts with fresh
|
||||
counters. Complementarily, the reconciler's create branch removes the
|
||||
stale Healthy condition and latency fields — a new VM shouldn't wear its
|
||||
predecessor's verdict.
|
||||
|
||||
Tests: a real CONNECT-capable proxy stub (hijack + bidirectional
|
||||
`io.Copy`) probing a real `httptest.NewTLSServer` — CONNECT success,
|
||||
refused CONNECT, unexpected status, dead proxy, plain-http forwarding;
|
||||
table-driven threshold/suppression/seeding/pruning tests driving
|
||||
`record`/`tick` directly; an end-to-end `Start` test (fake reader, fake
|
||||
probeFn, 5 ms tick) asserting event delivery, snapshot content, and clean
|
||||
shutdown on context cancel; and controller-side tests with a
|
||||
`fakeSnapshotter` proving Running+healthy ⇒ `Ready`, Running+unhealthy ⇒
|
||||
`Unhealthy`, no-verdict ⇒ no condition, and stale-verdict cleanup on
|
||||
replacement.
|
||||
|
||||
Verification (all green):
|
||||
|
||||
```bash
|
||||
make test # envtest + units; health 93.4%, controller 77.4%
|
||||
KUBEBUILDER_ASSETS="$PWD/bin/k8s/1.36.2-darwin-arm64" go test -race ./...
|
||||
go test -race -count=2 ./internal/health/ # shook out the emission-rule bug above
|
||||
```
|
||||
|
||||
Worth noting: `docs/architecture.md` (created between Steps 4 and 5 on
|
||||
user request) gained a §6 for the engine and now shows the third workqueue
|
||||
feed (`source.Channel`). The `Healthy` condition reasons live in the
|
||||
controller package (`ReasonProbeSucceeded`/`ReasonProbeFailed`) — the
|
||||
engine deliberately knows nothing about conditions except reading one at
|
||||
seed time, keeping the state/representation split honest. The
|
||||
`hint`-driven `wg.Go` idiom (Go 1.25+) replaced the classic
|
||||
`wg.Add/defer wg.Done` in the worker pool.
|
||||
|
||||
Reference in New Issue
Block a user