Add orphan GC sweeper and Prometheus metrics with explicit registration
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1,10 +1,10 @@
|
||||
# Architecture
|
||||
|
||||
> **Status:** the operator is built through Step 8 (GCP provider) of
|
||||
> **Status:** the operator is built through Step 9 (orphan GC + metrics) of
|
||||
> [docs/plans/2026-08-07-1747-proxy-operator.md](plans/2026-08-07-1747-proxy-operator.md).
|
||||
> This document currently covers the event/reconcile flow and the
|
||||
> HTTP-driven lease/discovery path; the components table and the Decisions
|
||||
> section arrive with Step 10, and the orphan-GC flow lands with Step 9.
|
||||
> This document covers the event/reconcile flow, the HTTP-driven
|
||||
> lease/discovery path, and the GC sweep; the components table and the
|
||||
> Decisions section arrive with Step 10.
|
||||
|
||||
## Event flow: cluster events → reconciler functions
|
||||
|
||||
@@ -240,3 +240,50 @@ Store.Start(ctx) ── manager Runnable, NOT leader-elected: sweeps expired
|
||||
leases + cooldowns; correctness never depends on the
|
||||
sweep (every read checks ExpiresAt against the clock)
|
||||
```
|
||||
|
||||
### 8. Orphan GC (`internal/gc/`) — the crash-safety net
|
||||
|
||||
Timer-driven, leader-elected (destructive ⇒ single writer). Exists for the
|
||||
one gap the reconciler cannot close alone: a crash after a provider Create
|
||||
but before the status write that records the instance.
|
||||
|
||||
```text
|
||||
Sweeper.Start(ctx) ── refuses to run when the cache is namespace-
|
||||
│ restricted unless --gc-allow-namespaced is explicit
|
||||
│ (an incomplete live set would "orphan" live VMs)
|
||||
└─ every Interval (10m; first sweep a full interval after start):
|
||||
sweep(ctx)
|
||||
│ Reader.List(Proxies) → live UID set
|
||||
│ List fails → skip the whole sweep (never guess)
|
||||
│ a CR with deletionTimestamp still counts as LIVE — its
|
||||
│ finalizer owns that deletion; GC racing it double-deletes
|
||||
└ per provider: ListByTag
|
||||
│ error → log, continue with the next provider
|
||||
└ delete only when ALL hold:
|
||||
has the proxy-operator-uid label (ownership proof)
|
||||
older than MinAge (10m) (not mid-create)
|
||||
UID matches no existing CR (truly orphaned)
|
||||
each kill logged loudly with provider, providerID, UID
|
||||
```
|
||||
|
||||
### 9. Metrics (`internal/metrics/`)
|
||||
|
||||
Registered explicitly from `cmd/main.go` (no `init()`; tests use fresh
|
||||
registries). Two kinds:
|
||||
|
||||
- **Scrape-time collectors** — `proxy_operator_proxies{phase}` and
|
||||
`proxy_operator_leases_active` read the cache / lease store at every
|
||||
scrape; reconcile-incremented gauges inevitably drift and leak series.
|
||||
- **Fed vectors** — `healthcheck_duration_seconds{proxy}` and
|
||||
`healthcheck_failures_total{proxy}` observe EVERY probe (status writes
|
||||
are transition-only; metrics carry the high-frequency signal), and the
|
||||
health engine deletes a proxy's series when it prunes its state;
|
||||
`lease_requests_total{outcome}` from the discovery handlers;
|
||||
`provider_requests_total{provider,op,result}` from the
|
||||
`provider.WithMetrics` decorator — the one place `Class()` is called
|
||||
purely for observability.
|
||||
|
||||
Each consuming package defines its own small recorder interface
|
||||
(`health.ProbeMetrics`, `discovery.LeaseMetrics`, `provider.RequestRecorder`);
|
||||
`metrics.Metrics` satisfies all of them structurally, so no package other
|
||||
than `cmd/main.go` imports the metrics package.
|
||||
|
||||
Reference in New Issue
Block a user