Add orphan GC sweeper and Prometheus metrics with explicit registration

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-08-09 15:42:29 +02:00
parent 8176a5eef8
commit add120c033
12 changed files with 891 additions and 6 deletions

View File

@@ -1,10 +1,10 @@
# Architecture
> **Status:** the operator is built through Step 8 (GCP provider) of
> **Status:** the operator is built through Step 9 (orphan GC + metrics) of
> [docs/plans/2026-08-07-1747-proxy-operator.md](plans/2026-08-07-1747-proxy-operator.md).
> This document currently covers the event/reconcile flow and the
> HTTP-driven lease/discovery path; the components table and the Decisions
> section arrive with Step 10, and the orphan-GC flow lands with Step 9.
> This document covers the event/reconcile flow, the HTTP-driven
> lease/discovery path, and the GC sweep; the components table and the
> Decisions section arrive with Step 10.
## Event flow: cluster events → reconciler functions
@@ -240,3 +240,50 @@ Store.Start(ctx) ── manager Runnable, NOT leader-elected: sweeps expired
leases + cooldowns; correctness never depends on the
sweep (every read checks ExpiresAt against the clock)
```
### 8. Orphan GC (`internal/gc/`) — the crash-safety net
Timer-driven, leader-elected (destructive ⇒ single writer). Exists for the
one gap the reconciler cannot close alone: a crash after a provider Create
but before the status write that records the instance.
```text
Sweeper.Start(ctx) ── refuses to run when the cache is namespace-
│ restricted unless --gc-allow-namespaced is explicit
│ (an incomplete live set would "orphan" live VMs)
└─ every Interval (10m; first sweep a full interval after start):
sweep(ctx)
│ Reader.List(Proxies) → live UID set
│ List fails → skip the whole sweep (never guess)
│ a CR with deletionTimestamp still counts as LIVE — its
│ finalizer owns that deletion; GC racing it double-deletes
└ per provider: ListByTag
│ error → log, continue with the next provider
└ delete only when ALL hold:
has the proxy-operator-uid label (ownership proof)
older than MinAge (10m) (not mid-create)
UID matches no existing CR (truly orphaned)
each kill logged loudly with provider, providerID, UID
```
### 9. Metrics (`internal/metrics/`)
Registered explicitly from `cmd/main.go` (no `init()`; tests use fresh
registries). Two kinds:
- **Scrape-time collectors** — `proxy_operator_proxies{phase}` and
`proxy_operator_leases_active` read the cache / lease store at every
scrape; reconcile-incremented gauges inevitably drift and leak series.
- **Fed vectors** — `healthcheck_duration_seconds{proxy}` and
`healthcheck_failures_total{proxy}` observe EVERY probe (status writes
are transition-only; metrics carry the high-frequency signal), and the
health engine deletes a proxy's series when it prunes its state;
`lease_requests_total{outcome}` from the discovery handlers;
`provider_requests_total{provider,op,result}` from the
`provider.WithMetrics` decorator — the one place `Class()` is called
purely for observability.
Each consuming package defines its own small recorder interface
(`health.ProbeMetrics`, `discovery.LeaseMetrics`, `provider.RequestRecorder`);
`metrics.Metrics` satisfies all of them structurally, so no package other
than `cmd/main.go` imports the metrics package.