Add orphan GC sweeper and Prometheus metrics with explicit registration

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-08-09 15:42:29 +02:00
parent 8176a5eef8
commit add120c033
12 changed files with 891 additions and 6 deletions

View File

@@ -13,7 +13,7 @@ Pairs with [docs/plans/2026-08-07-1747-proxy-operator.md](../plans/2026-08-07-17
- [x] Step 6 — Lease store (`internal/lease/`)
- [x] Step 7 — Discovery API (`internal/discovery/`)
- [x] Step 8 — GCP provider (`internal/provider/gcp/`)
- [ ] Step 9 — Orphan GC + metrics
- [x] Step 9 — Orphan GC + metrics
- [ ] Step 10 — Wiring, config, docs
- [ ] Step 11 — Tests
- [ ] Verification (vet/test/kind e2e) + commit, push, open MR
@@ -875,3 +875,67 @@ protobuf-construction idiom and matches every example in the SDK docs.
Registry wiring (`"gcp": gcp.New`) happens at the composition root in
Step 10, as designed in Step 2. `docs/architecture.md` §5 now shows both
providers' call mappings.
## Step 9 — Orphan GC + metrics
Implemented `internal/gc/gc.go` (the `Sweeper` Runnable) and
`internal/metrics/metrics.go` (explicit-registration metric set), plus the
`provider.WithMetrics` decorator deferred from Step 2 into
`internal/provider/metrics.go`, and the observation hooks in the health
engine and discovery server.
**GC**, per the plan: `NeedLeaderElection() = true` (destructive ⇒ single
writer), 10 min ticker with the first sweep one full interval after start,
per-provider `ListByTag` with log-and-continue on provider errors, and a
kill requires all three of: our UID label present, older than `MinAge`
(10 min), and the UID matching no existing CR — where a CR with a
`deletionTimestamp` still counts as live (its finalizer owns that
deletion; GC racing it would double-delete). Two safety behaviors worth
naming: a failed Proxy `List` skips the whole sweep (an unreadable live
set proves nothing orphaned), and the namespace-scope guard makes `Start`
refuse to run against a namespace-restricted cache unless
`--gc-allow-namespaced` is explicit, with the flag named in the error.
One deviation of record: the plan says kills log "at Warn", but logr has
no Warn level — kills log at Info with a `WARNING:` prefix carrying
provider, providerID, and UID, same convention as the discovery server's
empty-token warning.
**Metrics**, per the plan: no `init()``Metrics.Register(reg, phases,
leases)` is called explicitly by the composition root (Step 10), which is
also what lets every test use a fresh registry (asserted by a test that
registers two sets on two registries). `proxy_operator_proxies{phase}`
and `proxy_operator_leases_active` are scrape-time collectors fed by
closures — a reconcile-incremented gauge drifts and leaks series on
delete; reading the source of truth at scrape time cannot. The
probe vectors observe **every** probe (status stays transition-only;
metrics carry the high-frequency signal), and the health engine calls
`ForgetProxy` when it prunes a state entry so per-proxy series don't leak.
`provider_requests_total{provider,op,result}` comes from the
`WithMetrics` decorator — the one place `Class()` is called purely for
observability, with results labelled ok / not_found / quota_exceeded /
permanent / transient.
**Decoupling shape:** each consuming package defines its own small
recorder interface (`health.ProbeMetrics`, `discovery.LeaseMetrics`,
`provider.RequestRecorder`); `metrics.Metrics` satisfies all of them
structurally. Only `cmd/main.go` will import `internal/metrics`.
Tests (`-race -count=2` clean): GC's true-orphan matrix in one sweep
(live kept, deleting-CR kept, young kept, unlabelled kept, orphan
deleted), broken-provider isolation, list-failure skips sweep, the
namespace guard both ways, and the Start loop sweeping then stopping;
metrics via `prometheus/testutil``GatherAndCompare` on the scrape-time
collectors, ForgetProxy dropping series, outcome/result label counts;
the decorator's five-way classification table with error passthrough
asserted. Coverage: gc 86.1%, metrics 95.0%, provider up to 97.1%;
health/discovery re-ran green with the hooks in place.
```bash
go test -race -count=2 ./internal/gc/ ./internal/metrics/ ./internal/provider/ ./internal/health/ ./internal/discovery/
make test # whole repo green
```
Worth noting: `prometheus/client_golang` was already in the module via
controller-runtime's metrics server, so no new dependency — `go mod tidy`
just promoted it to direct. `docs/architecture.md` gained §8 (GC sweep)
and §9 (metrics shape).