egress-proxies-operator/docs/architecture.md

Architecture

Components

Component	Package	Runs as	Leader-elected	Role
Proxy + ProviderHealth CRDs	api/v1alpha1	types	—	Proxy spec/status, CEL validation, defaulting, pure helpers; the status-only, cluster-scoped ProviderHealth
Reconciler	internal/controller	controller	yes (with the manager)	the state machine: provision, replace, delete, represent health
Provider contract	internal/provider	library	—	Provider interface, optional Checker, error taxonomy, deterministic naming, config, metrics decorator; rules in provider-contract.md, per-provider profile table in providers-existing.md
kubernetes provider	internal/provider/kubernetes	library	—	real Squid pods in this cluster (local dev/CI)
gcp provider	internal/provider/gcp	library	—	Compute Engine VMs, four API calls, fire-and-forget ops
kvm provider	internal/provider/kvm	library	—	VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds
kvm2 provider	internal/provider/kvm2	library	—	the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial
Health engine	internal/health	Runnable	yes	through-the-proxy probes, thresholds, transition events
Lease store	internal/lease	Runnable (expiry sweep)	no	in-memory leases + cooldowns, single mutex
Discovery API	internal/discovery	Runnable	no	HTTP list/lease/release/report on :8090
Orphan GC	internal/gc	Runnable	yes	deletes tagged instances whose CR is gone
Provider health monitor	internal/providerhealth	Runnable	yes	periodic provider checks → one ProviderHealth per provider, target gauges, transition logs
Metrics	internal/metrics	library	—	explicit registration, scrape-time collectors
Composition root	cmd/main.go	binary	—	flags, provider registry, wires everything onto one manager

Event flow: cluster events → reconciler functions

Which functions run in response to which Kubernetes cluster events, as wired in internal/controller/proxy_controller.go.

1. How cluster events reach the reconciler

KUBERNETES CLUSTER EVENTS                                (wiring: SetupWithManager,
─────────────────────────                                 proxy_controller.go)

 Proxy CR created / spec edited /     Secret created /           health transition
 status patched / delete requested    updated / deleted          (engine, see §6)
              │                              │                          │
   watch: For(&crawlv1alpha1.Proxy{})   watch: Watches(          WatchesRawSource(
              │                         &corev1.Secret{}, ...)   source.Channel(
              │                              │                    r.HealthEvents, ...))
              │                r.proxiesForSecret(ctx, secret)          │
              │                  │ r.List(Proxies in namespace)         │
              │                  │ keeps those whose                    │
              │                  │ spec.cloudInit.secretRef matches     │
              │                  ▼                                      │
              │                [reconcile.Request per matching Proxy]   │
              ▼                              │                          │
        ┌────────────────────────────────────┴──────────────────────────┴──┐
        │                controller-runtime workqueue                      │◄── RequeueAfter
        │  (dedup by namespace/name, rate-limited,                         │    timers
        │   MaxConcurrentReconciles: 3)                                    │◄── error backoff
        └──────────────────────────┬────────────────────────────────────────┘
                                   ▼
                    ProxyReconciler.Reconcile(ctx, req)

The Secret watch makes rotating a cloud-init Secret a first-class event: it re-enqueues every Proxy referencing that Secret, which is how secret rotation triggers VM replacement even though the Proxy spec is untouched.

2. Inside Reconcile — dispatch and the single status write

Reconcile(ctx, req)
  │ r.Get(ctx, req.NamespacedName, &p)          ── fetch the Proxy (NotFound → done)
  │ base := p.DeepCopy()                        ── snapshot for the diff
  │ defer patchStatusIfChanged(ctx, base, &p) ──────────────────────────┐
  │                                                                     │
  ├─ p.DeletionTimestamp set ──► reconcileDelete(ctx, &p)               │
  ├─ p.Spec.Mode == External ──► reconcileExternal(ctx, &p)             │
  └─ otherwise (Managed) ──────► reconcileManaged(ctx, &p)              │
                                                                        ▼
                                     patchStatusIfChanged  (status.go)
                                       │ p.Status.ObservedGeneration = p.Generation
                                       │ p.Status.Phase = computePhase(&p)
                                       │ equality.Semantic.DeepEqual(base, p)?
                                       └─ changed → r.Status().Patch(...)   ◄── the ONLY
                                          unchanged → no API call               status write

3. reconcileManaged — the state machine

reconcileManaged(ctx, p)
  │
  ├─ controllerutil.AddFinalizer? ──► r.Update ──► return {}        (watch event re-triggers)
  ├─ permanent-failure latch (FindStatusCondition == PermanentError
  │    at this generation) ──► return {}                            (silent until spec edit)
  ├─ r.Providers[p.Spec.Provider] missing ──► setProvisioned(PermanentError) → Failed
  │
  ├─ resolveCloudInit(ctx, p) ──► r.Get(Secret) if secretRef        (error → CloudInitError + backoff)
  ├─ hash := specHash(p, cloudInit)                                  (spechash.go)
  │
  ├─ status.providerID == "" ─────────► prov.Create(CreateRequest{Name: NameFromUID(p.UID), ...})
  │                                       │ setSpecHash → r.Update (annotation)
  │                                       └ stage providerID + Provisioned=False/Provisioning
  │                                         ──► RequeueAfter: ProvisioningPoll
  │
  ├─ annotation != hash, annotation == "" ──► adopt: setSpecHash → r.Update
  │                                           ──► RequeueAfter: RequeueNow
  ├─ annotation != hash, annotation != "" ──► replaceInstance:
  │                                             prov.Get ─ NotFound → setSpecHash, clear ID/IP
  │                                             │            ──► RequeueNow (next pass creates)
  │                                             └ exists  → prov.Delete, Provisioned=False/Replacing
  │                                                          ──► RequeueAfter: DeletionPoll
  │
  └─ annotation == hash ──► prov.Get(providerID)
        ├─ ErrNotFound      ──► clear ID/IP            ──► RequeueNow (next pass creates)
        ├─ Provisioning     ──► Provisioned=False      ──► ProvisioningPoll
        ├─ Running          ──► status.ip = inst.IP,
        │                       Provisioned=True/Created,
        │                       applyHealth (see §6)   ──► DriftPoll
        └─ Stopped/Termin.  ──► prov.Delete (cattle)   ──► DeletionPoll

  any provider error ──► providerFailure(p, err) ── provider.Class(err):
        ├─ ErrQuotaExceeded ──► condition QuotaExceeded ──► RequeueAfter: QuotaRetry (nil error)
        ├─ ErrPermanent     ──► condition PermanentError ──► phase Failed, no retry
        └─ ErrTransient     ──► return err ──► workqueue exponential backoff

4. reconcileDelete and reconcileExternal

reconcileDelete(ctx, p)                          reconcileExternal(ctx, p)
  ├─ no finalizer ──► return {}                    │ status.ip = spec.endpoint.host
  ├─ providerID == "" ──► RemoveFinalizer          │ setProvisioned(True/ExternalEndpoint)
  │     → r.Update → object actually deleted       │ applyHealth (see §6)
  ├─ prov.Get → ErrNotFound ──► RemoveFinalizer    └─ return {}   (no finalizer,
  │     → r.Update → object actually deleted                       no provider calls ever)
  └─ exists ──► prov.Delete
        → Provisioned=False/Deleting
        ──► RequeueAfter: DeletionPoll (poll until gone)

5. What provider calls do in the outside world

kubernetes pod provider (internal/provider/kubernetes/)
prov.Create ──► buildPod (pure) ──► client.Create(corev1.Pod)   ─┐ these cause Pod events,
prov.Get    ──► client.Get(Pod) → phase/IP → InstanceState       │ but the operator does NOT
prov.Delete ──► client.Delete(Pod, tolerate NotFound)            │ watch Pods — it observes
prov.ListByTag ─► client.List(Pods by labels, all namespaces)   ─┘ them by polling prov.Get
                                                                    on each RequeueAfter tick

gcp provider (internal/provider/gcp/) — instances.{Insert,Get,Delete,AggregatedList}, nothing else
prov.Create ──► buildInsertRequest (pure) ──► instances.Insert   ─┐ fire-and-forget:
                  409 alreadyExists = success (idempotent retry)  │ Operation.Wait is never
prov.Get    ──► instances.Get → status/NatIP → InstanceState      │ called; readiness is
                  RUNNING without NatIP = still Provisioning      │ discovered by Get polls,
prov.Delete ──► instances.Delete (404 = success)                  │ exactly like the pod
prov.ListByTag ─► AggregatedList(label filter,                   ─┘ provider
                    ReturnPartialSuccess: true)
providerID = zones/<zone>/instances/<name> — zone-qualified, so Get/Delete
stay correct even mid-replacement after a zone edit

kvm provider (internal/provider/kvm/) — libvirt RPC over an SSH-forwarded unix socket, per host
prov.Create ──► DomainLookupByName: exists → start if needed, done      ─┐ every step tolerates
                else: ensure base image in pool (download on first use) │ the previous attempt
                      overlay volume (exists = reuse)                    │ having got that far;
                      seed ISO (always rebuilt — a cut-off upload must   │ the seed is the one
                        never be trusted)                                │ thing redone
                      DomainDefineXML + DomainCreate                    ─┘
prov.Get    ──► DomainGetState + DomainGetMetadata + DomainInterfaceAddresses
                  (ARP, then guest agent; first routable IPv4)
                  running without IP = Provisioning; nostate = Provisioning
prov.Delete ──► destroy (not running = fine) → undefine → delete overlay + seed by name
prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose
                    <metadata> carries managed=true; one host down fails the sweep,
                    one unreadable domain is skipped
providerID = hosts/<host>/domains/<name>; GC identity lives in the domain's
<metadata> element (libvirt has no labels). All RPCs run under a context-
bounded call: go-libvirt never times out on its own, so a cancelled reconcile
closes the SSH connection to unblock it.

kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract
prov.Create ──► placement checked against the catalogue (Permanent before any dial)
                DomainLookupByName: exists → mark+uid guard (no mark or foreign   ─┐ every step
                  uid = Permanent; paused = Permanent),                            │
                  start if needed, wait for running, done                          │ idempotent,
                else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a
                        + 16 MiB must fit, else ErrQuotaExceeded (reconciler       │ per-name lock
                        waits 5m)                                                  │ shared with
                      ensure base image: marker present = cached; otherwise GET    │ Delete
                        streamed into StorageVolUpload on a DEDICATED connection,  │
                        StoragePoolRefresh, marker written last                    │
                      overlay volume (exists = reuse)                              │
                      seed ISO (always rebuilt)                                    │
                      DomainDefineXML (<metadata> in kvm2's own namespace:         │
                        managed, uid, createdAt) + DomainCreate, poll until        │
                        running (15s cap; a domain that dies at once = Transient) ─┘
prov.Get    ──► DomainLookupByName + DomainGetMetadata + DomainGetState
                  + DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4)
                  running without IP = Provisioning; nostate = Provisioning;
                  only ErrNoDomain is NotFound
prov.Delete ──► destroy (not running = fine) → undefine (nvram, managed save, snapshots)
                  → delete overlay + seed by name; unconfigured host = logged no-op
prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose kvm2
                    <metadata> says managed=true; one host down fails the sweep,
                    one unreadable domain is skipped
Check       ──► per host, on a FRESH connection: version, hostname, pool lookup,
                  pool info, close; never the pooled one
providerID = hosts/<host>/domains/<name>. Every RPC has its own budget (dial 10s,
RPC 30s, DomainCreate 90s, seed upload 60s, image transfer 20m): a call that
outlives its ctx or budget kills the pooled connection (the only way to unblock
go-libvirt), so other calls in flight on that host fail once with Transient and
retry; the next call redials. Each RPC is a child span kvm2.<Method> under
provider.*, plus kvm2.dial and kvm2.download.

The reconciler never watches provider-side resources (Pods or GCP VMs). All instance-state observation is poll-based through the Provider interface, so the same flow works identically for a cloud API that has no watch mechanism at all.

6. Health engine (internal/health/) — probes and transitions

The engine is a leader-elected manager Runnable with its own goroutines, independent of the workqueue. It owns health state; the reconciler owns its representation in status — that split keeps exactly one writer of .status and makes write-only-on-transition fall out for free.

Engine.Start(ctx)                                (engine.go)
  ├─ spawns Workers (8) probe goroutines ◄─┐
  └─ ticker loop (Tick = 1s):              │ jobs channel (non-blocking send;
       tick(ctx, now, jobs)                │ saturated pool → retry next tick)
         │ Reader.List(Proxies)  ── from the manager cache
         │ per proxy: skip if no IP/host or deleting (state pruned →
         │   a replaced instance starts with fresh counters)
         │ newState: seed verdict from an existing Healthy condition
         │   (leader handover), jitter first probe across the interval
         │ due && !inFlight ──► jobs ◄── probe worker picks up
         └ prune states for proxies gone from the cache
                                           │
              probe(ctx, proxyURL, hc, tls)   (probe.go)
                │ fresh transport per probe, DisableKeepAlives=true
                │   (load-bearing: keep-alives would cache the CONNECT
                │    tunnel and later probes would never re-exercise it)
                │ https probe URL ⇒ CONNECT through the proxy + TLS inside
                └ success = err == nil AND expected status code
                                           │
              record(job, result, now)     ── under one mutex
                │ counters: consecOK/consecFail; verdict flips only at
                │   successThreshold / failureThreshold
                │ emit ONLY on: first-ever verdict │ threshold flip │
                │   latency Δ > max(20ms, 50% of reported) rate-limited
                │   to one report per MinReportInterval (60s)
                ▼
        Events chan (buffered 64, non-blocking send;
        on drop the reported markers do NOT advance → next probe retries)
                │
                ▼
        source.Channel → workqueue → Reconcile        (see §1)
                │
                ▼
        r.applyHealth(p)  ── reads Engine.Snapshot(key)   (status.go)
              stages the Healthy condition + latencyMillis +
              lastHealthCheckTime; computePhase turns Provisioned=True
              + Healthy=True/False into phase Ready / Unhealthy

Consequence worth knowing: status.lastHealthCheckTime is the time of the last status-affecting probe, not the most recent probe — suppressed probes deliberately never write status. True probe recency will live in metrics (Step 9).

7. Discovery + lease API (internal/discovery/, internal/lease/)

HTTP-driven, not cluster-event-driven: crawler clients call in; the only Kubernetes interaction is reading Proxies from the manager's cache. The server is a non-leader-elected Runnable (all replicas would serve, but the deployment ships replicas: 1 because lease state is per-process — an operator restart drops all leases and cooldowns, a documented caveat). Client-facing reference with request/response schemas and curl examples: api.md.

crawler client
  │  Authorization: Bearer $DISCOVERY_TOKEN   (empty token = auth disabled, loud startup warning)
  ▼
Server.handler()      middleware, outermost first        (server.go)
  recover → request-log → MaxBytesReader(64KiB) → bearer auth (constant-time; /healthz exempt)
  │
  ├─ GET  /healthz            ──► 200 ok (unauthenticated)
  │
  ├─ GET  /v1/proxies?attr.k=v&healthy=true            (handlers.go)
  │     Reader.List(Proxies) ── manager cache
  │     filter: attributes equality + Healthy condition
  │     + Store.Counts() for activeLeases
  │     ──► 200 {"proxies":[...], "count":N}   (empty list is 200, not 404)
  │
  ├─ POST /v1/leases  {"selector":{...},"ttlSeconds":300,"target":"..."}
  │     Reader.List → filter selector; unhealthy matches counted, not offered
  │     Store.Acquire(healthy candidates, target, ttl)   ── one lock: select+insert
  │     │    selection: fewest active leases, then latency, then name
  │     ├─ granted ──► 201 {leaseID, proxy:{...}, expiresAt, ttlSeconds}
  │     └─ ErrNoMatch ──► 409 {"error":"no_match", considered, atCapacity,
  │                            inCooldown, unhealthy}
  │
  ├─ DELETE /v1/leases/{id} ──► Store.Release ──► always 204 (idempotent)
  │
  └─ POST /v1/leases/{id}/report {"result":"ok|rate_limited|banned","target":"..."}
        Store.Report ── rate_limited/banned ⇒ cooldown[{proxy,target}] for
        │               CooldownWindow (target falls back: report → lease → global)
        ├─ 204 │ 400 invalid_result │ 404 unknown_lease
        └─ an expired lease still resolves for CooldownWindow past its TTL —
           a late report lands exactly when the proxy is being rate-limited

Store.Start(ctx)  ── manager Runnable, NOT leader-elected: sweeps expired
                     leases + cooldowns; correctness never depends on the
                     sweep (every read checks ExpiresAt against the clock)

8. Orphan GC (internal/gc/) — the crash-safety net

Timer-driven, leader-elected (destructive ⇒ single writer). Exists for the one gap the reconciler cannot close alone: a crash after a provider Create but before the status write that records the instance.

Sweeper.Start(ctx)   ── refuses to run when the cache is namespace-
  │                     restricted unless --gc-allow-namespaced is explicit
  │                     (an incomplete live set would "orphan" live VMs)
  └─ every Interval (10m; first sweep a full interval after start):
       sweep(ctx)
         │ Reader.List(Proxies) → live UID set
         │   List fails → skip the whole sweep (never guess)
         │   a CR with deletionTimestamp still counts as LIVE — its
         │   finalizer owns that deletion; GC racing it double-deletes
         └ per provider: ListByTag
             │ error → log, continue with the next provider
             └ delete only when ALL hold:
                 has the proxy-operator-uid label   (ownership proof)
                 older than MinAge (10m)            (not mid-create)
                 UID matches no existing CR         (truly orphaned)
               each kill logged loudly with provider, providerID, UID

9. Metrics (internal/metrics/)

Registered explicitly from cmd/main.go (no init(); tests use fresh registries). Two kinds:

• Scrape-time collectors — proxy_operator_proxies{phase} and proxy_operator_leases_active read the cache / lease store at every scrape; reconcile-incremented gauges inevitably drift and leak series. The provider-check gauges (§11) read the health monitor's snapshot the same way.
• Fed vectors — healthcheck_duration_seconds{proxy} and healthcheck_failures_total{proxy} observe EVERY probe (status writes are transition-only; metrics carry the high-frequency signal), and the health engine deletes a proxy's series when it prunes its state; lease_requests_total{outcome} from the discovery handlers; provider_requests_total{provider,op,result} from the provider.WithMetrics decorator — the one place Class() is called purely for observability.

Each consuming package defines its own small recorder interface (health.ProbeMetrics, discovery.LeaseMetrics, provider.RequestRecorder); metrics.Metrics satisfies all of them structurally, so no package other than cmd/main.go imports the metrics package.

10. Tracing (internal/tracing/)

OpenTelemetry tracing, integrated with — not replacing — the logr/zap logging. Everything hangs off standard OTEL_* env vars: with no OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_TRACES_EXPORTER set, no SDK is installed, spans are no-ops, and logs are byte-for-byte what they were. OTEL_SDK_DISABLED=true and OTEL_TRACES_EXPORTER=none force it off; console prints spans to stdout for local dev; sampling follows OTEL_TRACES_SAMPLER(_ARG).

Span topology — each unit of work is a new trace (watch events carry no incoming trace context):

• Reconcile Proxy (root, via a reconcile.Reconciler decorator) → reconcile.managed / reconcile.replaceInstance / reconcile.delete, plus status.patch from the deferred flush. reconcileExternal does no I/O and is unspanned.
• provider.create|get|delete|list (client spans, provider.WithTracing decorator wired outermost around WithMetrics); the GCP SDK's own otelhttp transport contributes HTTP child spans automatically.
• kvm2 child spans: under provider.*, one kvm2.<Method> client span per libvirt RPC (for example kvm2.DomainDefineXML, attributes kvm2.host, kvm2.domain, kvm2.pool, kvm2.volume, kvm2.libvirt_code on failure), kvm2.dial around the SSH handshake and libvirt connect, and kvm2.download around an image fetch; under providerhealth.check, kvm2.dial and the four kvm2.<Method> spans of a check. They come from a span decorator over the seam, applied once in the dialer, so no RPC is unspanned. The libvirt call log line of an RPC is written inside its span. The kvm provider has no child spans.
• Kubernetes API calls become client child spans via a wrapped rest.Config transport — gated on an existing parent span, so informer list/watch long-polls and leader-election renewals never create root spans. The kube-apiserver ignores incoming traceparent by design; these spans are leaves.
• Discovery API: one server span per request (named from the route pattern), incoming W3C traceparent honored so clients' traces continue into the operator; /healthz excluded.
• gc.sweep per GC pass; providerhealth.check per round of provider checks; health.probe per probe only with --trace-health-probes (default off — probes run ~1/s per proxy).

Log correlation: logr sinks never see a context, so trace IDs ride on the logger — tracing.Start re-derives the ctx logger from a captured base with traceID/spanID values (lowerCamel, matching reconcileID; Grafana/Loki derived-field regexes must match traceID, not trace_id). Re-deriving from the base rather than layering keeps zap from emitting duplicate keys on nested spans. GCP V(5) wire logs get the same keys from the slog handler's ctx.

11. Provider health checks (internal/providerhealth/)

Answers "can this cluster reach each provider's backend right now?" without waiting for a Proxy to fail. A provider opts in by implementing provider.Checker: Check(ctx) []TargetStatus returns one verdict per target and never fails as a whole. All four providers do:

Provider	Target	What one check does
kvm	each kvm.hosts entry	over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space (details)
kvm2	each kvm2.hosts entry	on a fresh SSH+libvirt connection dialled for the check and closed after it: libvirt version and hostname, then the configured pool must exist and be running; details carry user, pool, hostname, libvirtVersion, poolState, poolCapacityGiB and poolAvailableGiB (details)
kubernetes	apiserver	lists at most one managed Pod cluster-wide: the verb and scope ListByTag and GC rely on
gcp	the project	one page of an aggregated instance list with maxResults=1: credentials, Compute API enabled, compute.instances.list

Monitor.Start(ctx)   ── leader-only; --provider-check-interval (1m; 0 = off,
  │                     otherwise ≥ 10s), --provider-check-timeout (15s, or
  │                     the interval if that is shorter)
  ├─ once: delete managed-by-labelled ProviderHealth objects whose provider
  │   is not configured (all of them when checks are off) — the providers
  │   config is read once per process
  └─ at start, then every Interval: checkOnce
       │ every provider's Check concurrently, under one context.WithTimeout
       │ snapshot → the scrape-time gauges
       │ log transitions: down or reason changed → "WARNING: provider target
       │   not ready"; back up → "Provider target ready"; steady → silent
       └ per provider: APIReader.Get → Create if missing → status merge patch

• One status write per provider per round, since lastCheckTime moves every round. At the 10s minimum interval that is still far below leader-election renewals. Proxy status avoids writes like this only because its probes run once a second per proxy.
• Reads go through the API reader, not the cache. The objects are cluster-scoped; a cached Get would start a cluster-wide informer (and need watch) even with --proxy-namespace.
• Checks bypass the provider decorators. cmd/main.go collects the Checkers before wrapping the providers in WithTracing(WithMetrics(...)), so a check never counts as a provider request. KVM RPCs still appear in the V(1) libvirt call trace, and each round is one providerhealth.check span.
• Not wired to /readyz. A dead hypervisor must not make the manager unready while the other providers keep working.
• Gauges proxy_operator_provider_target_up, proxy_operator_provider_check_duration_seconds and proxy_operator_provider_check_timestamp_seconds, each labelled {provider,target}. Only the leader exports them.

Illustrative output:

$ kubectl get providerhealth          # short name: ph
NAME         TYPE         READY   TARGETS   LAST CHECK   AGE
kubernetes   kubernetes   True    1/1       12s          3d
kvm-home     kvm          False   0/1       12s          3d
$ kubectl get ph kvm-home -o yaml     # per target: reason, message, details

Decisions

Judgment calls the spec left open, and deliberate deviations — recorded so they read as choices, not accidents. Chronological by build step.

• Registry takes its constructor map as a parameter instead of holding a package-level map: avoids the provider⇄registry import cycle and puts the wiring at the composition root, where it is visible.
• Mock provider replaced by the kubernetes-pod provider (user decision, mid-build): a simulated in-memory provider was too far from the real system to build confidence in. Local dev/CI now runs real ubuntu/squid pods (Canonical's actively maintained image, verified 50M+ pulls, pinned tag) in the operator's own cluster. Trade-off accepted: envtest has no kubelet, so end-to-end proof lives in the kind quickstart, and cluster pods share one egress IP — distinct egress paths remain the GCP provider's job.
• RequeueAfter: RequeueNow instead of the plan's Requeue: true: ctrl.Result.Requeue is deprecated in controller-runtime v0.24; a fifth configurable interval (default 1s) keeps identical semantics and stays shrinkable in tests.
• Quota exhaustion is a wait, not a failure: ErrQuotaExceeded sets a condition and requeues slowly (5m) with a nil error — off the backoff curve, out of the error log, and never phase: Failed. Only ErrPermanent latches Failed, keyed to the generation so a spec edit auto-recovers.
• The finalizer path never latches permanent failures: a permanent error during deletion keeps retrying visibly instead — latching there would wedge the object forever with no path out but manual finalizer surgery.
• Health transitions travel reconciler-ward over a channel (source.Channel), not direct status patches: phase derives from both provisioning and health, so two status writers would race and flap. One writer of status; the engine owns health state, the reconciler its representation; write-only-on-transition falls out for free.
• Health state seeds from the existing Healthy condition on leader handover (verdict kept, counters zeroed, first probe jittered), so a healthy fleet doesn't flap to Unknown on restart — but a real transition still needs a full threshold run. A never-probed proxy skips the jitter and probes on the next tick: startup spread matters for restarts, not for a single new proxy.
• Latency suppression is max(20ms, 50%) + a 60s rate limit, and only while the verdict is healthy. The spec's bare ">50% change" is undefined at 0 and lets a proxy jittering 40↔61ms write status forever; the healthy-only guard (found by test) stops a below-threshold success streak from emitting latency updates for a proxy still reported unhealthy. Consequence: status.lastHealthCheckTime means "last status-affecting probe" — true probe recency is in the metrics.
• Deterministic instance names are proxy- + 16 chars of base32(SHA-256(CR UID)): legal for both GCP ([a-z2-7] ⊂ [-a-z0-9], 22 ≤ 63 chars) and Pod names, 80 bits against birthday collisions at a fleet of tens. The replacement VM therefore has the same name as the one being deleted — which is why replacement polls to NotFound before recreating instead of racing a 409.
• banned and rate_limited share one cooldown window: a second duration knob the spec doesn't ask for; the report's semantic difference is preserved in the API but not the store.
• Report targets fall back report → lease → global, so a client that leased with a target can't accidentally poison the proxy's global pool by omitting the target in its report.
• The 409 body's considered counts unhealthy matches too (the store only ever sees healthy candidates): considered = atCapacity + inCooldown + unhealthy + eligible-but-outranked, keeping the numbers additive for a human debugging "why no proxy?".
• TTLs above --max-lease-ttl are a 400, not a silent clamp — a client asking for a week should find out.
• Discovery is not leader-elected and ships replicas: 1: caches start before non-leader-election runnables (verified in controller-runtime's ordering), and a leader-elected server would leave non-leader replicas as broken Service endpoints. One replica because lease state is per-process.
• GCP Create requires zone, machineType, and image and fails ErrPermanent naming the missing field — inventing machine-type defaults would silently create billable VMs of arbitrary shape.
• Unknown GCP instance statuses map to Stopped: the reconciler's answer to Stopped is delete-and-recreate, the always-safe move for cattle when the API grows a new state.
• KVM hosts are reached over SSH with the libvirt socket forwarded, not libvirtd's TCP/TLS listener: nothing to configure or expose on the hypervisors beyond sshd, pure Go on the operator side (no cgo, no libvirt libraries in the image), and the hosts list can grow by a config edit because the providerID carries the host name.
• KVM base images are fetched by the operator into the host's pool through libvirt's upload stream, rather than expected to be pre-placed: the operator then needs no filesystem access on the host and a new host is usable with zero manual staging. The cost is a synchronous first-time download inside Create.
• The KVM seed ISO is rebuilt on every Create attempt before the domain exists, instead of "exists → skip": a reconcile cancelled mid upload leaves a truncated seed, and the compensating delete cannot run because the connection was already closed to unblock the RPC. Skipping it would boot a VM with no cloud-init and no retry would ever repair it.
• KVM nostate maps to Provisioning, not Stopped: unlike GCP's unknown statuses, nostate means libvirt could not determine the state at all; answering that with delete-and-recreate is too drastic, so the next poll decides.
• KVM guests boot UEFI by default, secure boot off. Found on the first real host: Debian 13's generic image hangs in GRUB under SeaBIOS on a headless VM (Debian bug #1111240: terminal_output gfxterm serial with a serial console and no video device) regardless of CPU model or seed, and boots under OVMF with the serial console working. The alternatives were a virtio console (loses the kernel's serial output) or a video device the VMs have no use for. Every current distro cloud image supports UEFI, so it is the default and firmware: bios is the per-image escape; libvirt's firmware autoselection (<os firmware="efi">) keeps the OVMF path out of the operator's config. Secure boot is off so unsigned images boot too.
• A KVM domain GC cannot read is skipped, a host it cannot reach fails the sweep. GC never deletes what it does not see, so leaving one domain out is safe; one corrupt domain blocking GC for every other one is not. A whole host missing from the view would hide many, so that still aborts.
• Kubernetes 403s classify as ErrPermanent even though quota exhaustion also surfaces as 403 (indistinguishable from RBAC denial in apierrors): not hammering an API server that may never allow the request is the safer default; a real ResourceQuota 403 forgoes the gentler quota backoff. Documented at the classification site.
• GC kills log at Info with a WARNING: prefix — logr has no Warn level; the plan's "log at Warn" is met in spirit with provider, providerID, and UID always attached. Same convention as the discovery server's empty-token warning.
• GC trusts only provable orphans: instances without the UID label are never deleted, a CR with a deletionTimestamp still counts as live (its finalizer owns that deletion), and an unreadable Proxy list skips the whole sweep. The namespace guard refuses to sweep a namespace-restricted cache without --gc-allow-namespaced.
• Cloud-init Secrets must carry crawl.example.com/cloud-init: "true": the manager caches only labelled Secrets (the operator holds cluster-wide Secret read RBAC — an unrestricted cache would hold every Secret in scope). Unlabelled referenced Secrets are invisible by construction, surfacing as CloudInitError.
• Deletion never gives up on its own, but can be released by hand: a permanent cleanup failure keeps retrying with the finalizer on, because dropping it automatically would leak a running instance on any provider that answers "permission denied". Where Get fails permanently the reconciler still calls Delete, since it is idempotent and may finish where Get cannot converge. crawl.example.com/skip-cleanup: "true" abandons the instance and releases the object; it is checked before the provider lookup, so it also frees a Proxy whose provider is gone from the config.
• Events RBAC from the plan is omitted: nothing wires an EventRecorder in the prototype, and granting verbs nothing uses would be RBAC lint noise. Add the marker together with the recorder if events land later.
• logr, not slog, inside controller paths: the repo convention says slog, but log.FromContext(ctx) hands controller-runtime's logr logger to everything running under the manager — fighting that would mean two logging systems in one process. Noted as a deviation rather than silently ignored.
• Tracing is env-gated, not flag-gated: it activates only when OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_TRACES_EXPORTER is set (user decision). No collector configured means no SDK installed, no-op spans, unchanged logs — the safe default for every existing deployment. The one flag is --trace-health-probes, off by default, because probes at ~1/s per proxy would dominate trace volume.
• Trace keys are traceID/spanID, lowerCamel like reconcileID and providerID, deliberately not the trace_id many Grafana derived-field examples assume — configure the derived-field regex accordingly. logr sinks can't read ctx, so the IDs ride on the ctx logger, re-derived from a captured base per span so zap never emits duplicate keys.
• Probe→reconcile trace links are not attempted: the health engine's GenericEvent carries only namespace/name (no ctx), and the workqueue coalesces events, so any link would be a guess. A health-triggered reconcile starts a fresh trace; the probe that caused it is findable via its own (opt-in) span and shared proxy attributes.
• No traceparent toward probe targets: probe transports stay uninstrumented so trace headers can never leak through a proxy to external sites. The kube-apiserver ignores incoming traceparent by design (public endpoint), so k8s client spans are leaves — in-process traces, not cross-process ones.
• Provider health is a status-only, cluster-scoped CRD (user decision). kubectl get providerhealth works without Prometheus, which the deployment does not ship (its ServiceMonitor is commented out). There is no spec: the providers config file stays the one place providers are configured, and the object only reports on it. The gauges carry the same verdicts for alerting.
• KVM checks reuse the pooled connection instead of dialing fresh. A fresh dial would not pick up a rotated Secret anyway, since the SSH material is cached. It would also open an sshd session per host per minute. And a working cached connection is exactly what reconciles use. Sharing must not cost the reconciles anything, so the check calls through hostPool.probe. A check that runs out of time abandons its RPCs instead of closing the connection, which would kill an upload or a Create running on it. At most one abandoned check per host is left waiting, and until it returns, the next check reports Timeout at once. Waiting for a host's connection slot also ends with the caller's deadline, so a reconcile redialing a dead host cannot stall the round.
• Provider transitions are logs, not Events, in line with the EventRecorder decision above. They log at Info with a WARNING: prefix, and each target's lastTransitionTime in the object records when it flipped.
• A provider name that is not a valid object name gets metrics but no ProviderHealth object, with a warning at start. Tightening config validation instead would crash-loop existing deployments on upgrade.
• kvm2 is a second provider type built clean-room from the contract, not a rewrite of kvm in place. Its spec (docs/kvm2-provider-spec.md) and code were written from provider-contract.md, the kvm user guide and go-libvirt alone, never from internal/provider/kvm/, as a test of whether the contract is enough to build a provider. Both types ship, share the KVMConfig block shape and the kvm-ssh Secret, and may share a host.
• kvm2 image downloads run on a dedicated connection. A cancelled or overlong call kills the connection it ran on (the only way to unblock go-libvirt), so a 435 MB upload on the pooled connection could die because a poll for another Proxy was cancelled. The transfer dials its own connection and closes it afterwards; the cheap checks and the marker stay on the pooled one.
• kvm2 health checks dial fresh connections, the opposite of the kvm decision above. With credentials re-read on every dial a fresh connection does pick up a rotated Secret, and a check that runs out of time then closes only its own connection, so it cannot disrupt a reconcile. The cost is one sshd session per host per round.
• A cancelled kvm2 call kills the pooled connection. Every RPC has a budget; when a call outlives its ctx or budget the adapter closes the socket and the SSH client, returns Transient, and marks the connection dead so the next call redials. Other calls in flight on that host fail once with Transient and their callers retry. Documented in the guide rather than avoided: the alternative, leaving a hung RPC blocked forever, pins a worker.
• kvm2 marks domains in its own metadata namespace, so it never lists or adopts domains the kvm provider made. Switching an entry from kvm to kvm2 means deleting its Proxies first; the two types can share a host without one collecting the other's VMs.
• kvm2 turns pool free space into the quota signal. libvirt reports a full pool as an ordinary error, which would retry on the fast backoff. Before creating any volume, Create compares StoragePoolGetInfo's available bytes with the image (if uncached) plus diskSizeGb plus 16 MiB for the seed, and returns ErrQuotaExceeded below that, so the reconciler waits 5 minutes instead.
• kvm2 classifies SSH authentication and host-key failures as Permanent. A key missing from authorized_keys or a host key missing from or different in known_hosts needs an admin; kvm retried these forever. A missing key or known_hosts file stays Transient, because the optional Secret may land later.
• kvm2 re-reads the SSH key and known_hosts on every dial, so rotating the Secret needs no restart; nothing is cached. The constructor only checks that the files exist, to warn early without failing the provider set.