# Architecture ## Components | Component | Package | Runs as | Leader-elected | Role | |---|---|---|---|---| | Proxy + ProviderHealth CRDs | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers; the status-only, cluster-scoped `ProviderHealth` | | Reconciler | `internal/controller` | controller | yes (with the manager) | the state machine: provision, replace, delete, represent health | | Provider contract | `internal/provider` | library | — | `Provider` interface, optional `Checker`, error taxonomy, deterministic naming, config, metrics decorator; rules in [provider-contract.md](provider-contract.md), per-provider profile table in [providers-existing.md](providers-existing.md) | | kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) | | gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops | | kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds | | kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial | | Shared provider helpers | `internal/provider/nocloud`, `internal/provider/keyedlock` | library | — | the NoCloud seed ISO and vendor-data builder, and one-slot semaphores by key; used by kvm2 and proxmox | | Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events | | Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex | | Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` | | Orphan GC | `internal/gc` | Runnable | yes | deletes tagged instances whose CR is gone | | Provider health monitor | `internal/providerhealth` | Runnable | yes | periodic provider checks → one `ProviderHealth` per provider, target gauges, transition logs | | Metrics | `internal/metrics` | library | — | explicit registration, scrape-time collectors | | Composition root | `cmd/main.go` | binary | — | flags, provider registry, wires everything onto one manager | ## Event flow: cluster events → reconciler functions Which functions run in response to which Kubernetes cluster events, as wired in `internal/controller/proxy_controller.go`. ### 1. How cluster events reach the reconciler ```text KUBERNETES CLUSTER EVENTS (wiring: SetupWithManager, ───────────────────────── proxy_controller.go) Proxy CR created / spec edited / Secret created / health transition status patched / delete requested updated / deleted (engine, see §6) │ │ │ watch: For(&crawlv1alpha1.Proxy{}) watch: Watches( WatchesRawSource( │ &corev1.Secret{}, ...) source.Channel( │ │ r.HealthEvents, ...)) │ r.proxiesForSecret(ctx, secret) │ │ │ r.List(Proxies in namespace) │ │ │ keeps those whose │ │ │ spec.cloudInit.secretRef matches │ │ ▼ │ │ [reconcile.Request per matching Proxy] │ ▼ │ │ ┌────────────────────────────────────┴──────────────────────────┴──┐ │ controller-runtime workqueue │◄── RequeueAfter │ (dedup by namespace/name, rate-limited, │ timers │ MaxConcurrentReconciles: 3) │◄── error backoff └──────────────────────────┬────────────────────────────────────────┘ ▼ ProxyReconciler.Reconcile(ctx, req) ``` The Secret watch makes *rotating a cloud-init Secret* a first-class event: it re-enqueues every Proxy referencing that Secret, which is how secret rotation triggers VM replacement even though the Proxy spec is untouched. ### 2. Inside `Reconcile` — dispatch and the single status write ```text Reconcile(ctx, req) │ r.Get(ctx, req.NamespacedName, &p) ── fetch the Proxy (NotFound → done) │ base := p.DeepCopy() ── snapshot for the diff │ defer patchStatusIfChanged(ctx, base, &p) ──────────────────────────┐ │ │ ├─ p.DeletionTimestamp set ──► reconcileDelete(ctx, &p) │ ├─ p.Spec.Mode == External ──► reconcileExternal(ctx, &p) │ └─ otherwise (Managed) ──────► reconcileManaged(ctx, &p) │ ▼ patchStatusIfChanged (status.go) │ p.Status.ObservedGeneration = p.Generation │ p.Status.Phase = computePhase(&p) │ equality.Semantic.DeepEqual(base, p)? └─ changed → r.Status().Patch(...) ◄── the ONLY unchanged → no API call status write ``` ### 3. `reconcileManaged` — the state machine ```text reconcileManaged(ctx, p) │ ├─ controllerutil.AddFinalizer? ──► r.Update ──► return {} (watch event re-triggers) ├─ permanent-failure latch (FindStatusCondition == PermanentError │ at this generation) ──► return {} (silent until spec edit) ├─ r.Providers[p.Spec.Provider] missing ──► setProvisioned(PermanentError) → Failed │ ├─ resolveCloudInit(ctx, p) ──► r.Get(Secret) if secretRef (error → CloudInitError + backoff) ├─ hash := specHash(p, cloudInit) (spechash.go) │ ├─ status.providerID == "" ─────────► prov.Create(CreateRequest{Name: NameFromUID(p.UID), ...}) │ │ setSpecHash → r.Update (annotation) │ └ stage providerID + Provisioned=False/Provisioning │ ──► RequeueAfter: ProvisioningPoll │ ├─ annotation != hash, annotation == "" ──► adopt: setSpecHash → r.Update │ ──► RequeueAfter: RequeueNow ├─ annotation != hash, annotation != "" ──► replaceInstance: │ prov.Get ─ NotFound → setSpecHash, clear ID/IP │ │ ──► RequeueNow (next pass creates) │ └ exists → prov.Delete, Provisioned=False/Replacing │ ──► RequeueAfter: DeletionPoll │ └─ annotation == hash ──► prov.Get(providerID) ├─ ErrNotFound ──► clear ID/IP ──► RequeueNow (next pass creates) ├─ Provisioning ──► Provisioned=False ──► ProvisioningPoll ├─ Running ──► status.ip = inst.IP, │ Provisioned=True/Created, │ applyHealth (see §6) ──► DriftPoll └─ Stopped/Termin. ──► prov.Delete (cattle) ──► DeletionPoll any provider error ──► providerFailure(p, err) ── provider.Class(err): ├─ ErrQuotaExceeded ──► condition QuotaExceeded ──► RequeueAfter: QuotaRetry (nil error) ├─ ErrPermanent ──► condition PermanentError ──► phase Failed, no retry └─ ErrTransient ──► return err ──► workqueue exponential backoff ``` ### 4. `reconcileDelete` and `reconcileExternal` ```text reconcileDelete(ctx, p) reconcileExternal(ctx, p) ├─ no finalizer ──► return {} │ status.ip = spec.endpoint.host ├─ providerID == "" ──► RemoveFinalizer │ setProvisioned(True/ExternalEndpoint) │ → r.Update → object actually deleted │ applyHealth (see §6) ├─ prov.Get → ErrNotFound ──► RemoveFinalizer └─ return {} (no finalizer, │ → r.Update → object actually deleted no provider calls ever) └─ exists ──► prov.Delete → Provisioned=False/Deleting ──► RequeueAfter: DeletionPoll (poll until gone) ``` ### 5. What provider calls do in the outside world ```text kubernetes pod provider (internal/provider/kubernetes/) prov.Create ──► buildPod (pure) ──► client.Create(corev1.Pod) ─┐ these cause Pod events, prov.Get ──► client.Get(Pod) → phase/IP → InstanceState │ but the operator does NOT prov.Delete ──► client.Delete(Pod, tolerate NotFound) │ watch Pods — it observes prov.ListByTag ─► client.List(Pods by labels, all namespaces) ─┘ them by polling prov.Get on each RequeueAfter tick gcp provider (internal/provider/gcp/) — instances.{Insert,Get,Delete,AggregatedList}, nothing else prov.Create ──► buildInsertRequest (pure) ──► instances.Insert ─┐ fire-and-forget: 409 alreadyExists = success (idempotent retry) │ Operation.Wait is never prov.Get ──► instances.Get → status/NatIP → InstanceState │ called; readiness is RUNNING without NatIP = still Provisioning │ discovered by Get polls, prov.Delete ──► instances.Delete (404 = success) │ exactly like the pod prov.ListByTag ─► AggregatedList(label filter, ─┘ provider ReturnPartialSuccess: true) providerID = zones//instances/ — zone-qualified, so Get/Delete stay correct even mid-replacement after a zone edit kvm provider (internal/provider/kvm/) — libvirt RPC over an SSH-forwarded unix socket, per host prov.Create ──► DomainLookupByName: exists → start if needed, done ─┐ every step tolerates else: ensure base image in pool (download on first use) │ the previous attempt overlay volume (exists = reuse) │ having got that far; seed ISO (always rebuilt — a cut-off upload must │ the seed is the one never be trusted) │ thing redone DomainDefineXML + DomainCreate ─┘ prov.Get ──► DomainGetState + DomainGetMetadata + DomainInterfaceAddresses (ARP, then guest agent; first routable IPv4) running without IP = Provisioning; nostate = Provisioning prov.Delete ──► destroy (not running = fine) → undefine → delete overlay + seed by name prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose carries managed=true; one host down fails the sweep, one unreadable domain is skipped providerID = hosts//domains/; GC identity lives in the domain's element (libvirt has no labels). All RPCs run under a context- bounded call: go-libvirt never times out on its own, so a cancelled reconcile closes the SSH connection to unblock it. kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract prov.Create ──► placement checked against the catalogue (Permanent before any dial) DomainLookupByName: exists → mark+uid guard (no mark or foreign ─┐ every step uid = Permanent; paused = Permanent), │ start if needed, wait for running, done │ idempotent, else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a + 16 MiB must fit, else ErrQuotaExceeded (reconciler │ per-name lock waits 5m) │ shared with ensure base image: marker present = cached; otherwise GET │ Delete streamed into StorageVolUpload on a DEDICATED connection, │ StoragePoolRefresh, marker written last │ overlay volume (exists = reuse) │ seed ISO (always rebuilt) │ DomainDefineXML ( in kvm2's own namespace: │ managed, uid, createdAt) + DomainCreate, poll until │ running (15s cap; a domain that dies at once = Transient) ─┘ prov.Get ──► DomainLookupByName + DomainGetMetadata + DomainGetState + DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4) running without IP = Provisioning; nostate = Provisioning; only ErrNoDomain is NotFound prov.Delete ──► destroy (not running = fine) → undefine (nvram, managed save, snapshots) → delete overlay + seed by name; unconfigured host = logged no-op prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose kvm2 says managed=true; one host down fails the sweep, one unreadable domain is skipped Check ──► per host, on a FRESH connection: version, hostname, pool lookup, pool info, close; never the pooled one providerID = hosts//domains/. Every RPC has its own budget (dial 10s, RPC 30s, DomainCreate 90s, seed upload 60s, image transfer 20m): a call that outlives its ctx or budget kills the pooled connection (the only way to unblock go-libvirt), so other calls in flight on that host fail once with Transient and retry; the next call redials. Each RPC is a child span kvm2. under provider.*, plus kvm2.dial and kvm2.download. ``` The reconciler never watches provider-side resources (Pods or GCP VMs). All instance-state observation is poll-based through the `Provider` interface, so the same flow works identically for a cloud API that has no watch mechanism at all. ### 6. Health engine (`internal/health/`) — probes and transitions The engine is a leader-elected manager Runnable with its own goroutines, independent of the workqueue. It owns health *state*; the reconciler owns its *representation* in status — that split keeps exactly one writer of `.status` and makes write-only-on-transition fall out for free. ```text Engine.Start(ctx) (engine.go) ├─ spawns Workers (8) probe goroutines ◄─┐ └─ ticker loop (Tick = 1s): │ jobs channel (non-blocking send; tick(ctx, now, jobs) │ saturated pool → retry next tick) │ Reader.List(Proxies) ── from the manager cache │ per proxy: skip if no IP/host or deleting (state pruned → │ a replaced instance starts with fresh counters) │ newState: seed verdict from an existing Healthy condition │ (leader handover), jitter first probe across the interval │ due && !inFlight ──► jobs ◄── probe worker picks up └ prune states for proxies gone from the cache │ probe(ctx, proxyURL, hc, tls) (probe.go) │ fresh transport per probe, DisableKeepAlives=true │ (load-bearing: keep-alives would cache the CONNECT │ tunnel and later probes would never re-exercise it) │ https probe URL ⇒ CONNECT through the proxy + TLS inside └ success = err == nil AND expected status code │ record(job, result, now) ── under one mutex │ counters: consecOK/consecFail; verdict flips only at │ successThreshold / failureThreshold │ emit ONLY on: first-ever verdict │ threshold flip │ │ latency Δ > max(20ms, 50% of reported) rate-limited │ to one report per MinReportInterval (60s) ▼ Events chan (buffered 64, non-blocking send; on drop the reported markers do NOT advance → next probe retries) │ ▼ source.Channel → workqueue → Reconcile (see §1) │ ▼ r.applyHealth(p) ── reads Engine.Snapshot(key) (status.go) stages the Healthy condition + latencyMillis + lastHealthCheckTime; computePhase turns Provisioned=True + Healthy=True/False into phase Ready / Unhealthy ``` Consequence worth knowing: `status.lastHealthCheckTime` is the time of the last *status-affecting* probe, not the most recent probe — suppressed probes deliberately never write status. True probe recency will live in metrics (Step 9). ### 7. Discovery + lease API (`internal/discovery/`, `internal/lease/`) HTTP-driven, not cluster-event-driven: crawler clients call in; the only Kubernetes interaction is reading Proxies from the manager's cache. The server is a non-leader-elected Runnable (all replicas would serve, but the deployment ships `replicas: 1` because lease state is per-process — an operator restart drops all leases and cooldowns, a documented caveat). Client-facing reference with request/response schemas and curl examples: [api.md](api.md). ```text crawler client │ Authorization: Bearer $DISCOVERY_TOKEN (empty token = auth disabled, loud startup warning) ▼ Server.handler() middleware, outermost first (server.go) recover → request-log → MaxBytesReader(64KiB) → bearer auth (constant-time; /healthz exempt) │ ├─ GET /healthz ──► 200 ok (unauthenticated) │ ├─ GET /v1/proxies?attr.k=v&healthy=true (handlers.go) │ Reader.List(Proxies) ── manager cache │ filter: attributes equality + Healthy condition │ + Store.Counts() for activeLeases │ ──► 200 {"proxies":[...], "count":N} (empty list is 200, not 404) │ ├─ POST /v1/leases {"selector":{...},"ttlSeconds":300,"target":"..."} │ Reader.List → filter selector; unhealthy matches counted, not offered │ Store.Acquire(healthy candidates, target, ttl) ── one lock: select+insert │ │ selection: fewest active leases, then latency, then name │ ├─ granted ──► 201 {leaseID, proxy:{...}, expiresAt, ttlSeconds} │ └─ ErrNoMatch ──► 409 {"error":"no_match", considered, atCapacity, │ inCooldown, unhealthy} │ ├─ DELETE /v1/leases/{id} ──► Store.Release ──► always 204 (idempotent) │ └─ POST /v1/leases/{id}/report {"result":"ok|rate_limited|banned","target":"..."} Store.Report ── rate_limited/banned ⇒ cooldown[{proxy,target}] for │ CooldownWindow (target falls back: report → lease → global) ├─ 204 │ 400 invalid_result │ 404 unknown_lease └─ an expired lease still resolves for CooldownWindow past its TTL — a late report lands exactly when the proxy is being rate-limited Store.Start(ctx) ── manager Runnable, NOT leader-elected: sweeps expired leases + cooldowns; correctness never depends on the sweep (every read checks ExpiresAt against the clock) ``` ### 8. Orphan GC (`internal/gc/`) — the crash-safety net Timer-driven, leader-elected (destructive ⇒ single writer). Exists for the one gap the reconciler cannot close alone: a crash after a provider Create but before the status write that records the instance. ```text Sweeper.Start(ctx) ── refuses to run when the cache is namespace- │ restricted unless --gc-allow-namespaced is explicit │ (an incomplete live set would "orphan" live VMs) └─ every Interval (10m; first sweep a full interval after start): sweep(ctx) │ Reader.List(Proxies) → live UID set │ List fails → skip the whole sweep (never guess) │ a CR with deletionTimestamp still counts as LIVE — its │ finalizer owns that deletion; GC racing it double-deletes └ per provider: ListByTag │ error → log, continue with the next provider └ delete only when ALL hold: has the proxy-operator-uid label (ownership proof) older than MinAge (10m) (not mid-create) UID matches no existing CR (truly orphaned) each kill logged loudly with provider, providerID, UID ``` ### 9. Metrics (`internal/metrics/`) Registered explicitly from `cmd/main.go` (no `init()`; tests use fresh registries). Two kinds: - **Scrape-time collectors** — `proxy_operator_proxies{phase}` and `proxy_operator_leases_active` read the cache / lease store at every scrape; reconcile-incremented gauges inevitably drift and leak series. The provider-check gauges (§11) read the health monitor's snapshot the same way. - **Fed vectors** — `healthcheck_duration_seconds{proxy}` and `healthcheck_failures_total{proxy}` observe EVERY probe (status writes are transition-only; metrics carry the high-frequency signal), and the health engine deletes a proxy's series when it prunes its state; `lease_requests_total{outcome}` from the discovery handlers; `provider_requests_total{provider,op,result}` from the `provider.WithMetrics` decorator — the one place `Class()` is called purely for observability. Each consuming package defines its own small recorder interface (`health.ProbeMetrics`, `discovery.LeaseMetrics`, `provider.RequestRecorder`); `metrics.Metrics` satisfies all of them structurally, so no package other than `cmd/main.go` imports the metrics package. ### 10. Tracing (`internal/tracing/`) OpenTelemetry tracing, integrated with — not replacing — the logr/zap logging. Everything hangs off standard `OTEL_*` env vars: with no `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_TRACES_EXPORTER` set, no SDK is installed, spans are no-ops, and logs are byte-for-byte what they were. `OTEL_SDK_DISABLED=true` and `OTEL_TRACES_EXPORTER=none` force it off; `console` prints spans to stdout for local dev; sampling follows `OTEL_TRACES_SAMPLER(_ARG)`. Span topology — each unit of work is a new trace (watch events carry no incoming trace context): - `Reconcile Proxy` (root, via a `reconcile.Reconciler` decorator) → `reconcile.managed` / `reconcile.replaceInstance` / `reconcile.delete`, plus `status.patch` from the deferred flush. `reconcileExternal` does no I/O and is unspanned. - `provider.create|get|delete|list` (client spans, `provider.WithTracing` decorator wired outermost around `WithMetrics`); the GCP SDK's own otelhttp transport contributes HTTP child spans automatically. - kvm2 child spans: under `provider.*`, one `kvm2.` client span per libvirt RPC (for example `kvm2.DomainDefineXML`, attributes `kvm2.host`, `kvm2.domain`, `kvm2.pool`, `kvm2.volume`, `kvm2.libvirt_code` on failure), `kvm2.dial` around the SSH handshake and libvirt connect, and `kvm2.download` around an image fetch; under `providerhealth.check`, `kvm2.dial` and the four `kvm2.` spans of a check. They come from a span decorator over the seam, applied once in the dialer, so no RPC is unspanned. The `libvirt call` log line of an RPC is written inside its span. The `kvm` provider has no child spans. - Kubernetes API calls become client child spans via a wrapped `rest.Config` transport — gated on an existing parent span, so informer list/watch long-polls and leader-election renewals never create root spans. The kube-apiserver ignores incoming `traceparent` by design; these spans are leaves. - Discovery API: one server span per request (named from the route pattern), incoming W3C `traceparent` honored so clients' traces continue into the operator; `/healthz` excluded. - `gc.sweep` per GC pass; `providerhealth.check` per round of provider checks; `health.probe` per probe **only** with `--trace-health-probes` (default off — probes run ~1/s per proxy). Log correlation: logr sinks never see a context, so trace IDs ride on the logger — `tracing.Start` re-derives the ctx logger from a captured base with `traceID`/`spanID` values (lowerCamel, matching `reconcileID`; Grafana/Loki derived-field regexes must match `traceID`, not `trace_id`). Re-deriving from the base rather than layering keeps zap from emitting duplicate keys on nested spans. GCP V(5) wire logs get the same keys from the slog handler's ctx. ### 11. Provider health checks (`internal/providerhealth/`) Answers "can this cluster reach each provider's backend right now?" without waiting for a Proxy to fail. A provider opts in by implementing `provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per target and never fails as a whole. All four providers do: | Provider | Target | What one check does | |---|---|---| | kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) | | kvm2 | each `kvm2.hosts` entry | on a fresh SSH+libvirt connection dialled for the check and closed after it: libvirt version and hostname, then the configured pool must exist and be running; `details` carry `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB` and `poolAvailableGiB` ([details](kvm2-provider.md#health-check)) | | kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on | | gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` | ```text Monitor.Start(ctx) ── leader-only; --provider-check-interval (1m; 0 = off, │ otherwise ≥ 10s), --provider-check-timeout (15s, or │ the interval if that is shorter) ├─ once: delete managed-by-labelled ProviderHealth objects whose provider │ is not configured (all of them when checks are off) — the providers │ config is read once per process └─ at start, then every Interval: checkOnce │ every provider's Check concurrently, under one context.WithTimeout │ snapshot → the scrape-time gauges │ log transitions: down or reason changed → "WARNING: provider target │ not ready"; back up → "Provider target ready"; steady → silent └ per provider: APIReader.Get → Create if missing → status merge patch ``` - **One status write per provider per round**, since `lastCheckTime` moves every round. At the 10s minimum interval that is still far below leader-election renewals. Proxy status avoids writes like this only because its probes run once a second per proxy. - **Reads go through the API reader**, not the cache. The objects are cluster-scoped; a cached Get would start a cluster-wide informer (and need `watch`) even with `--proxy-namespace`. - **Checks bypass the provider decorators.** `cmd/main.go` collects the Checkers before wrapping the providers in `WithTracing(WithMetrics(...))`, so a check never counts as a provider request. KVM RPCs still appear in the V(1) `libvirt call` trace, and each round is one `providerhealth.check` span. - **Not wired to `/readyz`.** A dead hypervisor must not make the manager unready while the other providers keep working. - **Gauges** `proxy_operator_provider_target_up`, `proxy_operator_provider_check_duration_seconds` and `proxy_operator_provider_check_timestamp_seconds`, each labelled `{provider,target}`. Only the leader exports them. Illustrative output: ```sh $ kubectl get providerhealth # short name: ph NAME TYPE READY TARGETS LAST CHECK AGE kubernetes kubernetes True 1/1 12s 3d kvm-home kvm False 0/1 12s 3d $ kubectl get ph kvm-home -o yaml # per target: reason, message, details ``` ## Decisions Judgment calls the spec left open, and deliberate deviations — recorded so they read as choices, not accidents. Chronological by build step. - **Registry takes its constructor map as a parameter** instead of holding a package-level map: avoids the provider⇄registry import cycle and puts the wiring at the composition root, where it is visible. - **Mock provider replaced by the kubernetes-pod provider** (user decision, mid-build): a simulated in-memory provider was too far from the real system to build confidence in. Local dev/CI now runs real `ubuntu/squid` pods (Canonical's actively maintained image, verified 50M+ pulls, pinned tag) in the operator's own cluster. Trade-off accepted: envtest has no kubelet, so end-to-end proof lives in the kind quickstart, and cluster pods share one egress IP — distinct egress paths remain the GCP provider's job. - **`RequeueAfter: RequeueNow` instead of the plan's `Requeue: true`:** `ctrl.Result.Requeue` is deprecated in controller-runtime v0.24; a fifth configurable interval (default 1s) keeps identical semantics and stays shrinkable in tests. - **Quota exhaustion is a wait, not a failure:** `ErrQuotaExceeded` sets a condition and requeues slowly (5m) with a nil error — off the backoff curve, out of the error log, and never `phase: Failed`. Only `ErrPermanent` latches Failed, keyed to the generation so a spec edit auto-recovers. - **The finalizer path never latches permanent failures:** a permanent error during deletion keeps retrying visibly instead — latching there would wedge the object forever with no path out but manual finalizer surgery. - **Health transitions travel reconciler-ward over a channel** (`source.Channel`), not direct status patches: `phase` derives from both provisioning and health, so two status writers would race and flap. One writer of status; the engine owns health *state*, the reconciler its *representation*; write-only-on-transition falls out for free. - **Health state seeds from the existing Healthy condition on leader handover** (verdict kept, counters zeroed, first probe jittered), so a healthy fleet doesn't flap to Unknown on restart — but a real transition still needs a full threshold run. A never-probed proxy skips the jitter and probes on the next tick: startup spread matters for restarts, not for a single new proxy. - **Latency suppression is `max(20ms, 50%)` + a 60s rate limit, and only while the verdict is healthy.** The spec's bare ">50% change" is undefined at 0 and lets a proxy jittering 40↔61ms write status forever; the healthy-only guard (found by test) stops a below-threshold success streak from emitting latency updates for a proxy still reported unhealthy. Consequence: `status.lastHealthCheckTime` means "last status-affecting probe" — true probe recency is in the metrics. - **Deterministic instance names** are `proxy-` + 16 chars of base32(SHA-256(CR UID)): legal for both GCP (`[a-z2-7]` ⊂ `[-a-z0-9]`, 22 ≤ 63 chars) and Pod names, 80 bits against birthday collisions at a fleet of tens. The replacement VM therefore has the *same name* as the one being deleted — which is why replacement polls to NotFound before recreating instead of racing a 409. - **`banned` and `rate_limited` share one cooldown window:** a second duration knob the spec doesn't ask for; the report's semantic difference is preserved in the API but not the store. - **Report targets fall back report → lease → global**, so a client that leased with a target can't accidentally poison the proxy's global pool by omitting the target in its report. - **The 409 body's `considered` counts unhealthy matches too** (the store only ever sees healthy candidates): `considered = atCapacity + inCooldown + unhealthy + eligible-but-outranked`, keeping the numbers additive for a human debugging "why no proxy?". - **TTLs above `--max-lease-ttl` are a 400, not a silent clamp** — a client asking for a week should find out. - **Discovery is not leader-elected and ships `replicas: 1`:** caches start before non-leader-election runnables (verified in controller-runtime's ordering), and a leader-elected server would leave non-leader replicas as broken Service endpoints. One replica because lease state is per-process. - **GCP `Create` requires zone, machineType, and image** and fails `ErrPermanent` naming the missing field — inventing machine-type defaults would silently create billable VMs of arbitrary shape. - **Unknown GCP instance statuses map to `Stopped`:** the reconciler's answer to Stopped is delete-and-recreate, the always-safe move for cattle when the API grows a new state. - **KVM hosts are reached over SSH with the libvirt socket forwarded,** not libvirtd's TCP/TLS listener: nothing to configure or expose on the hypervisors beyond sshd, pure Go on the operator side (no cgo, no libvirt libraries in the image), and the hosts list can grow by a config edit because the providerID carries the host name. - **KVM base images are fetched by the operator into the host's pool** through libvirt's upload stream, rather than expected to be pre-placed: the operator then needs no filesystem access on the host and a new host is usable with zero manual staging. The cost is a synchronous first-time download inside Create. - **The KVM seed ISO is rebuilt on every Create attempt** before the domain exists, instead of "exists → skip": a reconcile cancelled mid upload leaves a truncated seed, and the compensating delete cannot run because the connection was already closed to unblock the RPC. Skipping it would boot a VM with no cloud-init and no retry would ever repair it. - **KVM `nostate` maps to `Provisioning`, not `Stopped`:** unlike GCP's unknown statuses, `nostate` means libvirt could not determine the state at all; answering that with delete-and-recreate is too drastic, so the next poll decides. - **KVM guests boot UEFI by default, secure boot off.** Found on the first real host: Debian 13's generic image hangs in GRUB under SeaBIOS on a headless VM (Debian bug #1111240: `terminal_output gfxterm serial` with a serial console and no video device) regardless of CPU model or seed, and boots under OVMF with the serial console working. The alternatives were a virtio console (loses the kernel's serial output) or a video device the VMs have no use for. Every current distro cloud image supports UEFI, so it is the default and `firmware: bios` is the per-image escape; libvirt's firmware autoselection (``) keeps the OVMF path out of the operator's config. Secure boot is off so unsigned images boot too. - **A KVM domain GC cannot read is skipped, a host it cannot reach fails the sweep.** GC never deletes what it does not see, so leaving one domain out is safe; one corrupt domain blocking GC for every other one is not. A whole host missing from the view would hide many, so that still aborts. - **Kubernetes 403s classify as `ErrPermanent`** even though quota exhaustion also surfaces as 403 (indistinguishable from RBAC denial in `apierrors`): not hammering an API server that may never allow the request is the safer default; a real ResourceQuota 403 forgoes the gentler quota backoff. Documented at the classification site. - **GC kills log at Info with a `WARNING:` prefix** — logr has no Warn level; the plan's "log at Warn" is met in spirit with provider, providerID, and UID always attached. Same convention as the discovery server's empty-token warning. - **GC trusts only provable orphans:** instances without the UID label are never deleted, a CR with a deletionTimestamp still counts as live (its finalizer owns that deletion), and an unreadable Proxy list skips the whole sweep. The namespace guard refuses to sweep a namespace-restricted cache without `--gc-allow-namespaced`. - **Cloud-init Secrets must carry `crawl.example.com/cloud-init: "true"`:** the manager caches only labelled Secrets (the operator holds cluster-wide Secret read RBAC — an unrestricted cache would hold every Secret in scope). Unlabelled referenced Secrets are invisible by construction, surfacing as `CloudInitError`. - **Deletion never gives up on its own, but can be released by hand:** a permanent cleanup failure keeps retrying with the finalizer on, because dropping it automatically would leak a running instance on any provider that answers "permission denied". Where `Get` fails permanently the reconciler still calls `Delete`, since it is idempotent and may finish where `Get` cannot converge. `crawl.example.com/skip-cleanup: "true"` abandons the instance and releases the object; it is checked before the provider lookup, so it also frees a Proxy whose provider is gone from the config. - **Events RBAC from the plan is omitted:** nothing wires an EventRecorder in the prototype, and granting verbs nothing uses would be RBAC lint noise. Add the marker together with the recorder if events land later. - **logr, not slog, inside controller paths:** the repo convention says `slog`, but `log.FromContext(ctx)` hands controller-runtime's logr logger to everything running under the manager — fighting that would mean two logging systems in one process. Noted as a deviation rather than silently ignored. - **Tracing is env-gated, not flag-gated:** it activates only when `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_TRACES_EXPORTER` is set (user decision). No collector configured means no SDK installed, no-op spans, unchanged logs — the safe default for every existing deployment. The one flag is `--trace-health-probes`, off by default, because probes at ~1/s per proxy would dominate trace volume. - **Trace keys are `traceID`/`spanID`**, lowerCamel like `reconcileID` and `providerID`, deliberately not the `trace_id` many Grafana derived-field examples assume — configure the derived-field regex accordingly. logr sinks can't read ctx, so the IDs ride on the ctx logger, re-derived from a captured base per span so zap never emits duplicate keys. - **Probe→reconcile trace links are not attempted:** the health engine's `GenericEvent` carries only namespace/name (no ctx), and the workqueue coalesces events, so any link would be a guess. A health-triggered reconcile starts a fresh trace; the probe that caused it is findable via its own (opt-in) span and shared proxy attributes. - **No `traceparent` toward probe targets:** probe transports stay uninstrumented so trace headers can never leak through a proxy to external sites. The kube-apiserver ignores incoming `traceparent` by design (public endpoint), so k8s client spans are leaves — in-process traces, not cross-process ones. - **Provider health is a status-only, cluster-scoped CRD** (user decision). `kubectl get providerhealth` works without Prometheus, which the deployment does not ship (its ServiceMonitor is commented out). There is no spec: the providers config file stays the one place providers are configured, and the object only reports on it. The gauges carry the same verdicts for alerting. - **KVM checks reuse the pooled connection instead of dialing fresh.** A fresh dial would not pick up a rotated Secret anyway, since the SSH material is cached. It would also open an sshd session per host per minute. And a working cached connection is exactly what reconciles use. Sharing must not cost the reconciles anything, so the check calls through `hostPool.probe`. A check that runs out of time abandons its RPCs instead of closing the connection, which would kill an upload or a Create running on it. At most one abandoned check per host is left waiting, and until it returns, the next check reports `Timeout` at once. Waiting for a host's connection slot also ends with the caller's deadline, so a reconcile redialing a dead host cannot stall the round. - **Provider transitions are logs, not Events**, in line with the EventRecorder decision above. They log at Info with a `WARNING:` prefix, and each target's `lastTransitionTime` in the object records when it flipped. - **A provider name that is not a valid object name** gets metrics but no `ProviderHealth` object, with a warning at start. Tightening config validation instead would crash-loop existing deployments on upgrade. - **kvm2 is a second provider type built clean-room from the contract,** not a rewrite of `kvm` in place. Its spec (`docs/kvm2-provider-spec.md`) and code were written from [provider-contract.md](provider-contract.md), the kvm user guide and go-libvirt alone, never from `internal/provider/kvm/`, as a test of whether the contract is enough to build a provider. Both types ship, share the `KVMConfig` block shape and the `kvm-ssh` Secret, and may share a host. - **kvm2 image downloads run on a dedicated connection.** A cancelled or overlong call kills the connection it ran on (the only way to unblock go-libvirt), so a 435 MB upload on the pooled connection could die because a poll for another Proxy was cancelled. The transfer dials its own connection and closes it afterwards; the cheap checks and the marker stay on the pooled one. - **kvm2 health checks dial fresh connections**, the opposite of the kvm decision above. With credentials re-read on every dial a fresh connection does pick up a rotated Secret, and a check that runs out of time then closes only its own connection, so it cannot disrupt a reconcile. The cost is one sshd session per host per round. - **A cancelled kvm2 call kills the pooled connection.** Every RPC has a budget; when a call outlives its ctx or budget the adapter closes the socket and the SSH client, returns Transient, and marks the connection dead so the next call redials. Other calls in flight on that host fail once with Transient and their callers retry. Documented in the guide rather than avoided: the alternative, leaving a hung RPC blocked forever, pins a worker. - **kvm2 marks domains in its own metadata namespace**, so it never lists or adopts domains the `kvm` provider made. Switching an entry from `kvm` to `kvm2` means deleting its Proxies first; the two types can share a host without one collecting the other's VMs. - **kvm2 turns pool free space into the quota signal.** libvirt reports a full pool as an ordinary error, which would retry on the fast backoff. Before creating any volume, `Create` compares `StoragePoolGetInfo`'s available bytes with the image (if uncached) plus `diskSizeGb` plus 16 MiB for the seed, and returns `ErrQuotaExceeded` below that, so the reconciler waits 5 minutes instead. - **kvm2 classifies SSH authentication and host-key failures as Permanent.** A key missing from `authorized_keys` or a host key missing from or different in known_hosts needs an admin; kvm retried these forever. A missing key or known_hosts file stays Transient, because the optional Secret may land later. - **kvm2 re-reads the SSH key and known_hosts on every dial**, so rotating the Secret needs no restart; nothing is cached. The constructor only checks that the files exist, to warn early without failing the provider set.