kacerrandClaude Opus 5
4e5b5d62df
The health check shares each host's libvirt connection with the reconciles, so it must not be able to take that connection away from them. - A timed-out check no longer drops the link. The check goes through the new hostPool.probe, which abandons its RPCs when the context ends and leaves the connection alone, so an image upload or a Create running on the same link is not cut off. A link that really died is closed by the keepalive. One abandoned probe per host may stay waiting; until it returns, the next check reports Timeout at once (errProbeBusy). - Waiting for a host's connection slot now ends with the caller's context. hostEntry used a mutex, so a check could wait out a reconcile redialing a dead host — minutes — past its own timeout. The slot is a one-slot channel and acquire bounds only the wait, never the uncontended take. - --provider-check-timeout defaults to 0 and derives min(15s, interval). The flat 15s default contradicted the validation that the timeout may not exceed the interval, so --provider-check-interval=12s alone failed to start. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
616 lines
37 KiB
Markdown
616 lines
37 KiB
Markdown
# Architecture
|
|
|
|
## Components
|
|
|
|
| Component | Package | Runs as | Leader-elected | Role |
|
|
|---|---|---|---|---|
|
|
| Proxy + ProviderHealth CRDs | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers; the status-only, cluster-scoped `ProviderHealth` |
|
|
| Reconciler | `internal/controller` | controller | yes (with the manager) | the state machine: provision, replace, delete, represent health |
|
|
| Provider contract | `internal/provider` | library | — | `Provider` interface, optional `Checker`, error taxonomy, deterministic naming, config, metrics decorator |
|
|
| kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) |
|
|
| gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
|
|
| kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
|
|
| Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events |
|
|
| Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
|
|
| Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
|
|
| Orphan GC | `internal/gc` | Runnable | yes | deletes tagged instances whose CR is gone |
|
|
| Provider health monitor | `internal/providerhealth` | Runnable | yes | periodic provider checks → one `ProviderHealth` per provider, target gauges, transition logs |
|
|
| Metrics | `internal/metrics` | library | — | explicit registration, scrape-time collectors |
|
|
| Composition root | `cmd/main.go` | binary | — | flags, provider registry, wires everything onto one manager |
|
|
|
|
## Event flow: cluster events → reconciler functions
|
|
|
|
Which functions run in response to which Kubernetes cluster events, as
|
|
wired in `internal/controller/proxy_controller.go`.
|
|
|
|
### 1. How cluster events reach the reconciler
|
|
|
|
```text
|
|
KUBERNETES CLUSTER EVENTS (wiring: SetupWithManager,
|
|
───────────────────────── proxy_controller.go)
|
|
|
|
Proxy CR created / spec edited / Secret created / health transition
|
|
status patched / delete requested updated / deleted (engine, see §6)
|
|
│ │ │
|
|
watch: For(&crawlv1alpha1.Proxy{}) watch: Watches( WatchesRawSource(
|
|
│ &corev1.Secret{}, ...) source.Channel(
|
|
│ │ r.HealthEvents, ...))
|
|
│ r.proxiesForSecret(ctx, secret) │
|
|
│ │ r.List(Proxies in namespace) │
|
|
│ │ keeps those whose │
|
|
│ │ spec.cloudInit.secretRef matches │
|
|
│ ▼ │
|
|
│ [reconcile.Request per matching Proxy] │
|
|
▼ │ │
|
|
┌────────────────────────────────────┴──────────────────────────┴──┐
|
|
│ controller-runtime workqueue │◄── RequeueAfter
|
|
│ (dedup by namespace/name, rate-limited, │ timers
|
|
│ MaxConcurrentReconciles: 3) │◄── error backoff
|
|
└──────────────────────────┬────────────────────────────────────────┘
|
|
▼
|
|
ProxyReconciler.Reconcile(ctx, req)
|
|
```
|
|
|
|
The Secret watch makes *rotating a cloud-init Secret* a first-class event:
|
|
it re-enqueues every Proxy referencing that Secret, which is how secret
|
|
rotation triggers VM replacement even though the Proxy spec is untouched.
|
|
|
|
### 2. Inside `Reconcile` — dispatch and the single status write
|
|
|
|
```text
|
|
Reconcile(ctx, req)
|
|
│ r.Get(ctx, req.NamespacedName, &p) ── fetch the Proxy (NotFound → done)
|
|
│ base := p.DeepCopy() ── snapshot for the diff
|
|
│ defer patchStatusIfChanged(ctx, base, &p) ──────────────────────────┐
|
|
│ │
|
|
├─ p.DeletionTimestamp set ──► reconcileDelete(ctx, &p) │
|
|
├─ p.Spec.Mode == External ──► reconcileExternal(ctx, &p) │
|
|
└─ otherwise (Managed) ──────► reconcileManaged(ctx, &p) │
|
|
▼
|
|
patchStatusIfChanged (status.go)
|
|
│ p.Status.ObservedGeneration = p.Generation
|
|
│ p.Status.Phase = computePhase(&p)
|
|
│ equality.Semantic.DeepEqual(base, p)?
|
|
└─ changed → r.Status().Patch(...) ◄── the ONLY
|
|
unchanged → no API call status write
|
|
```
|
|
|
|
### 3. `reconcileManaged` — the state machine
|
|
|
|
```text
|
|
reconcileManaged(ctx, p)
|
|
│
|
|
├─ controllerutil.AddFinalizer? ──► r.Update ──► return {} (watch event re-triggers)
|
|
├─ permanent-failure latch (FindStatusCondition == PermanentError
|
|
│ at this generation) ──► return {} (silent until spec edit)
|
|
├─ r.Providers[p.Spec.Provider] missing ──► setProvisioned(PermanentError) → Failed
|
|
│
|
|
├─ resolveCloudInit(ctx, p) ──► r.Get(Secret) if secretRef (error → CloudInitError + backoff)
|
|
├─ hash := specHash(p, cloudInit) (spechash.go)
|
|
│
|
|
├─ status.providerID == "" ─────────► prov.Create(CreateRequest{Name: NameFromUID(p.UID), ...})
|
|
│ │ setSpecHash → r.Update (annotation)
|
|
│ └ stage providerID + Provisioned=False/Provisioning
|
|
│ ──► RequeueAfter: ProvisioningPoll
|
|
│
|
|
├─ annotation != hash, annotation == "" ──► adopt: setSpecHash → r.Update
|
|
│ ──► RequeueAfter: RequeueNow
|
|
├─ annotation != hash, annotation != "" ──► replaceInstance:
|
|
│ prov.Get ─ NotFound → setSpecHash, clear ID/IP
|
|
│ │ ──► RequeueNow (next pass creates)
|
|
│ └ exists → prov.Delete, Provisioned=False/Replacing
|
|
│ ──► RequeueAfter: DeletionPoll
|
|
│
|
|
└─ annotation == hash ──► prov.Get(providerID)
|
|
├─ ErrNotFound ──► clear ID/IP ──► RequeueNow (next pass creates)
|
|
├─ Provisioning ──► Provisioned=False ──► ProvisioningPoll
|
|
├─ Running ──► status.ip = inst.IP,
|
|
│ Provisioned=True/Created,
|
|
│ applyHealth (see §6) ──► DriftPoll
|
|
└─ Stopped/Termin. ──► prov.Delete (cattle) ──► DeletionPoll
|
|
|
|
any provider error ──► providerFailure(p, err) ── provider.Class(err):
|
|
├─ ErrQuotaExceeded ──► condition QuotaExceeded ──► RequeueAfter: QuotaRetry (nil error)
|
|
├─ ErrPermanent ──► condition PermanentError ──► phase Failed, no retry
|
|
└─ ErrTransient ──► return err ──► workqueue exponential backoff
|
|
```
|
|
|
|
### 4. `reconcileDelete` and `reconcileExternal`
|
|
|
|
```text
|
|
reconcileDelete(ctx, p) reconcileExternal(ctx, p)
|
|
├─ no finalizer ──► return {} │ status.ip = spec.endpoint.host
|
|
├─ providerID == "" ──► RemoveFinalizer │ setProvisioned(True/ExternalEndpoint)
|
|
│ → r.Update → object actually deleted │ applyHealth (see §6)
|
|
├─ prov.Get → ErrNotFound ──► RemoveFinalizer └─ return {} (no finalizer,
|
|
│ → r.Update → object actually deleted no provider calls ever)
|
|
└─ exists ──► prov.Delete
|
|
→ Provisioned=False/Deleting
|
|
──► RequeueAfter: DeletionPoll (poll until gone)
|
|
```
|
|
|
|
### 5. What provider calls do in the outside world
|
|
|
|
```text
|
|
kubernetes pod provider (internal/provider/kubernetes/)
|
|
prov.Create ──► buildPod (pure) ──► client.Create(corev1.Pod) ─┐ these cause Pod events,
|
|
prov.Get ──► client.Get(Pod) → phase/IP → InstanceState │ but the operator does NOT
|
|
prov.Delete ──► client.Delete(Pod, tolerate NotFound) │ watch Pods — it observes
|
|
prov.ListByTag ─► client.List(Pods by labels, all namespaces) ─┘ them by polling prov.Get
|
|
on each RequeueAfter tick
|
|
|
|
gcp provider (internal/provider/gcp/) — instances.{Insert,Get,Delete,AggregatedList}, nothing else
|
|
prov.Create ──► buildInsertRequest (pure) ──► instances.Insert ─┐ fire-and-forget:
|
|
409 alreadyExists = success (idempotent retry) │ Operation.Wait is never
|
|
prov.Get ──► instances.Get → status/NatIP → InstanceState │ called; readiness is
|
|
RUNNING without NatIP = still Provisioning │ discovered by Get polls,
|
|
prov.Delete ──► instances.Delete (404 = success) │ exactly like the pod
|
|
prov.ListByTag ─► AggregatedList(label filter, ─┘ provider
|
|
ReturnPartialSuccess: true)
|
|
providerID = zones/<zone>/instances/<name> — zone-qualified, so Get/Delete
|
|
stay correct even mid-replacement after a zone edit
|
|
|
|
kvm provider (internal/provider/kvm/) — libvirt RPC over an SSH-forwarded unix socket, per host
|
|
prov.Create ──► DomainLookupByName: exists → start if needed, done ─┐ every step tolerates
|
|
else: ensure base image in pool (download on first use) │ the previous attempt
|
|
overlay volume (exists = reuse) │ having got that far;
|
|
seed ISO (always rebuilt — a cut-off upload must │ the seed is the one
|
|
never be trusted) │ thing redone
|
|
DomainDefineXML + DomainCreate ─┘
|
|
prov.Get ──► DomainGetState + DomainGetMetadata + DomainInterfaceAddresses
|
|
(ARP, then guest agent; first routable IPv4)
|
|
running without IP = Provisioning; nostate = Provisioning
|
|
prov.Delete ──► destroy (not running = fine) → undefine → delete overlay + seed by name
|
|
prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose
|
|
<metadata> carries managed=true; one host down fails the sweep,
|
|
one unreadable domain is skipped
|
|
providerID = hosts/<host>/domains/<name>; GC identity lives in the domain's
|
|
<metadata> element (libvirt has no labels). All RPCs run under a context-
|
|
bounded call: go-libvirt never times out on its own, so a cancelled reconcile
|
|
closes the SSH connection to unblock it.
|
|
```
|
|
|
|
The reconciler never watches provider-side resources (Pods or GCP VMs).
|
|
All instance-state observation is poll-based through the `Provider`
|
|
interface, so the same flow works identically for a cloud API that has no
|
|
watch mechanism at all.
|
|
|
|
### 6. Health engine (`internal/health/`) — probes and transitions
|
|
|
|
The engine is a leader-elected manager Runnable with its own goroutines,
|
|
independent of the workqueue. It owns health *state*; the reconciler owns
|
|
its *representation* in status — that split keeps exactly one writer of
|
|
`.status` and makes write-only-on-transition fall out for free.
|
|
|
|
```text
|
|
Engine.Start(ctx) (engine.go)
|
|
├─ spawns Workers (8) probe goroutines ◄─┐
|
|
└─ ticker loop (Tick = 1s): │ jobs channel (non-blocking send;
|
|
tick(ctx, now, jobs) │ saturated pool → retry next tick)
|
|
│ Reader.List(Proxies) ── from the manager cache
|
|
│ per proxy: skip if no IP/host or deleting (state pruned →
|
|
│ a replaced instance starts with fresh counters)
|
|
│ newState: seed verdict from an existing Healthy condition
|
|
│ (leader handover), jitter first probe across the interval
|
|
│ due && !inFlight ──► jobs ◄── probe worker picks up
|
|
└ prune states for proxies gone from the cache
|
|
│
|
|
probe(ctx, proxyURL, hc, tls) (probe.go)
|
|
│ fresh transport per probe, DisableKeepAlives=true
|
|
│ (load-bearing: keep-alives would cache the CONNECT
|
|
│ tunnel and later probes would never re-exercise it)
|
|
│ https probe URL ⇒ CONNECT through the proxy + TLS inside
|
|
└ success = err == nil AND expected status code
|
|
│
|
|
record(job, result, now) ── under one mutex
|
|
│ counters: consecOK/consecFail; verdict flips only at
|
|
│ successThreshold / failureThreshold
|
|
│ emit ONLY on: first-ever verdict │ threshold flip │
|
|
│ latency Δ > max(20ms, 50% of reported) rate-limited
|
|
│ to one report per MinReportInterval (60s)
|
|
▼
|
|
Events chan (buffered 64, non-blocking send;
|
|
on drop the reported markers do NOT advance → next probe retries)
|
|
│
|
|
▼
|
|
source.Channel → workqueue → Reconcile (see §1)
|
|
│
|
|
▼
|
|
r.applyHealth(p) ── reads Engine.Snapshot(key) (status.go)
|
|
stages the Healthy condition + latencyMillis +
|
|
lastHealthCheckTime; computePhase turns Provisioned=True
|
|
+ Healthy=True/False into phase Ready / Unhealthy
|
|
```
|
|
|
|
Consequence worth knowing: `status.lastHealthCheckTime` is the time of the
|
|
last *status-affecting* probe, not the most recent probe — suppressed
|
|
probes deliberately never write status. True probe recency will live in
|
|
metrics (Step 9).
|
|
|
|
### 7. Discovery + lease API (`internal/discovery/`, `internal/lease/`)
|
|
|
|
HTTP-driven, not cluster-event-driven: crawler clients call in; the only
|
|
Kubernetes interaction is reading Proxies from the manager's cache. The
|
|
server is a non-leader-elected Runnable (all replicas would serve, but the
|
|
deployment ships `replicas: 1` because lease state is per-process — an
|
|
operator restart drops all leases and cooldowns, a documented caveat).
|
|
Client-facing reference with request/response schemas and curl examples:
|
|
[api.md](api.md).
|
|
|
|
```text
|
|
crawler client
|
|
│ Authorization: Bearer $DISCOVERY_TOKEN (empty token = auth disabled, loud startup warning)
|
|
▼
|
|
Server.handler() middleware, outermost first (server.go)
|
|
recover → request-log → MaxBytesReader(64KiB) → bearer auth (constant-time; /healthz exempt)
|
|
│
|
|
├─ GET /healthz ──► 200 ok (unauthenticated)
|
|
│
|
|
├─ GET /v1/proxies?attr.k=v&healthy=true (handlers.go)
|
|
│ Reader.List(Proxies) ── manager cache
|
|
│ filter: attributes equality + Healthy condition
|
|
│ + Store.Counts() for activeLeases
|
|
│ ──► 200 {"proxies":[...], "count":N} (empty list is 200, not 404)
|
|
│
|
|
├─ POST /v1/leases {"selector":{...},"ttlSeconds":300,"target":"..."}
|
|
│ Reader.List → filter selector; unhealthy matches counted, not offered
|
|
│ Store.Acquire(healthy candidates, target, ttl) ── one lock: select+insert
|
|
│ │ selection: fewest active leases, then latency, then name
|
|
│ ├─ granted ──► 201 {leaseID, proxy:{...}, expiresAt, ttlSeconds}
|
|
│ └─ ErrNoMatch ──► 409 {"error":"no_match", considered, atCapacity,
|
|
│ inCooldown, unhealthy}
|
|
│
|
|
├─ DELETE /v1/leases/{id} ──► Store.Release ──► always 204 (idempotent)
|
|
│
|
|
└─ POST /v1/leases/{id}/report {"result":"ok|rate_limited|banned","target":"..."}
|
|
Store.Report ── rate_limited/banned ⇒ cooldown[{proxy,target}] for
|
|
│ CooldownWindow (target falls back: report → lease → global)
|
|
├─ 204 │ 400 invalid_result │ 404 unknown_lease
|
|
└─ an expired lease still resolves for CooldownWindow past its TTL —
|
|
a late report lands exactly when the proxy is being rate-limited
|
|
|
|
Store.Start(ctx) ── manager Runnable, NOT leader-elected: sweeps expired
|
|
leases + cooldowns; correctness never depends on the
|
|
sweep (every read checks ExpiresAt against the clock)
|
|
```
|
|
|
|
### 8. Orphan GC (`internal/gc/`) — the crash-safety net
|
|
|
|
Timer-driven, leader-elected (destructive ⇒ single writer). Exists for the
|
|
one gap the reconciler cannot close alone: a crash after a provider Create
|
|
but before the status write that records the instance.
|
|
|
|
```text
|
|
Sweeper.Start(ctx) ── refuses to run when the cache is namespace-
|
|
│ restricted unless --gc-allow-namespaced is explicit
|
|
│ (an incomplete live set would "orphan" live VMs)
|
|
└─ every Interval (10m; first sweep a full interval after start):
|
|
sweep(ctx)
|
|
│ Reader.List(Proxies) → live UID set
|
|
│ List fails → skip the whole sweep (never guess)
|
|
│ a CR with deletionTimestamp still counts as LIVE — its
|
|
│ finalizer owns that deletion; GC racing it double-deletes
|
|
└ per provider: ListByTag
|
|
│ error → log, continue with the next provider
|
|
└ delete only when ALL hold:
|
|
has the proxy-operator-uid label (ownership proof)
|
|
older than MinAge (10m) (not mid-create)
|
|
UID matches no existing CR (truly orphaned)
|
|
each kill logged loudly with provider, providerID, UID
|
|
```
|
|
|
|
### 9. Metrics (`internal/metrics/`)
|
|
|
|
Registered explicitly from `cmd/main.go` (no `init()`; tests use fresh
|
|
registries). Two kinds:
|
|
|
|
- **Scrape-time collectors** — `proxy_operator_proxies{phase}` and
|
|
`proxy_operator_leases_active` read the cache / lease store at every
|
|
scrape; reconcile-incremented gauges inevitably drift and leak series.
|
|
The provider-check gauges (§11) read the health monitor's snapshot the
|
|
same way.
|
|
- **Fed vectors** — `healthcheck_duration_seconds{proxy}` and
|
|
`healthcheck_failures_total{proxy}` observe EVERY probe (status writes
|
|
are transition-only; metrics carry the high-frequency signal), and the
|
|
health engine deletes a proxy's series when it prunes its state;
|
|
`lease_requests_total{outcome}` from the discovery handlers;
|
|
`provider_requests_total{provider,op,result}` from the
|
|
`provider.WithMetrics` decorator — the one place `Class()` is called
|
|
purely for observability.
|
|
|
|
Each consuming package defines its own small recorder interface
|
|
(`health.ProbeMetrics`, `discovery.LeaseMetrics`, `provider.RequestRecorder`);
|
|
`metrics.Metrics` satisfies all of them structurally, so no package other
|
|
than `cmd/main.go` imports the metrics package.
|
|
|
|
### 10. Tracing (`internal/tracing/`)
|
|
|
|
OpenTelemetry tracing, integrated with — not replacing — the logr/zap
|
|
logging. Everything hangs off standard `OTEL_*` env vars: with no
|
|
`OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_TRACES_EXPORTER` set, no SDK is
|
|
installed, spans are no-ops, and logs are byte-for-byte what they were.
|
|
`OTEL_SDK_DISABLED=true` and `OTEL_TRACES_EXPORTER=none` force it off;
|
|
`console` prints spans to stdout for local dev; sampling follows
|
|
`OTEL_TRACES_SAMPLER(_ARG)`.
|
|
|
|
Span topology — each unit of work is a new trace (watch events carry no
|
|
incoming trace context):
|
|
|
|
- `Reconcile Proxy` (root, via a `reconcile.Reconciler` decorator) →
|
|
`reconcile.managed` / `reconcile.replaceInstance` / `reconcile.delete`,
|
|
plus `status.patch` from the deferred flush. `reconcileExternal` does no
|
|
I/O and is unspanned.
|
|
- `provider.create|get|delete|list` (client spans, `provider.WithTracing`
|
|
decorator wired outermost around `WithMetrics`); the GCP SDK's own
|
|
otelhttp transport contributes HTTP child spans automatically.
|
|
- Kubernetes API calls become client child spans via a wrapped
|
|
`rest.Config` transport — gated on an existing parent span, so informer
|
|
list/watch long-polls and leader-election renewals never create root
|
|
spans. The kube-apiserver ignores incoming `traceparent` by design;
|
|
these spans are leaves.
|
|
- Discovery API: one server span per request (named from the route
|
|
pattern), incoming W3C `traceparent` honored so clients' traces continue
|
|
into the operator; `/healthz` excluded.
|
|
- `gc.sweep` per GC pass; `providerhealth.check` per round of provider
|
|
checks; `health.probe` per probe **only** with
|
|
`--trace-health-probes` (default off — probes run ~1/s per proxy).
|
|
|
|
Log correlation: logr sinks never see a context, so trace IDs ride on the
|
|
logger — `tracing.Start` re-derives the ctx logger from a captured base
|
|
with `traceID`/`spanID` values (lowerCamel, matching `reconcileID`;
|
|
Grafana/Loki derived-field regexes must match `traceID`, not `trace_id`).
|
|
Re-deriving from the base rather than layering keeps zap from emitting
|
|
duplicate keys on nested spans. GCP V(5) wire logs get the same keys from
|
|
the slog handler's ctx.
|
|
|
|
### 11. Provider health checks (`internal/providerhealth/`)
|
|
|
|
Answers "can this cluster reach each provider's backend right now?"
|
|
without waiting for a Proxy to fail. A provider opts in by implementing
|
|
`provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per
|
|
target and never fails as a whole. All three providers do:
|
|
|
|
| Provider | Target | What one check does |
|
|
|---|---|---|
|
|
| kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) |
|
|
| kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on |
|
|
| gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` |
|
|
|
|
```text
|
|
Monitor.Start(ctx) ── leader-only; --provider-check-interval (1m; 0 = off,
|
|
│ otherwise ≥ 10s), --provider-check-timeout (15s, or
|
|
│ the interval if that is shorter)
|
|
├─ once: delete managed-by-labelled ProviderHealth objects whose provider
|
|
│ is not configured (all of them when checks are off) — the providers
|
|
│ config is read once per process
|
|
└─ at start, then every Interval: checkOnce
|
|
│ every provider's Check concurrently, under one context.WithTimeout
|
|
│ snapshot → the scrape-time gauges
|
|
│ log transitions: down or reason changed → "WARNING: provider target
|
|
│ not ready"; back up → "Provider target ready"; steady → silent
|
|
└ per provider: APIReader.Get → Create if missing → status merge patch
|
|
```
|
|
|
|
- **One status write per provider per round**, since `lastCheckTime`
|
|
moves every round. At the 10s minimum interval that is still far below
|
|
leader-election renewals. Proxy status avoids writes like this only
|
|
because its probes run once a second per proxy.
|
|
- **Reads go through the API reader**, not the cache. The objects are
|
|
cluster-scoped; a cached Get would start a cluster-wide informer (and
|
|
need `watch`) even with `--proxy-namespace`.
|
|
- **Checks bypass the provider decorators.** `cmd/main.go` collects the
|
|
Checkers before wrapping the providers in `WithTracing(WithMetrics(...))`,
|
|
so a check never counts as a provider request. KVM RPCs still appear in
|
|
the V(1) `libvirt call` trace, and each round is one
|
|
`providerhealth.check` span.
|
|
- **Not wired to `/readyz`.** A dead hypervisor must not make the manager
|
|
unready while the other providers keep working.
|
|
- **Gauges** `proxy_operator_provider_target_up`,
|
|
`proxy_operator_provider_check_duration_seconds` and
|
|
`proxy_operator_provider_check_timestamp_seconds`, each labelled
|
|
`{provider,target}`. Only the leader exports them.
|
|
|
|
Illustrative output:
|
|
|
|
```sh
|
|
$ kubectl get providerhealth # short name: ph
|
|
NAME TYPE READY TARGETS LAST CHECK AGE
|
|
kubernetes kubernetes True 1/1 12s 3d
|
|
kvm-home kvm False 0/1 12s 3d
|
|
$ kubectl get ph kvm-home -o yaml # per target: reason, message, details
|
|
```
|
|
|
|
## Decisions
|
|
|
|
Judgment calls the spec left open, and deliberate deviations — recorded so
|
|
they read as choices, not accidents. Chronological by build step.
|
|
|
|
- **Registry takes its constructor map as a parameter** instead of holding
|
|
a package-level map: avoids the provider⇄registry import cycle and puts
|
|
the wiring at the composition root, where it is visible.
|
|
- **Mock provider replaced by the kubernetes-pod provider** (user
|
|
decision, mid-build): a simulated in-memory provider was too far from
|
|
the real system to build confidence in. Local dev/CI now runs real
|
|
`ubuntu/squid` pods (Canonical's actively maintained image, verified
|
|
50M+ pulls, pinned tag) in the operator's own cluster. Trade-off
|
|
accepted: envtest has no kubelet, so end-to-end proof lives in the kind
|
|
quickstart, and cluster pods share one egress IP — distinct egress
|
|
paths remain the GCP provider's job.
|
|
- **`RequeueAfter: RequeueNow` instead of the plan's `Requeue: true`:**
|
|
`ctrl.Result.Requeue` is deprecated in controller-runtime v0.24; a fifth
|
|
configurable interval (default 1s) keeps identical semantics and stays
|
|
shrinkable in tests.
|
|
- **Quota exhaustion is a wait, not a failure:** `ErrQuotaExceeded` sets a
|
|
condition and requeues slowly (5m) with a nil error — off the backoff
|
|
curve, out of the error log, and never `phase: Failed`. Only
|
|
`ErrPermanent` latches Failed, keyed to the generation so a spec edit
|
|
auto-recovers.
|
|
- **The finalizer path never latches permanent failures:** a permanent
|
|
error during deletion keeps retrying visibly instead — latching there
|
|
would wedge the object forever with no path out but manual finalizer
|
|
surgery.
|
|
- **Health transitions travel reconciler-ward over a channel**
|
|
(`source.Channel`), not direct status patches: `phase` derives from both
|
|
provisioning and health, so two status writers would race and flap. One
|
|
writer of status; the engine owns health *state*, the reconciler its
|
|
*representation*; write-only-on-transition falls out for free.
|
|
- **Health state seeds from the existing Healthy condition on leader
|
|
handover** (verdict kept, counters zeroed, first probe jittered), so a
|
|
healthy fleet doesn't flap to Unknown on restart — but a real
|
|
transition still needs a full threshold run. A never-probed proxy skips
|
|
the jitter and probes on the next tick: startup spread matters for
|
|
restarts, not for a single new proxy.
|
|
- **Latency suppression is `max(20ms, 50%)` + a 60s rate limit, and only
|
|
while the verdict is healthy.** The spec's bare ">50% change" is
|
|
undefined at 0 and lets a proxy jittering 40↔61ms write status forever;
|
|
the healthy-only guard (found by test) stops a below-threshold success
|
|
streak from emitting latency updates for a proxy still reported
|
|
unhealthy. Consequence: `status.lastHealthCheckTime` means "last
|
|
status-affecting probe" — true probe recency is in the metrics.
|
|
- **Deterministic instance names** are `proxy-` + 16 chars of
|
|
base32(SHA-256(CR UID)): legal for both GCP (`[a-z2-7]` ⊂ `[-a-z0-9]`,
|
|
22 ≤ 63 chars) and Pod names, 80 bits against birthday collisions at a
|
|
fleet of tens. The replacement VM therefore has the *same name* as the
|
|
one being deleted — which is why replacement polls to NotFound before
|
|
recreating instead of racing a 409.
|
|
- **`banned` and `rate_limited` share one cooldown window:** a second
|
|
duration knob the spec doesn't ask for; the report's semantic
|
|
difference is preserved in the API but not the store.
|
|
- **Report targets fall back report → lease → global**, so a client that
|
|
leased with a target can't accidentally poison the proxy's global pool
|
|
by omitting the target in its report.
|
|
- **The 409 body's `considered` counts unhealthy matches too** (the store
|
|
only ever sees healthy candidates): `considered = atCapacity +
|
|
inCooldown + unhealthy + eligible-but-outranked`, keeping the numbers
|
|
additive for a human debugging "why no proxy?".
|
|
- **TTLs above `--max-lease-ttl` are a 400, not a silent clamp** — a
|
|
client asking for a week should find out.
|
|
- **Discovery is not leader-elected and ships `replicas: 1`:** caches
|
|
start before non-leader-election runnables (verified in
|
|
controller-runtime's ordering), and a leader-elected server would leave
|
|
non-leader replicas as broken Service endpoints. One replica because
|
|
lease state is per-process.
|
|
- **GCP `Create` requires zone, machineType, and image** and fails
|
|
`ErrPermanent` naming the missing field — inventing machine-type
|
|
defaults would silently create billable VMs of arbitrary shape.
|
|
- **Unknown GCP instance statuses map to `Stopped`:** the reconciler's
|
|
answer to Stopped is delete-and-recreate, the always-safe move for
|
|
cattle when the API grows a new state.
|
|
- **KVM hosts are reached over SSH with the libvirt socket forwarded,**
|
|
not libvirtd's TCP/TLS listener: nothing to configure or expose on the
|
|
hypervisors beyond sshd, pure Go on the operator side (no cgo, no
|
|
libvirt libraries in the image), and the hosts list can grow by a
|
|
config edit because the providerID carries the host name.
|
|
- **KVM base images are fetched by the operator into the host's pool**
|
|
through libvirt's upload stream, rather than expected to be pre-placed:
|
|
the operator then needs no filesystem access on the host and a new
|
|
host is usable with zero manual staging. The cost is a synchronous
|
|
first-time download inside Create.
|
|
- **The KVM seed ISO is rebuilt on every Create attempt** before the
|
|
domain exists, instead of "exists → skip": a reconcile cancelled mid
|
|
upload leaves a truncated seed, and the compensating delete cannot run
|
|
because the connection was already closed to unblock the RPC. Skipping
|
|
it would boot a VM with no cloud-init and no retry would ever repair it.
|
|
- **KVM `nostate` maps to `Provisioning`, not `Stopped`:** unlike GCP's
|
|
unknown statuses, `nostate` means libvirt could not determine the
|
|
state at all; answering that with delete-and-recreate is too drastic,
|
|
so the next poll decides.
|
|
- **KVM guests boot UEFI by default, secure boot off.** Found on the
|
|
first real host: Debian 13's generic image hangs in GRUB under SeaBIOS
|
|
on a headless VM (Debian bug #1111240: `terminal_output gfxterm
|
|
serial` with a serial console and no video device) regardless of CPU
|
|
model or seed, and boots under OVMF with the serial console working.
|
|
The alternatives were a virtio console (loses the kernel's serial
|
|
output) or a video device the VMs have no use for. Every
|
|
current distro cloud image supports UEFI, so it is the default and
|
|
`firmware: bios` is the per-image escape; libvirt's firmware
|
|
autoselection (`<os firmware="efi">`) keeps the OVMF path out of the
|
|
operator's config. Secure boot is off so unsigned images boot too.
|
|
- **A KVM domain GC cannot read is skipped, a host it cannot reach fails
|
|
the sweep.** GC never deletes what it does not see, so leaving one
|
|
domain out is safe; one corrupt domain blocking GC for every other one
|
|
is not. A whole host missing from the view would hide many, so that
|
|
still aborts.
|
|
- **Kubernetes 403s classify as `ErrPermanent`** even though quota
|
|
exhaustion also surfaces as 403 (indistinguishable from RBAC denial in
|
|
`apierrors`): not hammering an API server that may never allow the
|
|
request is the safer default; a real ResourceQuota 403 forgoes the
|
|
gentler quota backoff. Documented at the classification site.
|
|
- **GC kills log at Info with a `WARNING:` prefix** — logr has no Warn
|
|
level; the plan's "log at Warn" is met in spirit with provider,
|
|
providerID, and UID always attached. Same convention as the
|
|
discovery server's empty-token warning.
|
|
- **GC trusts only provable orphans:** instances without the UID label
|
|
are never deleted, a CR with a deletionTimestamp still counts as live
|
|
(its finalizer owns that deletion), and an unreadable Proxy list skips
|
|
the whole sweep. The namespace guard refuses to sweep a
|
|
namespace-restricted cache without `--gc-allow-namespaced`.
|
|
- **Cloud-init Secrets must carry `crawl.example.com/cloud-init: "true"`:**
|
|
the manager caches only labelled Secrets (the operator holds
|
|
cluster-wide Secret read RBAC — an unrestricted cache would hold every
|
|
Secret in scope). Unlabelled referenced Secrets are invisible by
|
|
construction, surfacing as `CloudInitError`.
|
|
- **Deletion never gives up on its own, but can be released by hand:** a
|
|
permanent cleanup failure keeps retrying with the finalizer on, because
|
|
dropping it automatically would leak a running instance on any provider
|
|
that answers "permission denied". Where `Get` fails permanently the
|
|
reconciler still calls `Delete`, since it is idempotent and may finish
|
|
where `Get` cannot converge. `crawl.example.com/skip-cleanup: "true"`
|
|
abandons the instance and releases the object; it is checked before the
|
|
provider lookup, so it also frees a Proxy whose provider is gone from the
|
|
config.
|
|
- **Events RBAC from the plan is omitted:** nothing wires an
|
|
EventRecorder in the prototype, and granting verbs nothing uses would
|
|
be RBAC lint noise. Add the marker together with the recorder if events
|
|
land later.
|
|
- **logr, not slog, inside controller paths:** the repo convention says
|
|
`slog`, but `log.FromContext(ctx)` hands controller-runtime's logr
|
|
logger to everything running under the manager — fighting that would
|
|
mean two logging systems in one process. Noted as a deviation rather
|
|
than silently ignored.
|
|
- **Tracing is env-gated, not flag-gated:** it activates only when
|
|
`OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_TRACES_EXPORTER` is set (user
|
|
decision). No collector configured means no SDK installed, no-op spans,
|
|
unchanged logs — the safe default for every existing deployment. The
|
|
one flag is `--trace-health-probes`, off by default, because probes at
|
|
~1/s per proxy would dominate trace volume.
|
|
- **Trace keys are `traceID`/`spanID`**, lowerCamel like `reconcileID` and
|
|
`providerID`, deliberately not the `trace_id` many Grafana derived-field
|
|
examples assume — configure the derived-field regex accordingly. logr
|
|
sinks can't read ctx, so the IDs ride on the ctx logger, re-derived from
|
|
a captured base per span so zap never emits duplicate keys.
|
|
- **Probe→reconcile trace links are not attempted:** the health engine's
|
|
`GenericEvent` carries only namespace/name (no ctx), and the workqueue
|
|
coalesces events, so any link would be a guess. A health-triggered
|
|
reconcile starts a fresh trace; the probe that caused it is findable via
|
|
its own (opt-in) span and shared proxy attributes.
|
|
- **No `traceparent` toward probe targets:** probe transports stay
|
|
uninstrumented so trace headers can never leak through a proxy to
|
|
external sites. The kube-apiserver ignores incoming `traceparent` by
|
|
design (public endpoint), so k8s client spans are leaves — in-process
|
|
traces, not cross-process ones.
|
|
- **Provider health is a status-only, cluster-scoped CRD** (user
|
|
decision). `kubectl get providerhealth` works without Prometheus, which
|
|
the deployment does not ship (its ServiceMonitor is commented out). There
|
|
is no spec: the providers config file stays the one place providers are
|
|
configured, and the object only reports on it. The gauges carry the same
|
|
verdicts for alerting.
|
|
- **KVM checks reuse the pooled connection instead of dialing fresh.** A
|
|
fresh dial would not pick up a rotated Secret anyway, since the SSH
|
|
material is cached. It would also open an sshd session per host per
|
|
minute. And a working cached connection is exactly what reconciles use.
|
|
Sharing must not cost the reconciles anything, so the check calls
|
|
through `hostPool.probe`. A check that runs out of time abandons its
|
|
RPCs instead of closing the connection, which would kill an upload or a
|
|
Create running on it. At most one abandoned check per host is left
|
|
waiting, and until it returns, the next check reports `Timeout` at once.
|
|
Waiting for a host's connection slot also ends with the caller's
|
|
deadline, so a reconcile redialing a dead host cannot stall the round.
|
|
- **Provider transitions are logs, not Events**, in line with the
|
|
EventRecorder decision above. They log at Info with a `WARNING:` prefix,
|
|
and each target's `lastTransitionTime` in the object records when it
|
|
flipped.
|
|
- **A provider name that is not a valid object name** gets metrics but no
|
|
`ProviderHealth` object, with a warning at start. Tightening config
|
|
validation instead would crash-loop existing deployments on upgrade.
|