kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

proxy-operator: Kubernetes operator for crawling-proxy fleets #1

Merged
kacerr merged 34 commits from feat/proxy-operator into main 2026-08-11 19:21:14 +02:00
Conversation 0 Commits 34 Files Changed 95 +12651 -749
kacerr commented 2026-08-10 09:37:05 +02:00
Owner
Copy Link

Implements the full proxy-operator prototype per docs/plans/2026-08-07-1747-proxy-operator.md (execution log with per-step details: docs/plans-executions/2026-08-07-1747-proxy-operator.md).

What's in here:

  • Proxy CRD (crawl.example.com/v1alpha1) with CEL cross-field validation and full structural defaulting
  • Crash-safe reconciler state machine: immutable-cattle replacement via spec-hash annotation, finalizer cleanup, permanent-failure latch, quota-aware backoff
  • Two providers behind one minimal interface: kubernetes (real Squid pods, local dev/CI) and gcp (Compute Engine, four API calls, fire-and-forget operations)
  • Health engine probing through the proxy (real CONNECT), threshold logic, transition-only status writes via source.Channel
  • Discovery/lease HTTP API on :8090: list/lease/release/report with per-target cooldowns; in-memory lease store behind a swappable interface
  • Orphan GC sweeper + Prometheus metrics (scrape-time collectors, explicit registration)
  • Manifests, samples, kind quickstart README, architecture doc with a 20+-entry decision log

Verification: make test green with -race across 11 packages (75–100% coverage on the internal ones); full end-to-end pass on a throwaway kind cluster — Squid pod Ready via a real CONNECT probe, lease grant/report/cooldown-409/release through the API, finalizer cleanup. Two things the kind run caught and fixed: squid FD-table OOM under containerd's unlimited RLIMIT_NOFILE, and the quickstart moving in-cluster because host-side probes can't reach kind pod IPs.

Note: main was pushed at the scaffold baseline commit (the repo had no remote branches before this MR), so the diff here is the entire feature on top of the untouched kubebuilder scaffold.

🤖 Generated with Claude Code

Implements the full proxy-operator prototype per docs/plans/2026-08-07-1747-proxy-operator.md (execution log with per-step details: docs/plans-executions/2026-08-07-1747-proxy-operator.md). **What's in here:** - `Proxy` CRD (crawl.example.com/v1alpha1) with CEL cross-field validation and full structural defaulting - Crash-safe reconciler state machine: immutable-cattle replacement via spec-hash annotation, finalizer cleanup, permanent-failure latch, quota-aware backoff - Two providers behind one minimal interface: kubernetes (real Squid pods, local dev/CI) and gcp (Compute Engine, four API calls, fire-and-forget operations) - Health engine probing through the proxy (real CONNECT), threshold logic, transition-only status writes via source.Channel - Discovery/lease HTTP API on :8090: list/lease/release/report with per-target cooldowns; in-memory lease store behind a swappable interface - Orphan GC sweeper + Prometheus metrics (scrape-time collectors, explicit registration) - Manifests, samples, kind quickstart README, architecture doc with a 20+-entry decision log **Verification:** `make test` green with -race across 11 packages (75–100% coverage on the internal ones); full end-to-end pass on a throwaway kind cluster — Squid pod Ready via a real CONNECT probe, lease grant/report/cooldown-409/release through the API, finalizer cleanup. Two things the kind run caught and fixed: squid FD-table OOM under containerd's unlimited RLIMIT_NOFILE, and the quickstart moving in-cluster because host-side probes can't reach kind pod IPs. Note: `main` was pushed at the scaffold baseline commit (the repo had no remote branches before this MR), so the diff here is the entire feature on top of the untouched kubebuilder scaffold. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kacerr added 24 commits 2026-08-10 09:37:05 +02:00
Codify plan-save ordering and add plan execution summaries a5f0aa95d2 
The plan for this build was written to docs/plans/ only after Step 0's
implementation had already started, instead of as its own first action
right after ExitPlanMode. CLAUDE.md said *where* to save plans but not
*when* relative to other work, so that ordering wasn't actually enforced.
Makes it explicit: copying the plan into docs/plans/ and committing it is
its own checkpoint that blocks starting Step 0.

Also adds a docs/plans-executions/ convention: one running, chronological
summary file per plan, appended to after each completed step, for a
human-readable narrative of how the plan actually went (judgment calls,
spec gaps, surprises) without duplicating the plan or the diff.

Backfills the Step 0 entry for the proxy-operator plan retroactively.

Co-Authored-By: Claude <noreply@anthropic.com>
Require exact command snippets in plan execution summaries e614264597 
The Step 0 entry described what happened in prose but omitted the actual
kubebuilder/go install invocations, which is exactly the detail a future
reader would want to copy and reproduce. Codifies "include the real
command line, not a paraphrase" in CLAUDE.md and rewrites the Step 0
entry with the exact commands run, including the go install path
correction (kubebuilder v4.15.0 is the module root now, not
.../cmd/kubebuilder) and the controller-gen invocations kubebuilder ran
on its own.

Co-Authored-By: Claude <noreply@anthropic.com>
Pin down structure and depth for plan execution summaries 652f155baf 
Codifies the pattern the rewritten Step 0 entry demonstrated: one line of
context before each command snippet explaining why it was run that way,
explicit call-outs when something deviated from the plan rather than
presenting the working command as the first thing tried, and a closing
"worth noting" paragraph. Points at the Step 0 entry itself as the
reference example so future steps match its depth without re-deriving it.

Co-Authored-By: Claude <noreply@anthropic.com>
Add a status checklist as the authoritative plan-progress record 26991fbe13 
Completion status previously lived only in my in-session TodoWrite list
(ephemeral, doesn't survive the session) or had to be inferred from which
steps had an entry in the execution log. Neither is a durable, explicit
answer to "what's left."

Adds a markdown checklist at the top of each plans-executions file, one
line per plan step, checked in the same commit as that step's entry. A
new session resuming this plan can read the file alone and know exactly
where to pick up.

Co-Authored-By: Claude <noreply@anthropic.com>
Add Proxy API types with CEL validation (Step 1) f28766fce3 
Full ProxySpec/ProxyStatus/Proxy types per the plan: PlacementSpec,
CloudInitSpec, EndpointSpec, HealthCheckSpec, SecretKeySelector, all
defaults, and 7 CEL XValidation rules enforcing mode/provider
immutability, provider/endpoint required-iff-Managed/External, and
cloud-init exactly-one-of inline/secretRef.

Applies the four corrections identified during planning that would
otherwise be silent bugs: MaxLeases as *int32 (so an explicit 0 survives
Go round-trips instead of re-defaulting to 5), HealthCheck's
default={} marker (so nested defaults apply even when the field is
omitted entirely), MinLength=1 on Provider/CloudInit.Inline (so the CEL
has() checks stay simple), and listType=map on Conditions.

Adds pure helpers (EffectivePort, EffectiveHost, HealthCheckOrDefault,
MaxLeasesOrDefault) with table-driven tests, for use by the health
engine, discovery API, and spec-hash computation in later steps.

Patches the scaffolded placeholder controller test's resource literal to
a schema-valid spec so it survives the new CRD validation — the test
itself is rewritten wholesale in Step 4 alongside the real reconciler.

Regenerated deepcopy and the CRD; make test green (envtest confirmed all
7 CEL rules enforced by a real apiserver).

Co-Authored-By: Claude <noreply@anthropic.com>
Add the provider contract, error taxonomy, naming, and config (Step 2) a4a483acbc 
The Provider interface (Create/Get/Delete/ListByTag), Instance, and
CreateRequest that every cloud backend implements — kept independent of
api/v1alpha1 so this package has no CRD-type coupling.

Error taxonomy (ErrNotFound/ErrQuotaExceeded/ErrTransient/ErrPermanent)
wrapped via a multi-error Unwrap() []error, so errors.Is and errors.As
both work off the same value: the reconciler branches on classification,
logs keep the underlying SDK error. Unclassified errors default to
ErrTransient — retrying is always safer than latching Failed.

Deterministic instance naming (SHA-256 -> base32 -> 16 chars, 22 total
with the "proxy-" prefix) satisfying GCP's RFC1035 name rules with
headroom, and idempotency-tested across 10k UIDs with zero collisions.

--providers-config YAML parsing (config.go) with fail-fast validation:
unknown type, duplicate name, missing gcp.project, mismatched
type/config-block, and strict-mode rejection of unknown keys.

internal/provider/registry/registry.go takes its type->constructor map
as a parameter rather than hardcoding it, so the package has zero import
on internal/provider/mock or internal/provider/gcp (neither exists yet —
mock is Step 3, gcp is Step 8) and compiles today. Explicit wiring moves
to the composition root in cmd/main.go (Step 10).

Deferred internal/provider/metrics.go (the WithMetrics decorator) to
Step 9, where the Prometheus vectors it needs actually get built —
nothing in this step depends on it.

internal/provider at 96.2% coverage, internal/provider/registry at 100%.
make test green.

Co-Authored-By: Claude <noreply@anthropic.com>
Add the mock provider with a real CONNECT proxy per port (Step 3) ef1387dc01 
An in-memory provider.Provider whose state (Provisioning -> Running ->
Terminated -> purged) is a pure function of an injectable clock, not
background timers, so it's deterministic under tests and correct under
real time with no goroutine lifecycle to leak.

Once an instance is observed Running, it lazily acquires a real HTTP
CONNECT proxy listener so the health engine's through-the-proxy probe
(later steps) genuinely tunnels a request end to end, instead of the
healthcheck being simulated or bypassed for local development.

Redesigned the listener sharing model from what the plan assumed: the
plan's "one loopback IP per instance" doesn't work on macOS (only
127.0.0.1 binds without a privileged ifconfig alias, unlike Linux where
the whole 127.0.0.0/8 routes to loopback by default), and there's no
channel for a provider to report a port back to the reconciler anyway
(EffectivePort() is spec-only). Instances now share one real listener
per port, reference-counted at the package level rather than per
Provider instance, since a bound TCP port is a genuinely process-global
OS resource -- two separately configured mock-typed provider entries
must not both try to bind the same default port.

Fault injection wired both ways: MockConfig.FailNextCreates/FailWith for
demos, InjectCreateFailures(n, class) for tests. Create is idempotent by
name.

Caught and fixed a real test flake (not a logic bug): the freePort test
helper asked the OS for a free port via bind-then-close, a TOCTOU race
under t.Parallel() that let two tests collide on the same "free" port.
Replaced it with a monotonic counter, since these tests only need
uniqueness within the test run.

internal/provider/mock at 91.1% coverage, including an end-to-end test
that opens real sockets: Create -> Get past provisionDelay -> a real
http.Client tunnelling a CONNECT through the mock to a real TLS origin.
make test green across the whole repo.

Co-Authored-By: Claude <noreply@anthropic.com>
Remove the in-memory mock provider 4529594fb7 
The mock provider (state simulated via an injectable clock, a hand-rolled
shared/refcounted CONNECT-proxy listener per port to work around macOS's
loopback restrictions) worked, but the user felt it was too far removed
from the real system to build confidence in, and doesn't need the
automated test suite to stay fast enough to justify that complexity — a
kind-based verification pass "once in a while" is an acceptable trade for
tests that actually look like the final product.

Replacing it with a provider that creates real Pods in the same cluster,
running an actual Squid container. internal/provider/registry was already
designed to have zero dependency on any concrete provider package, so
removing this one required no changes anywhere else in the tree — go
build is clean with nothing implementing provider.Provider yet.

docs/plans/2026-08-07-1747-proxy-operator.md's Step 3 (and every other
reference to the mock provider throughout the plan) is updated in this
same commit to describe the replacement. Narrative on why and the
replacement's design lands in docs/plans-executions once it's built.

Co-Authored-By: Claude <noreply@anthropic.com>
Add the Kubernetes pod provider (replaces the removed mock) ff859ebd84 
Create/Get/Delete/ListByTag against real corev1.Pod objects in the same
cluster the operator runs in, running an ubuntu/squid container -- picked
by actually checking Docker Hub metadata (Canonical-published, rebuilt
the same day this was decided, 50M+ pulls) rather than guessing an image
reference. It's a public image, so kind nodes pull it directly with no
build/load step.

providerID is "<namespace>/<podName>", parsed via
cache.SplitMetaNamespaceKey -- the same self-contained-providerID
reasoning the plan already calls for on the GCP provider's zone-qualified
IDs. Pod state maps to InstanceState with Succeeded/Failed/Unknown all
collapsing to Terminated, since the reconciler already treats Stopped and
Terminated identically; Running-without-PodIP maps to Provisioning so an
empty IP is never published.

The client is built internally via ctrl.GetConfig() (in-cluster or local
kubeconfig, whichever applies), not threaded through the registry
Constructor signature -- this is what lets `make run` against a local
kind cluster and running in-cluster share the exact same code path with
no provider-specific wiring in cmd/main.go. New() is deliberately
untested (0% coverage): it's the one function that must never run under
`go test`, since it would happily connect to whatever cluster the
developer's kubeconfig points at. Tests construct Provider via an
unexported newWithClient(client, cfg) instead.

ListByTag lists Pods across every namespace (orphan GC needs to find
every tagged Pod regardless of where it landed), which means this
provider's RBAC has to be a ClusterRole rather than namespace-scoped --
flagged now, wired in Step 10.

provider.Config gains KubernetesConfig (replacing MockConfig) and drops
the FailWith*/fault-injection surface entirely, since that need is now
served by a small in-test stub Provider for reconciler tests (Step 4),
not a config-driven mechanism on a real provider package.

Tests use sigs.k8s.io/controller-runtime/pkg/client/fake -- real Pod
objects, the real client.Client interface -- at 77.6% coverage.
make test green across the whole repo.

Co-Authored-By: Claude <noreply@anthropic.com>
Remove unused cert-manager/webhook scaffolding 7700358764 
kubebuilder's generic scaffold defensively wires up webhook TLS-cert
machinery and an unconditional cert-manager install in the e2e suite, in
case a project grows admission webhooks later. This one never will --
the spec's non-goals explicitly rule out admission webhooks and
cert-manager wiring -- so none of it does anything. Verified before
removing: no config/webhook/, no +kubebuilder:webhook markers anywhere,
and config/*/kustomization.yaml's [CERTMANAGER] blocks are all inert
(never uncommented).

cmd/main.go: drops the webhook import, the three webhook-cert-* flags,
and the WebhookServer wiring on ctrl.Options -- the manager now runs
with no webhook server, correctly, since nothing registers one. Left
the metrics-cert flags alone; those are unrelated to webhooks.

test/e2e/e2e_suite_test.go: drops the unconditional cert-manager
install/uninstall around the suite.

test/utils/utils.go: drops the now-dead InstallCertManager/
UninstallCertManager/IsCertManagerCRDsInstalled and their warnError
helper, plus UncommentCode -- unrelated to cert-manager, but found to
have zero callers even before this cleanup.

Left the inert commented-out [WEBHOOK]/[CERTMANAGER] kustomize blocks
and kubebuilder's scaffold marker comments alone: pure comments, no
runtime behavior, unlike the cert-manager install this actually removed.

Verified clean with both build tags (go build/vet, and -tags=e2e for
test/e2e). make test unchanged and green.

Co-Authored-By: Claude <noreply@anthropic.com>
Add plan: lean-down cleanup of non-goal scaffold 05d490c0b8 
Approved plan for stripping the remaining webhook-only scaffold remnants
and config/network-policy/ from the application footprint, with explicit
keep decisions for prometheus/monitoring manifests, the paired
metrics-TLS plumbing, all RBAC manifests, and all developer tooling.
Also records why the webhook machinery existed at all (kubebuilder init
emits it unconditionally; verified no init flag can suppress it) and the
Step 0 process gap that let it survive until now.

Co-Authored-By: Claude <noreply@anthropic.com>
Strip remaining webhook-only scaffold and network-policy manifests 4282d73c74 
Follows the approved lean-down plan
(docs/plans/2026-08-08-1335-lean-scaffold-cleanup.md). Removes from the
application's deployed footprint:

- config/network-policy/ and its commented enable line -- the user does
  not need network policies at the moment.
- The webhook-only halves of config/default/kustomization.yaml: the
  commented ../webhook and ../certmanager resource lines, the
  manager_webhook_patch.yaml reference, the serving-cert ->
  Validating/Mutating WebhookConfiguration cainjection replacement
  blocks, and the crdkustomizecainjection* scaffold markers -- anchors
  only for `kubebuilder create webhook`, which is a permanent non-goal.
- The two commented [WEBHOOK] blocks in config/crd/kustomization.yaml
  plus the now-empty patches: key; kept the one-line
  crdkustomizeresource marker since `kubebuilder create api` could
  legitimately run again.
- config/crd/kustomizeconfig.yaml, whose only consumer was the removed
  configurations: block.

Explicitly kept per user direction: all of config/prometheus/, the
paired metrics-TLS-via-cert-manager plumbing (cert_metrics_manager_patch
+ the metrics-certs/ServiceMonitor replacement halves), all RBAC
manifests including the admin/editor/viewer helper roles, and all
developer tooling.

Also records in the execution log why the webhook machinery existed at
all: kubebuilder init emits it unconditionally, verified against the
v4.15.0 binary that no init flag can suppress it -- scaffold-then-prune
is the only supported path, and the pruning pass should have happened
at Step 0.

Verified: kustomize build clean on config/default and config/crd,
go build/vet clean with and without -tags=e2e, make test green with
coverage identical to pre-cleanup.

Co-Authored-By: Claude <noreply@anthropic.com>
Pin the manager image stanza; fix stale checklist line 5c408cc284 
config/manager/kustomization.yaml: commit the images: stanza that
`kustomize edit set image` (run by make deploy, including inside make
test-e2e) writes into this tracked file. It showed up as unexplained
drift twice; committing it once ends that -- the edit is idempotent, so
future deploy/e2e runs produce no diff. The example.com image name is
the e2e suite's placeholder default and gets overridden by IMG= on any
real deploy.

Execution log: the Status checklist's Step 3 line still said "Mock
provider" from before the pivot; a fresh session resuming from the
checklist alone would have been misled.

Co-Authored-By: Claude <noreply@anthropic.com>
Add the Proxy reconciler state machine with action-table, phase, and envtest suites 1125f74221 
Co-Authored-By: Claude <noreply@anthropic.com>
Seed docs/architecture.md with the event-to-function reconcile flow diagrams 71c00c40d1 
Co-Authored-By: Claude <noreply@anthropic.com>
Add the health engine: through-proxy probes, thresholds, channel-fed transitions 801a9fbe5f 
Co-Authored-By: Claude <noreply@anthropic.com>
Add the in-memory lease store: least-loaded selection, cooldowns, TTL retention 223b6a8fd6 
Co-Authored-By: Claude <noreply@anthropic.com>
Clarify plan: ServeMux patterns are a Go-1.22-era stdlib feature, project stays on Go 1.26 f6d50e4744 
Co-Authored-By: Claude <noreply@anthropic.com>
Add the discovery HTTP API: list, lease, release, report over the manager cache 4aa3d47e3c 
Co-Authored-By: Claude <noreply@anthropic.com>
Add the GCP provider: four-call surface, fire-and-forget ops, zone-qualified IDs 8176a5eef8 
Co-Authored-By: Claude <noreply@anthropic.com>
Add orphan GC sweeper and Prometheus metrics with explicit registration add120c033 
Co-Authored-By: Claude <noreply@anthropic.com>
Wire the composition root: flags, providers, runnables, manifests, samples, docs c489832ce7 
Co-Authored-By: Claude <noreply@anthropic.com>
Close Step 11 test gaps: CEL envtest cases, defaults assertion, quota/permanent/adopt flows; make test runs -race d595a93d36 
Co-Authored-By: Claude <noreply@anthropic.com>
Verify end-to-end on kind: fix Squid FD-table OOM, make the quickstart in-cluster 0fe62ef314 
Co-Authored-By: Claude <noreply@anthropic.com>
kacerr added 1 commit 2026-08-10 09:44:28 +02:00
Add docs/testing.md: suite inventory, run instructions, deliberate gaps, kind e2e c108a06a94 
Co-Authored-By: Claude <noreply@anthropic.com>
kacerr added 2 commits 2026-08-11 17:53:28 +02:00
Add plan: verbose V-level logging in the GCP provider c137028364 
Co-Authored-By: Claude <noreply@anthropic.com>
Add V(1)/V(2) verbose logging to the GCP provider 837e374228 
One V(1) line per GCP API call (insert/get/delete/aggregatedList) with
outcome and operation name, V(2) request/per-instance detail, and raw
googleapi status+reasons logged before classify collapses them. Curated
fields only — cloud-init user-data never reaches logs (test-enforced).

Co-Authored-By: Claude <noreply@anthropic.com>
kacerr added 3 commits 2026-08-11 18:12:45 +02:00
Add plan: bake git commit into the operator binary and log it at startup e4d2a191d0 
Co-Authored-By: Claude <noreply@anthropic.com>
Bake git commit into the binary and log it at startup ae434a7167 
New internal/version package: ldflags-stamped Commit with a
debug.ReadBuildInfo VCS fallback for host builds. Startup log line
carries commit + Go version; --version prints the hash and exits.
Makefile computes GIT_COMMIT (12 chars, -dirty on any local change) and
passes it to docker-build/buildx; Dockerfile injects it via -ldflags and
an org.opencontainers.image.revision label. make build now uses ./cmd —
file-argument builds skip Go's automatic VCS stamp.

Co-Authored-By: Claude <noreply@anthropic.com>
Record completed Docker verification of the version stamp 5a7f0a30c3 
Co-Authored-By: Claude <noreply@anthropic.com>
kacerr added 2 commits 2026-08-11 18:41:03 +02:00
Add plan: GCP HTTP wire logging at V(5) 4619c352c0 
Co-Authored-By: Claude <noreply@anthropic.com>
Surface GCP SDK HTTP wire logs at V(5) ed59a4c384 
Inject an option.WithLogger slog logger bridged to the zap sink via
logr.ToSlogHandler with a V(1) shift, so the SDK's Debug-level
"api request"/"api response" records (URL, headers, full payloads)
appear only at --zap-log-level=5. Startup warning when active, since
raw insert payloads include cloud-init user-data. Note WithLogger
overrides GOOGLE_SDK_GO_LOGGING_LEVEL for this client.

Co-Authored-By: Claude <noreply@anthropic.com>
kacerr added 1 commit 2026-08-11 19:06:59 +02:00
Wire logging: drop auth token-exchange records, elide huge payload fields 420c3509b0 
The option.WithLogger logger also reaches cloud.google.com/go/auth,
which logged its token exchange at Debug — JWT assertion and bearer
token included. wireLogger now allowlists only the compute client's
api request/response records at Debug (fail-closed for future SDK
additions); Warn/Error pass through. String fields over 1KiB (e.g.
Shielded-VM UEFI dbx blobs) are elided recursively by default; the new
--gcp-wire-log-full-payloads flag restores verbatim payloads.

Co-Authored-By: Claude <noreply@anthropic.com>
kacerr added 1 commit 2026-08-11 19:18:52 +02:00
Add GCP deployment docs, PR review notes, and Claude tooling updates 19d6a8dfba 
docs/gcp-in-specific-project.md: SA + firewall setup for the egress-proxy
project, in-kube secret, and apply-ready ConfigMap/Deployment/Proxy
manifests (Ubuntu image — debian-cloud lacks cloud-init).
docs/gcp-vm-validation.md: end-to-end GCP VM validation walkthrough.
docs/reviews/: proxy-operator PR review notes from 2026-08-10.
.claude/: operator-reviewer agent, accumulated permission allowlist.
.gitignore: never commit sa_key.json (live SA key stays untracked).

Co-Authored-By: Claude <noreply@anthropic.com>
kacerr merged commit f7000f7514 into main 2026-08-11 19:21:14 +02:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
kacerr
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kacerr/egress-proxies-operator#1
Delete Branch "feat/proxy-operator"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?