egress-proxies-operator

A Kubernetes operator that manages a fleet of HTTP egress proxies for crawling: each proxy is a Proxy custom resource that the operator provisions (or merely tracks), actively health-checks through the proxy itself, and hands out to crawler clients via an HTTP list/lease API.

Architecture in 60 seconds

• Proxy CRD (crawl.example.com/v1alpha1, namespaced, kubectl get px): Managed proxies are provisioned by a configured provider; External proxies exist elsewhere and are only tracked and health-checked.
• Reconciler — a crash-safe state machine: every reconcile derives one action from (spec, status, provider Get). Proxies are immutable cattle: any meaningful spec change (placement, cloud-init, port) deletes and recreates the VM — never in-place mutation.
• Providers behind one minimal interface: kubernetes (a real Squid pod in this cluster — local dev/CI), gcp (Compute Engine VMs with ephemeral external IPs — the real egress fleet) and kvm (VMs on your own libvirt hypervisors, reached over SSH). Config is a YAML file (--providers-config) with named instances (gcp-eu, kvm-home, ...).
• Health engine probes every proxy by fetching a URL through it (a real CONNECT tunnel — a proxy that accepts TCP but can't egress goes Unhealthy), with threshold logic and transition-only status writes.
• Discovery API (:8090): list healthy proxies filtered by attributes, lease one (least-loaded, TTL-based), release, and report rate-limiting — reports put the proxy in a per-target cooldown.
• Orphan GC sweeps each provider for tagged instances whose owning CR is gone — the safety net for crashes mid-create.

Details, diagrams, and recorded design decisions: docs/architecture.md.

Quickstart on kind (~5 minutes)

Requires: kind, kubectl, docker, Go 1.26, jq (optional). The kubernetes-pod provider needs no cloud account — proxies are real ubuntu/squid pods in the kind cluster itself.

The operator runs in-cluster for this quickstart. (Running it on your laptop with make run-dev provisions pods fine, but the health probe then originates on your machine, which cannot reach kind's pod IPs — the proxy would sit at Unhealthy forever. In-cluster, probes run where the pod network is routable.)

kind create cluster --name proxy-operator-demo
make install                                          # install the CRD
make docker-build IMG=egress-proxies-operator:dev
kind load docker-image egress-proxies-operator:dev --name proxy-operator-demo
make deploy IMG=egress-proxies-operator:dev
kubectl -n egress-proxies-operator-system rollout status deploy/egress-proxies-operator-controller-manager

Create a proxy and watch it come up:

kubectl apply -f config/samples/proxy_kubernetes.yaml
kubectl get px -w
# NAME                      MODE      PROVIDER     PHASE   IP           HEALTHY
# proxy-kubernetes-sample   Managed   kubernetes   Ready   10.244.x.x   True

Once it's Ready, port-forward the discovery API and use it (full reference with schemas and error codes: docs/api.md):

kubectl -n egress-proxies-operator-system port-forward \
  svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090 &

# List healthy proxies
curl -s 'localhost:8090/v1/proxies?healthy=true' | jq

# Lease one (5-minute TTL)
curl -s -XPOST localhost:8090/v1/leases \
  -d '{"selector":{"geo":"local"},"ttlSeconds":300}' | jq
# → {"leaseID":"...", "proxy":{"id":"default/proxy-kubernetes-sample", "ip":..., ...}}

# Actually crawl through it (from inside the cluster, or port-forward the pod)
# curl -x http://<proxy-ip>:3128 https://example.com

# Report the proxy got rate-limited by a site → 15-minute cooldown for that target
curl -s -XPOST localhost:8090/v1/leases/<leaseID>/report \
  -d '{"result":"rate_limited","target":"example.com"}'

# Release early (idempotent — 204 both times)
curl -si -XDELETE localhost:8090/v1/leases/<leaseID>

Tear down:

kubectl delete -f config/samples/proxy_kubernetes.yaml   # finalizer deletes the pod
kind delete cluster --name proxy-operator-demo

Deploying in-cluster

make docker-build IMG=<registry>/egress-proxies-operator:dev
make deploy IMG=<registry>/egress-proxies-operator:dev

• Provider config comes from the providers-config ConfigMap (config/manager/providers_config.yaml); the default ships only the kubernetes provider.

• The discovery API is exposed by the controller-manager-discovery-service Service on port 8090.

• Auth: create the token Secret, or the API serves unauthenticated (it warns loudly at startup):

  kubectl -n egress-proxies-operator-system create secret generic discovery-token \
    --from-literal=token="$(openssl rand -hex 24)"
  

GCP setup

1. Add a gcp entry to the providers config (see config/samples/providers-config.yaml) — only project is required.
2. Credentials are Application Default Credentials: workload identity in-cluster, gcloud auth application-default login locally. No key-file plumbing exists.
3. The identity needs roles/compute.instanceAdmin.v1 on the project — plus roles/iam.serviceAccountUser if instances attach a service account.
4. Managed GCP proxies must set all of placement.zone, placement.machineType, and placement.image (see config/samples/proxy_gcp.yaml, which also installs Squid via cloud-init). A missing field fails the Proxy with a message naming it.

KVM setup

Managed proxies as VMs on libvirt/KVM hosts you own. Full guide: docs/kvm-provider.md.

1. Each host needs libvirtd, an SSH user in the libvirt group, a bridge to a network with DHCP, and a storage pool.

2. Create the kvm-ssh Secret (private key + known_hosts); the manager mounts it at /etc/proxy-operator/kvm:

   kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
     --from-file=id_ed25519=<key> --from-file=known_hosts=<known_hosts>
   

3. Add a kvm entry to the providers config with images, machineTypes and hosts (see config/samples/providers-config.yaml). Base images are downloaded into each host's pool on first use.

4. A Proxy names a host (placement.zone, optional with one host), a size (placement.machineType) and an image (placement.image) — see config/samples/proxy_kvm.yaml.

Cloud-init from a Secret: the Secret must carry the label crawl.example.com/cloud-init: "true" — the operator's cache only holds labelled Secrets, so an unlabelled one is invisible (the Proxy reports CloudInitError). Rotating the Secret's content triggers VM replacement.

Providers

Managed proxies are provisioned by a provider — a small compute backend behind one minimal interface. Providers are configured in the --providers-config YAML file as named instances: spec.provider on a Proxy refers to an entry's name, not its type, so gcp-eu and gcp-us can be two differently-configured instances of the same gcp type (see config/samples/providers-config.yaml).

Implemented providers

Type	Creates	Per-instance config	Notes
kubernetes	A real Squid pod (ubuntu/squid:6.6-24.04_edge by default) in the same cluster the operator runs in	image (optional)	Needs no cloud account — local dev, CI, and the kind quickstart. Pods share the cluster's egress IP, so it exercises the full lifecycle but not distinct egress paths.
gcp	A Compute Engine VM with an ephemeral external IP	project (required), network, networkTag, diskSizeGb	The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: Create returns as soon as the operation is submitted and the reconciler discovers progress by polling Get. Auth is Application Default Credentials — workload identity in-cluster, gcloud ADC locally; no key-file plumbing.

Adding a provider

A new backend (Hetzner, AWS, ...) is four pieces; the contract lives in internal/provider/provider.go:

1. Implement the 4-method Provider interface in a new internal/provider/<type>/ package:
   • Create submits and returns — it never blocks until the VM runs, and must be idempotent keyed on req.Name (a deterministic name derived from the Proxy's UID), so a repeat call after a crash finds the existing instance instead of duplicating it.
   • Get returns provider.ErrNotFound as a normal outcome — the reconciler branches on it for replacement and adoption, so don't treat it as exceptional.
   • Delete is idempotent: deleting an already-gone instance is not an error.
   • ListByTag returns every instance the operator ever tagged, for orphan GC.
2. Tag every created resource with LabelManaged=true and LabelUID=<Proxy UID>, and report CreatedAt — orphan GC relies on all three to find owned resources and skip in-flight creates.
3. Classify every returned error with provider.Wrap into the four-sentinel taxonomy in internal/provider/errors.go (ErrNotFound / ErrQuotaExceeded / ErrTransient / ErrPermanent) — the reconciler decides retry, slow backoff, or latching Failed purely from that classification, never from provider-specific types.
4. Wire it up: add a type-specific config block in internal/provider/config.go, and register the constructor in the builtins map in cmd/main.go ("<type>": <pkg>.New). The registry (internal/provider/registry) handles named instances, and the metrics wrapper is applied automatically.

Test against a fake API seam rather than the real cloud — see the instancesAPI seam in internal/provider/gcp/gcp.go for the pattern.

Tracing

OpenTelemetry tracing is built in but off by default — the operator only exports spans when pointed at an OTLP receiver:

# In config/manager/manager.yaml (commented-out block is already there):
#   OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger-collector.observability:4318
# Local dev: print spans to stdout instead
OTEL_TRACES_EXPORTER=console go run ./cmd --providers-config hack/providers-dev.yaml

Every reconcile, discovery API request, and GC sweep becomes a trace, with child spans for provider and Kubernetes API calls; log lines inside a traced operation carry matching traceID/spanID fields (point Grafana/Loki derived fields at traceID). Sampling and endpoints follow the standard OTEL_* env vars; --trace-health-probes additionally traces each health probe (high volume). Details in docs/architecture.md §10.

Caveats — read these two

Changing a proxy changes its IP. Proxies are immutable cattle: editing placement, cloudInit (or rotating its Secret), or port deletes the VM and creates a replacement with the same name but a new IP. Clients discover the new address via the discovery API; anything that pinned the old IP breaks by design.

Operator restart drops all leases and cooldowns. Lease state is in-memory (replicas: 1 accordingly). Clients must tolerate a lease vanishing — requests through the proxy keep working; they just re-lease. The lease store sits behind an interface so a persistent backend can replace it without touching the API handlers.

Smaller notes:

• status.lastHealthCheckTime is the time of the last status-affecting probe, not the most recent probe — status writes are transition-only by design. True probe recency lives in the metrics (proxy_operator_healthcheck_*).
• The discovery API is served by every replica but is not leader-elected; the operator ships with replicas: 1 (see the lease caveat above).

Version pins

Built and verified against the spec's pins with no substitutions needed: Go 1.26, kubebuilder v4.15.0, controller-runtime v0.24.1, k8s.io/* v0.36.3 (Kubernetes 1.36 API level), controller-tools v0.21.0, cloud.google.com/go/compute v1.65.0. envtest uses the 1.36.2 binary bundle (the latest 1.36 patch with published binaries — do not "fix" the Makefile's derived version to 1.36.3, which has none).

Gitea CI

.gitea/workflows/build.yaml builds the manager image and pushes it to this Gitea instance's container registry. It runs on any tag push or manually via Run workflow (with a tag input) — never on branch pushes. A lightweight check job (go vet, go build, go test -short) gates the build.

Every build pushes two tags to gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator:

• the human tag (the git tag, or the dispatch input), and
• an immutable sha-<12-char-commit> tag — pin deployments to this one.

:latest is additionally updated on real tag pushes only, so a manual dispatch of an old ref can never clobber it. The commit is baked into the binary (internal/version.Commit) via the GIT_COMMIT build arg.

Mandatory Gitea secrets

Set under Settings → Actions → Secrets in this repo:

Secret	Required by	What it is
REGISTRY_TOKEN	build.yaml (registry login)	Gitea PAT with the write:package scope

The token is paired with ${{ github.actor }} as the username, so it must belong to the user triggering the workflow — same convention as the other projects on this instance.

Without REGISTRY_TOKEN the check job still passes but the build job fails at the docker login step. No other secrets are needed — the workflow does not deploy anywhere.

Development

make test        # unit + envtest suites, with -race (sets up envtest binaries itself)
go test -short ./...   # skip the envtest suite
make run-dev     # run against the current kubeconfig context

make run-dev is for iterating on the operator itself: provisioning, replacement, the discovery API, and External proxies all work from your laptop. Health checks against in-cluster pods do not (see the quickstart note) — use the in-cluster deploy to see a kubernetes-provider proxy go Ready.

The full test inventory — what each suite covers, the deliberate gaps, and the manual kind verification procedure — is in docs/testing.md.

Project layout, reconcile-loop diagrams, and the decision log are in docs/architecture.md; the build history is in docs/plans-executions/.