egress-proxies-operator

A Kubernetes operator that manages a fleet of HTTP egress proxies for crawling: each proxy is a Proxy custom resource that the operator provisions (or merely tracks), actively health-checks through the proxy itself, and hands out to crawler clients via an HTTP list/lease API.

Architecture in 60 seconds

• Proxy CRD (crawl.example.com/v1alpha1, namespaced, kubectl get px): Managed proxies are provisioned by a configured provider; External proxies exist elsewhere and are only tracked and health-checked.
• Reconciler — a crash-safe state machine: every reconcile derives one action from (spec, status, provider Get). Proxies are immutable cattle: any meaningful spec change (placement, cloud-init, port) deletes and recreates the VM — never in-place mutation.
• Providers behind one minimal interface: kubernetes (a real Squid pod in this cluster — local dev/CI), gcp (Compute Engine VMs with ephemeral external IPs — the real egress fleet) and kvm (VMs on your own libvirt hypervisors, reached over SSH). Config is a YAML file (--providers-config) with named instances (gcp-eu, kvm-home, ...).
• Health engine probes every proxy by fetching a URL through it (a real CONNECT tunnel — a proxy that accepts TCP but can't egress goes Unhealthy), with threshold logic and transition-only status writes.
• Discovery API (:8090): list healthy proxies filtered by attributes, lease one (least-loaded, TTL-based), release, and report rate-limiting — reports put the proxy in a per-target cooldown.
• Orphan GC sweeps each provider for tagged instances whose owning CR is gone — the safety net for crashes mid-create.

Details, diagrams, and recorded design decisions: docs/architecture.md.

Quickstart on kind (~5 minutes)

Requires: kind, kubectl, docker, Go 1.26, jq (optional). The kubernetes-pod provider needs no cloud account — proxies are real ubuntu/squid pods in the kind cluster itself.

The operator runs in-cluster for this quickstart. (Running it on your laptop with make run-dev provisions pods fine, but the health probe then originates on your machine, which cannot reach kind's pod IPs — the proxy would sit at Unhealthy forever. In-cluster, probes run where the pod network is routable.)

kind create cluster --name proxy-operator-demo
make install                                          # install the CRD
make docker-build IMG=egress-proxies-operator:dev
kind load docker-image egress-proxies-operator:dev --name proxy-operator-demo
make deploy IMG=egress-proxies-operator:dev
kubectl -n egress-proxies-operator-system rollout status deploy/egress-proxies-operator-controller-manager

Create a proxy and watch it come up:

kubectl apply -f config/samples/proxy_kubernetes.yaml
kubectl get px -w
# NAME                      MODE      PROVIDER     PHASE   IP           HEALTHY
# proxy-kubernetes-sample   Managed   kubernetes   Ready   10.244.x.x   True

Once it's Ready, port-forward the discovery API and use it (full reference with schemas and error codes: docs/api.md):

kubectl -n egress-proxies-operator-system port-forward \
  svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090 &

# List healthy proxies
curl -s 'localhost:8090/v1/proxies?healthy=true' | jq

# Lease one (5-minute TTL)
curl -s -XPOST localhost:8090/v1/leases \
  -d '{"selector":{"geo":"local"},"ttlSeconds":300}' | jq
# → {"leaseID":"...", "proxy":{"id":"default/proxy-kubernetes-sample", "ip":..., ...}}

# Actually crawl through it (from inside the cluster, or port-forward the pod)
# curl -x http://<proxy-ip>:3128 https://example.com

# Report the proxy got rate-limited by a site → 15-minute cooldown for that target
curl -s -XPOST localhost:8090/v1/leases/<leaseID>/report \
  -d '{"result":"rate_limited","target":"example.com"}'

# Release early (idempotent — 204 both times)
curl -si -XDELETE localhost:8090/v1/leases/<leaseID>

Tear down:

kubectl delete -f config/samples/proxy_kubernetes.yaml   # finalizer deletes the pod
kind delete cluster --name proxy-operator-demo

Deploying in-cluster

make docker-build IMG=<registry>/egress-proxies-operator:dev
make deploy IMG=<registry>/egress-proxies-operator:dev

• Provider config comes from the providers-config ConfigMap (config/manager/providers_config.yaml); the default ships the kubernetes provider plus a kvm-home entry that needs the kvm-ssh Secret (without it the manager starts and warns).

• The discovery API is exposed by the controller-manager-discovery-service Service on port 8090.

• Auth: create the token Secret, or the API serves unauthenticated (it warns loudly at startup):

  kubectl -n egress-proxies-operator-system create secret generic discovery-token \
    --from-literal=token="$(openssl rand -hex 24)"
  

GCP setup

1. Add a gcp entry to the providers config (see config/samples/providers-config.yaml) — only project is required.
2. Credentials are Application Default Credentials: workload identity in-cluster, gcloud auth application-default login locally. No key-file plumbing exists.
3. The identity needs roles/compute.instanceAdmin.v1 on the project — plus roles/iam.serviceAccountUser if instances attach a service account.
4. Managed GCP proxies must set all of placement.zone, placement.machineType, and placement.image (see config/samples/proxy_gcp.yaml, which also installs Squid via cloud-init). A missing field fails the Proxy with a message naming it.

KVM setup

Managed proxies as VMs on libvirt/KVM hosts you own. Full guide: docs/kvm-provider.md.

1. Each host needs libvirtd, an SSH user in the libvirt group, a bridge to a network with DHCP, and a storage pool.

2. Create the kvm-ssh Secret from the operator's private key and a known_hosts with each host's key; the manager mounts it at /etc/proxy-operator/kvm:

   ssh-keyscan -H <host> >> ~/.ssh/kvm_known_hosts
   kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
     --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
     --from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
   # or: make kvm-ssh-secret   (same paths by default; KVM_SSH_KEY / KVM_KNOWN_HOSTS to override)
   

3. Add a kvm entry to the providers config with images, machineTypes and hosts (see config/samples/providers-config.yaml). Base images are downloaded into each host's pool on first use.

4. A Proxy names a host (placement.zone, optional with one host), a size (placement.machineType) and an image (placement.image) — see config/samples/proxy_kvm.yaml.

The kvm2 provider uses the same Secret and the same block shape (type: kvm2, block kvm2:), so the steps above apply to it unchanged; its guide is docs/kvm2-provider.md.

Cloud-init from a Secret: the Secret must carry the label crawl.example.com/cloud-init: "true" — the operator's cache only holds labelled Secrets, so an unlabelled one is invisible (the Proxy reports CloudInitError). Rotating the Secret's content triggers VM replacement.

Stuck deletion: a Proxy stays Terminating while the operator keeps failing to clean its instance up — credentials revoked, a provider or a KVM host dropped from the config. Annotate it with crawl.example.com/skip-cleanup: "true" to abandon the instance and release the object. Whatever is still running then has to be removed by hand.

Providers

Managed proxies are provisioned by a provider — a small compute backend behind one minimal interface. Providers are configured in the --providers-config YAML file as named instances: spec.provider on a Proxy refers to an entry's name, not its type, so gcp-eu and gcp-us can be two differently-configured instances of the same gcp type (see config/samples/providers-config.yaml).

Implemented providers

Type	Creates	Per-instance config	Notes
kubernetes	A real Squid pod (ubuntu/squid:6.6-24.04_edge by default) in the same cluster the operator runs in	image (optional)	Needs no cloud account — local dev, CI, and the kind quickstart. Pods share the cluster's egress IP, so it exercises the full lifecycle but not distinct egress paths.
gcp	A Compute Engine VM with an ephemeral external IP	project (required), network, networkTag, diskSizeGb	The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: Create returns as soon as the operation is submitted and the reconciler discovers progress by polling Get. Auth is Application Default Credentials — workload identity in-cluster, gcloud ADC locally; no key-file plumbing.
kvm	A libvirt VM on one of your own hypervisors, reached over SSH	sshKeyFile, knownHostsFile (unless insecureIgnoreHostKey), machineTypes, hosts (required); images, diskSizeGb, vendorData, installGuestAgent	Home-lab or on-prem egress. The base image is downloaded into each host's pool on first use; Create waits for the VM to start. See docs/kvm-provider.md.
kvm2	The same: a libvirt VM on one of your own hypervisors, reached over SSH	the same block shape as kvm, under kvm2:	The second-generation libvirt provider, built from the provider contract. Adds a child span per libvirt RPC, a pool-space precheck reported as quota (the reconciler waits instead of failing), SSH credentials re-read on every dial, and image downloads and health checks on connections of their own. It does not adopt kvm's VMs. See docs/kvm2-provider.md.
proxmox	A VM on a Proxmox VE cluster: a full clone of a template you prepare, reached through the Proxmox REST API with an API token	url, tokenFile, nodes, isoStorage, templates, machineTypes (required); caFile or insecureSkipTLSVerify, storage, bridge, vlanTag, vmidRange, requestTimeoutSeconds, vendorData, installGuestAgent	Home-lab or on-prem egress without SSH. Cloud-init goes on a seed ISO uploaded through the API; the address comes from the QEMU guest agent, so the proxy network needs a route to a package mirror unless the template ships the agent. Create waits for the VM to run. The token is read on every request, from the optional proxmox-token Secret (make proxmox-token-secret). Built from the provider contract; not yet run against a real host end to end. See docs/proxmox-provider.md.

Adding a provider

The generic contract is docs/provider-contract.md, the questionnaire a new provider's design fills in is docs/provider-spec-template.md, and how the existing providers meet the contract is docs/providers-existing.md; docs/providers.md indexes them. Start there: several rules (a self-contained providerID, Get returning ErrNotFound only when the instance is really gone, CreatedAt read back for orphan GC) are enforced by callers, not by the interface.

Tracing

OpenTelemetry tracing is built in but off by default — the operator only exports spans when pointed at an OTLP receiver:

# In config/manager/manager.yaml (commented-out block is already there):
#   OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger-collector.observability:4318
# Local dev: print spans to stdout instead
OTEL_TRACES_EXPORTER=console go run ./cmd --providers-config hack/providers-dev.yaml

Every reconcile, discovery API request, and GC sweep becomes a trace, with child spans for provider and Kubernetes API calls; log lines inside a traced operation carry matching traceID/spanID fields (point Grafana/Loki derived fields at traceID). Sampling and endpoints follow the standard OTEL_* env vars; --trace-health-probes additionally traces each health probe (high volume). Details in docs/architecture.md §10.

Caveats — read these two

Changing a proxy changes its IP. Proxies are immutable cattle: editing placement, cloudInit (or rotating its Secret), or port deletes the VM and creates a replacement with the same name but a new IP. Clients discover the new address via the discovery API; anything that pinned the old IP breaks by design.

Operator restart drops all leases and cooldowns. Lease state is in-memory (replicas: 1 accordingly). Clients must tolerate a lease vanishing — requests through the proxy keep working; they just re-lease. The lease store sits behind an interface so a persistent backend can replace it without touching the API handlers.

Smaller notes:

• status.lastHealthCheckTime is the time of the last status-affecting probe, not the most recent probe — status writes are transition-only by design. True probe recency lives in the metrics (proxy_operator_healthcheck_*).
• The discovery API is served by every replica but is not leader-elected; the operator ships with replicas: 1 (see the lease caveat above).

Version pins

Built and verified against the spec's pins with no substitutions needed: Go 1.26, kubebuilder v4.15.0, controller-runtime v0.24.1, k8s.io/* v0.36.3 (Kubernetes 1.36 API level), controller-tools v0.21.0, cloud.google.com/go/compute v1.65.0. envtest uses the 1.36.2 binary bundle (the latest 1.36 patch with published binaries — do not "fix" the Makefile's derived version to 1.36.3, which has none).

Gitea CI

.gitea/workflows/build.yaml builds the manager image and pushes it to this Gitea instance's container registry. It runs on any tag push or manually via Run workflow (with a tag input) — never on branch pushes. A lightweight check job (go vet, go build, go test -short) gates the build.

Every build pushes two tags to gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator:

• the human tag (the git tag, or the dispatch input), and
• an immutable sha-<12-char-commit> tag — pin deployments to this one.

:latest is additionally updated on real tag pushes only, so a manual dispatch of an old ref can never clobber it. The commit is baked into the binary (internal/version.Commit) via the GIT_COMMIT build arg.

Mandatory Gitea secrets

Set under Settings → Actions → Secrets in this repo:

Secret	Required by	What it is
REGISTRY_TOKEN	build.yaml (registry login)	Gitea PAT with the write:package scope

The token is paired with ${{ github.actor }} as the username, so it must belong to the user triggering the workflow — same convention as the other projects on this instance.

Without REGISTRY_TOKEN the check job still passes but the build job fails at the docker login step. No other secrets are needed — the workflow does not deploy anywhere.

Development

make test        # unit + envtest suites, with -race (sets up envtest binaries itself)
go test -short ./...   # skip the envtest suite
make run-dev     # run against the current kubeconfig context

make run-dev is for iterating on the operator itself: provisioning, replacement, the discovery API, and External proxies all work from your laptop. Health checks against in-cluster pods do not (see the quickstart note) — use the in-cluster deploy to see a kubernetes-provider proxy go Ready.

The full test inventory — what each suite covers, the deliberate gaps, and the manual kind verification procedure — is in docs/testing.md.

Project layout, reconcile-loop diagrams, and the decision log are in docs/architecture.md; the build history is in docs/plans-executions/. Day-to-day helpers for a running operator — log queries and one-off inspections — are collected in docs/operating.md.