egress-proxies-operator/README.md

# egress-proxies-operator

A Kubernetes operator that manages a fleet of HTTP egress proxies for
crawling: each proxy is a `Proxy` custom resource that the operator
provisions (or merely tracks), actively health-checks **through the proxy
itself**, and hands out to crawler clients via an HTTP list/lease API.

## Architecture in 60 seconds

- **`Proxy` CRD** (`crawl.example.com/v1alpha1`, namespaced, `kubectl get px`):
  `Managed` proxies are provisioned by a configured provider; `External`
  proxies exist elsewhere and are only tracked and health-checked.
- **Reconciler** — a crash-safe state machine: every reconcile derives one
  action from (spec, status, provider Get). Proxies are **immutable
  cattle**: any meaningful spec change (placement, cloud-init, port)
  deletes and recreates the VM — never in-place mutation.
- **Providers** behind one minimal interface: `kubernetes` (a real Squid
  pod in this cluster — local dev/CI), `gcp` (Compute Engine VMs with
  ephemeral external IPs — the real egress fleet) and `kvm` (VMs on your
  own libvirt hypervisors, reached over SSH). Config is a YAML file
  (`--providers-config`) with named instances (`gcp-eu`, `kvm-home`, ...).
- **Health engine** probes every proxy by fetching a URL *through* it (a
  real CONNECT tunnel — a proxy that accepts TCP but can't egress goes
  Unhealthy), with threshold logic and transition-only status writes.
- **Discovery API** (`:8090`): list healthy proxies filtered by
  attributes, lease one (least-loaded, TTL-based), release, and report
  rate-limiting — reports put the proxy in a per-target cooldown.
- **Orphan GC** sweeps each provider for tagged instances whose owning CR
  is gone — the safety net for crashes mid-create.

Details, diagrams, and recorded design decisions: [docs/architecture.md](docs/architecture.md).

## Quickstart on kind (~5 minutes)

Requires: kind, kubectl, docker, Go 1.26, jq (optional). The
kubernetes-pod provider needs no cloud account — proxies are real
`ubuntu/squid` pods in the kind cluster itself.

The operator runs **in-cluster** for this quickstart. (Running it on your
laptop with `make run-dev` provisions pods fine, but the health probe then
originates on your machine, which cannot reach kind's pod IPs — the proxy
would sit at `Unhealthy` forever. In-cluster, probes run where the pod
network is routable.)

```sh
kind create cluster --name proxy-operator-demo
make install                                          # install the CRD
make docker-build IMG=egress-proxies-operator:dev
kind load docker-image egress-proxies-operator:dev --name proxy-operator-demo
make deploy IMG=egress-proxies-operator:dev
kubectl -n egress-proxies-operator-system rollout status deploy/egress-proxies-operator-controller-manager
```

Create a proxy and watch it come up:

```sh
kubectl apply -f config/samples/proxy_kubernetes.yaml
kubectl get px -w
# NAME                      MODE      PROVIDER     PHASE   IP           HEALTHY
# proxy-kubernetes-sample   Managed   kubernetes   Ready   10.244.x.x   True
```

Once it's `Ready`, port-forward the discovery API and use it (full
reference with schemas and error codes: [docs/api.md](docs/api.md)):

```sh
kubectl -n egress-proxies-operator-system port-forward \
  svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090 &
```

```sh
# List healthy proxies
curl -s 'localhost:8090/v1/proxies?healthy=true' | jq

# Lease one (5-minute TTL)
curl -s -XPOST localhost:8090/v1/leases \
  -d '{"selector":{"geo":"local"},"ttlSeconds":300}' | jq
# → {"leaseID":"...", "proxy":{"id":"default/proxy-kubernetes-sample", "ip":..., ...}}

# Actually crawl through it (from inside the cluster, or port-forward the pod)
# curl -x http://<proxy-ip>:3128 https://example.com

# Report the proxy got rate-limited by a site → 15-minute cooldown for that target
curl -s -XPOST localhost:8090/v1/leases/<leaseID>/report \
  -d '{"result":"rate_limited","target":"example.com"}'

# Release early (idempotent — 204 both times)
curl -si -XDELETE localhost:8090/v1/leases/<leaseID>
```

Tear down:

```sh
kubectl delete -f config/samples/proxy_kubernetes.yaml   # finalizer deletes the pod
kind delete cluster --name proxy-operator-demo
```

## Deploying in-cluster

```sh
make docker-build IMG=<registry>/egress-proxies-operator:dev
make deploy IMG=<registry>/egress-proxies-operator:dev
```

- Provider config comes from the `providers-config` ConfigMap
  ([config/manager/providers_config.yaml](config/manager/providers_config.yaml));
  the default ships only the kubernetes provider.
- The discovery API is exposed by the
  `controller-manager-discovery-service` Service on port 8090.
- Auth: create the token Secret, or the API serves **unauthenticated**
  (it warns loudly at startup):

  ```sh
  kubectl -n egress-proxies-operator-system create secret generic discovery-token \
    --from-literal=token="$(openssl rand -hex 24)"
  ```

## GCP setup

1. Add a `gcp` entry to the providers config (see
   [config/samples/providers-config.yaml](config/samples/providers-config.yaml)) —
   only `project` is required.
2. Credentials are **Application Default Credentials**: workload identity
   in-cluster, `gcloud auth application-default login` locally. No
   key-file plumbing exists.
3. The identity needs `roles/compute.instanceAdmin.v1` on the project —
   plus `roles/iam.serviceAccountUser` if instances attach a service
   account.
4. Managed GCP proxies must set all of `placement.zone`,
   `placement.machineType`, and `placement.image`
   (see [config/samples/proxy_gcp.yaml](config/samples/proxy_gcp.yaml),
   which also installs Squid via cloud-init). A missing field fails the
   Proxy with a message naming it.

## KVM setup

Managed proxies as VMs on libvirt/KVM hosts you own. Full guide:
[docs/kvm-provider.md](docs/kvm-provider.md).

1. Each host needs `libvirtd`, an SSH user in the `libvirt` group, a
   bridge to a network with DHCP, and a storage pool.
2. Create the `kvm-ssh` Secret from the operator's **private** key and a
   `known_hosts` with each host's key; the manager mounts it at
   `/etc/proxy-operator/kvm`:

   ```sh
   ssh-keyscan -H <host> >> ~/.ssh/kvm_known_hosts
   kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
     --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
     --from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
   # or: make kvm-ssh-secret   (same paths by default; KVM_SSH_KEY / KVM_KNOWN_HOSTS to override)
   ```

3. Add a `kvm` entry to the providers config with `images`,
   `machineTypes` and `hosts` (see
   [config/samples/providers-config.yaml](config/samples/providers-config.yaml)).
   Base images are downloaded into each host's pool on first use.
4. A Proxy names a host (`placement.zone`, optional with one host), a
   size (`placement.machineType`) and an image (`placement.image`) — see
   [config/samples/proxy_kvm.yaml](config/samples/proxy_kvm.yaml).

Cloud-init from a Secret: the Secret **must** carry the label
`crawl.example.com/cloud-init: "true"` — the operator's cache only holds
labelled Secrets, so an unlabelled one is invisible (the Proxy reports
`CloudInitError`). Rotating the Secret's content triggers VM replacement.

Stuck deletion: a Proxy stays `Terminating` while the operator keeps failing
to clean its instance up — credentials revoked, a provider or a KVM host
dropped from the config. Annotate it with
`crawl.example.com/skip-cleanup: "true"` to abandon the instance and release
the object. Whatever is still running then has to be removed by hand.

## Providers

`Managed` proxies are provisioned by a provider — a small compute backend
behind one minimal interface. Providers are configured in the
`--providers-config` YAML file as **named instances**: `spec.provider` on a
Proxy refers to an entry's `name`, not its `type`, so `gcp-eu` and `gcp-us`
can be two differently-configured instances of the same `gcp` type (see
[config/samples/providers-config.yaml](config/samples/providers-config.yaml)).

### Implemented providers

| Type | Creates | Per-instance config | Notes |
| --- | --- | --- | --- |
| `kubernetes` | A real Squid pod (`ubuntu/squid:6.6-24.04_edge` by default) in the same cluster the operator runs in | `image` (optional) | Needs no cloud account — local dev, CI, and the kind quickstart. Pods share the cluster's egress IP, so it exercises the full lifecycle but not distinct egress paths. |
| `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. |

### Adding a provider

A new backend (Hetzner, AWS, ...) is four pieces; the contract lives in
[internal/provider/provider.go](internal/provider/provider.go):

1. **Implement the 4-method `Provider` interface** in a new
   `internal/provider/<type>/` package:
   - `Create` submits and returns — it never blocks until the VM runs, and
     must be idempotent keyed on `req.Name` (a deterministic name derived
     from the Proxy's UID), so a repeat call after a crash finds the
     existing instance instead of duplicating it.
   - `Get` returns `provider.ErrNotFound` as a *normal* outcome — the
     reconciler branches on it for replacement and adoption, so don't
     treat it as exceptional.
   - `Delete` is idempotent: deleting an already-gone instance is not an
     error.
   - `ListByTag` returns every instance the operator ever tagged, for
     orphan GC.
2. **Tag every created resource** with `LabelManaged=true` and
   `LabelUID=<Proxy UID>`, and report `CreatedAt` — orphan GC relies on
   all three to find owned resources and skip in-flight creates.
3. **Classify every returned error** with `provider.Wrap` into the
   four-sentinel taxonomy in
   [internal/provider/errors.go](internal/provider/errors.go)
   (`ErrNotFound` / `ErrQuotaExceeded` / `ErrTransient` / `ErrPermanent`)
   — the reconciler decides retry, slow backoff, or latching `Failed`
   purely from that classification, never from provider-specific types.
4. **Wire it up**: add a type-specific config block in
   [internal/provider/config.go](internal/provider/config.go), and
   register the constructor in the builtins map in
   [cmd/main.go](cmd/main.go) (`"<type>": <pkg>.New`). The registry
   ([internal/provider/registry](internal/provider/registry/registry.go))
   handles named instances, and the metrics wrapper is applied
   automatically.

Test against a fake API seam rather than the real cloud — see the
`instancesAPI` seam in
[internal/provider/gcp/gcp.go](internal/provider/gcp/gcp.go) for the
pattern.

## Tracing

OpenTelemetry tracing is built in but **off by default** — the operator
only exports spans when pointed at an OTLP receiver:

```bash
# In config/manager/manager.yaml (commented-out block is already there):
#   OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger-collector.observability:4318
# Local dev: print spans to stdout instead
OTEL_TRACES_EXPORTER=console go run ./cmd --providers-config hack/providers-dev.yaml
```

Every reconcile, discovery API request, and GC sweep becomes a trace, with
child spans for provider and Kubernetes API calls; log lines inside a traced
operation carry matching `traceID`/`spanID` fields (point Grafana/Loki
derived fields at `traceID`). Sampling and endpoints follow the standard
`OTEL_*` env vars; `--trace-health-probes` additionally traces each health
probe (high volume). Details in `docs/architecture.md` §10.

## Caveats — read these two

**Changing a proxy changes its IP.** Proxies are immutable cattle: editing
`placement`, `cloudInit` (or rotating its Secret), or `port` deletes the
VM and creates a replacement with the **same name but a new IP**. Clients
discover the new address via the discovery API; anything that pinned the
old IP breaks by design.

**Operator restart drops all leases and cooldowns.** Lease state is
in-memory (`replicas: 1` accordingly). Clients must tolerate a lease
vanishing — requests through the proxy keep working; they just re-lease.
The lease store sits behind an interface so a persistent backend can
replace it without touching the API handlers.

Smaller notes:

- `status.lastHealthCheckTime` is the time of the last *status-affecting*
  probe, not the most recent probe — status writes are transition-only by
  design. True probe recency lives in the metrics
  (`proxy_operator_healthcheck_*`).
- The discovery API is served by every replica but is not leader-elected;
  the operator ships with `replicas: 1` (see the lease caveat above).

## Version pins

Built and verified against the spec's pins with **no substitutions
needed**: Go 1.26, kubebuilder v4.15.0, controller-runtime v0.24.1,
k8s.io/* v0.36.3 (Kubernetes 1.36 API level), controller-tools v0.21.0,
cloud.google.com/go/compute v1.65.0. envtest uses the 1.36.2 binary
bundle (the latest 1.36 patch with published binaries — do not "fix" the
Makefile's derived version to 1.36.3, which has none).

## Gitea CI

[.gitea/workflows/build.yaml](.gitea/workflows/build.yaml) builds the
manager image and pushes it to this Gitea instance's container registry.
It runs on **any tag push** or manually via **Run workflow** (with a `tag`
input) — never on branch pushes. A lightweight `check` job (`go vet`,
`go build`, `go test -short`) gates the build.

Every build pushes two tags to
`gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator`:

- the human tag (the git tag, or the dispatch input), and
- an immutable `sha-<12-char-commit>` tag — pin deployments to this one.

`:latest` is additionally updated on real tag pushes only, so a manual
dispatch of an old ref can never clobber it. The commit is baked into the
binary (`internal/version.Commit`) via the `GIT_COMMIT` build arg.

### Mandatory Gitea secrets

Set under **Settings → Actions → Secrets** in this repo:

| Secret           | Required by                   | What it is                               |
| ---------------- | ----------------------------- | ---------------------------------------- |
| `REGISTRY_TOKEN` | `build.yaml` (registry login) | Gitea PAT with the `write:package` scope |

The token is paired with `${{ github.actor }}` as the username, so it
must belong to the user triggering the workflow — same convention as the
other projects on this instance.

Without `REGISTRY_TOKEN` the `check` job still passes but the build job
fails at the `docker login` step. No other secrets are needed — the
workflow does not deploy anywhere.

## Development

```sh
make test        # unit + envtest suites, with -race (sets up envtest binaries itself)
go test -short ./...   # skip the envtest suite
make run-dev     # run against the current kubeconfig context
```

`make run-dev` is for iterating on the operator itself: provisioning,
replacement, the discovery API, and External proxies all work from your
laptop. Health checks against in-cluster pods do **not** (see the
quickstart note) — use the in-cluster deploy to see a kubernetes-provider
proxy go `Ready`.

The full test inventory — what each suite covers, the deliberate gaps,
and the manual kind verification procedure — is in
[docs/testing.md](docs/testing.md).

Project layout, reconcile-loop diagrams, and the decision log are in
[docs/architecture.md](docs/architecture.md); the build history is in
[docs/plans-executions/](docs/plans-executions/).