egress-proxies-operator/docs/api.md

Discovery API reference

The operator serves an HTTP API (the discovery API) that crawler clients use to find and lease egress proxies: list healthy proxies filtered by attributes, acquire a TTL-based lease on one, release it early, and report how a target site treated the proxy. It is implemented in internal/discovery with lease state in internal/lease; the only Kubernetes interaction is reading Proxy resources from the manager's cache.

Base URL

The API listens on :8090 (--discovery-addr) inside the manager pod and is exposed by a Service (config/default/discovery_service.yaml).

In-cluster:

http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090

From a workstation, port-forward:

kubectl -n egress-proxies-operator-system port-forward \
  svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090 &

Set BASE_URL to wherever you reach the API; all examples below use it:

export BASE_URL=localhost:8090   # via the port-forward above
# or, from inside the cluster:
# export BASE_URL=http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090

Authentication

A single static bearer token, read from the DISCOVERY_TOKEN environment variable at startup. The shipped Deployment populates it from the discovery-token Secret (key token), which is optional — if the Secret is absent or the token is empty, the API serves unauthenticated (the manager logs a loud warning at startup). Create the Secret:

kubectl -n egress-proxies-operator-system create secret generic discovery-token \
  --from-literal=token="$(openssl rand -hex 24)"

Send the token on every request:

export TOKEN=<the token>
curl -s -H "Authorization: Bearer $TOKEN" "$BASE_URL/v1/proxies" | jq

A missing or wrong token gets 401 {"error":"unauthorized",...}. GET /healthz is always exempt.

The curl examples below omit the -H "Authorization: Bearer $TOKEN" flag for brevity — add it to every call when auth is enabled.

Conventions

• Requests and responses are JSON. Errors share one envelope:

  {"error": "<machine_code>", "message": "<human-readable text>"}
  

• Request bodies are capped at 64 KiB (larger bodies fail the JSON decode with 400 invalid_body).

• Proxies with a deletion timestamp (being finalized) are excluded from every response and never offered for lease.

Configuration

Setting	Default	Meaning
--discovery-addr	:8090	Listen address of the API
--max-lease-ttl	1h	Maximum ttlSeconds a client may request
--lease-cooldown	15m	Cooldown window applied on rate_limited/banned reports
DISCOVERY_TOKEN (env)	empty	Bearer token; empty disables auth

The shipped Deployment passes none of these flags, so the defaults apply.

Endpoints

GET /healthz

Liveness check. Unauthenticated, always 200 with body ok.

curl -s "$BASE_URL/healthz"

GET /v1/proxies — list proxies

Query parameters (all optional):

Parameter	Values	Effect
healthy	true | false	Keep only proxies whose Healthy condition matches. Any other value → 400 invalid_query.
attr.<key>	any string	Exact match on spec.attributes[<key>]. Repeatable; all given pairs must match.

List everything:

curl -s "$BASE_URL/v1/proxies" | jq

List healthy proxies in a given geo:

curl -s "$BASE_URL/v1/proxies?healthy=true&attr.geo=eu" | jq

Response — 200, proxies sorted by id, an empty match is 200 with "count": 0 (never 404):

{
  "proxies": [
    {
      "id": "default/proxy-kubernetes-sample",
      "ip": "10.244.1.7",
      "port": 3128,
      "attributes": {"geo": "local"},
      "phase": "Ready",
      "healthy": true,
      "latencyMillis": 42,
      "activeLeases": 1,
      "maxLeases": 5
    }
  ],
  "count": 1
}

Proxy object fields (the same shape appears inside lease responses):

Field	Meaning
id	namespace/name of the Proxy resource; used as the stable key everywhere
ip	Effective host — spec.endpoint.host for External proxies, status.ip for Managed (empty until the backing VM/pod is up)
port	Effective port (default 3128)
attributes	spec.attributes — free-form selection labels (geo, asn, purpose, …); omitted when empty
phase	Pending | Provisioning | Ready | Unhealthy | Deleting | Failed
healthy	true iff the Healthy condition is True (the through-the-proxy health probe passes)
latencyMillis	Latency of the last status-affecting health probe
activeLeases	Currently active leases on this proxy
maxLeases	Lease capacity (default 5; an explicit 0 means unleasable)

POST /v1/leases — acquire a lease

Picks a healthy proxy with free capacity matching the selector and grants an exclusive-slot, TTL-based lease on it.

Request body (every field optional; {} is valid):

{
  "selector": {"geo": "eu"},
  "ttlSeconds": 300,
  "target": "example.com"
}

Field	Default	Meaning
selector	none	Attribute equality filter, same semantics as attr.<key> above
ttlSeconds	300 (5 min)	Lease lifetime; must be ≤ --max-lease-ttl (default 1 h), else 400 invalid_ttl
target	none	The site you intend to crawl; enables per-target cooldowns (see below)

curl -s -XPOST "$BASE_URL/v1/leases" \
  -d '{"selector":{"geo":"eu"},"ttlSeconds":300,"target":"example.com"}' | jq

Success — 201:

{
  "leaseID": "P3X6HHQTPCM5UTGVGE3B5UPS3A",
  "proxy": {
    "id": "default/proxy-eu-1",
    "ip": "34.88.10.20",
    "port": 3128,
    "attributes": {"geo": "eu"},
    "phase": "Ready",
    "healthy": true,
    "latencyMillis": 42,
    "activeLeases": 1,
    "maxLeases": 5
  },
  "expiresAt": "2026-08-11T22:05:00Z",
  "ttlSeconds": 300
}

Use proxy.ip and proxy.port as an HTTP proxy for the lease's lifetime:

curl -x http://34.88.10.20:3128 https://example.com

No match — 409 with diagnostic counts explaining why nothing qualified:

{
  "error": "no_match",
  "message": "no healthy proxy with free capacity matched the selector",
  "considered": 3,
  "atCapacity": 1,
  "inCooldown": 1,
  "unhealthy": 1
}

Count	Meaning
considered	Proxies that matched the selector (before health/capacity checks)
atCapacity	Skipped because activeLeases >= maxLeases
inCooldown	Skipped because of an active cooldown for this target (or a global one)
unhealthy	Skipped because the Healthy condition is not True

Leases expire on their own — releasing is only needed to free the slot early. There is no renew/extend endpoint; acquire a new lease instead.

DELETE /v1/leases/{id} — release early

Frees the lease's capacity slot immediately. Idempotent: always 204, including for unknown or already-expired lease IDs.

curl -si -XDELETE "$BASE_URL/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A"

POST /v1/leases/{id}/report — report an outcome

Tell the operator how the target site treated the proxy. This is the feedback signal that drives cooldowns.

Request body:

{"result": "rate_limited", "target": "example.com"}

Field	Values	Meaning
result	ok | rate_limited | banned	Anything else → 400 invalid_result
target	optional	Which site produced the result; falls back to the lease's target, then to global

curl -si -XPOST "$BASE_URL/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A/report" \
  -d '{"result":"rate_limited","target":"example.com"}'

Responses: 204 on success, 404 unknown_lease if the lease ID was never issued or has aged out.

Semantics:

• ok is a pure acknowledgement — nothing is recorded.
• rate_limited and banned currently behave identically: both put the proxy in one cooldown window (default 15 min, --lease-cooldown) for the resolved target.
• An expired lease remains reportable for one cooldown window past its TTL, so a late "we got rate-limited" still lands.

Proxy selection and cooldowns

How POST /v1/leases picks among eligible proxies (healthy, matching the selector, not being deleted, not at capacity, not in cooldown), in order:

1. fewest activeLeases (least-loaded),
2. lowest latencyMillis,
3. lexicographic id (deterministic tie-break).

Cooldowns are keyed by (proxy, target):

• A report with a target blocks that proxy only for lease requests naming the same target. Other targets — and requests with no target — still get the proxy.
• A report without a target, on a lease that also had no target, creates a global cooldown: the proxy is blocked for all lease requests until the window passes. Always pass target on leases and reports unless you really mean "this proxy is bad for everyone".

End-to-end example

# 1. Acquire a lease for crawling example.com through an EU proxy
LEASE=$(curl -s -XPOST "$BASE_URL/v1/leases" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"selector":{"geo":"eu"},"ttlSeconds":600,"target":"example.com"}')
LEASE_ID=$(echo "$LEASE" | jq -r .leaseID)
PROXY=$(echo "$LEASE" | jq -r '"\(.proxy.ip):\(.proxy.port)"')

# 2. Crawl through the leased proxy
curl -x "http://$PROXY" https://example.com/some/page

# 3. Got a 429? Report it — example.com-bound leases will avoid this
#    proxy for the next 15 minutes
curl -s -XPOST "$BASE_URL/v1/leases/$LEASE_ID/report" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"result":"rate_limited","target":"example.com"}'

# 4. Done early? Release the slot (otherwise the TTL frees it)
curl -s -XDELETE "$BASE_URL/v1/leases/$LEASE_ID" \
  -H "Authorization: Bearer $TOKEN"

Caveats

• Lease and cooldown state is in-memory and per-process. An operator restart drops all active leases and cooldowns. Clients must tolerate a granted lease disappearing (a subsequent report returns 404).
• Run a single replica. The API is served by every manager replica but is not leader-elected, and lease state is not shared between replicas; the shipped Deployment pins replicas: 1.