diff --git a/README.md b/README.md index b00e9fd..fe8314b 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,8 @@ kubectl get px -w # proxy-kubernetes-sample Managed kubernetes Ready 10.244.x.x True ``` -Once it's `Ready`, port-forward the discovery API and use it: +Once it's `Ready`, port-forward the discovery API and use it (full +reference with schemas and error codes: [docs/api.md](docs/api.md)): ```sh kubectl -n egress-proxies-operator-system port-forward \ diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..714fd48 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,317 @@ +# Discovery API reference + +The operator serves an HTTP API (the *discovery API*) that crawler clients +use to find and lease egress proxies: list healthy proxies filtered by +attributes, acquire a TTL-based lease on one, release it early, and report +how a target site treated the proxy. It is implemented in +[`internal/discovery`](../internal/discovery/) with lease state in +[`internal/lease`](../internal/lease/); the only Kubernetes interaction is +reading `Proxy` resources from the manager's cache. + +## Base URL + +The API listens on `:8090` (`--discovery-addr`) inside the manager pod and +is exposed by a Service +([config/default/discovery_service.yaml](../config/default/discovery_service.yaml)). + +In-cluster: + +```text +http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090 +``` + +From a workstation, port-forward: + +```sh +kubectl -n egress-proxies-operator-system port-forward \ + svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090 & +``` + +All examples below assume `localhost:8090` via that port-forward. + +## Authentication + +A single static bearer token, read from the `DISCOVERY_TOKEN` environment +variable at startup. The shipped Deployment populates it from the +`discovery-token` Secret (key `token`), which is **optional** — if the +Secret is absent or the token is empty, the API serves **unauthenticated** +(the manager logs a loud warning at startup). Create the Secret: + +```sh +kubectl -n egress-proxies-operator-system create secret generic discovery-token \ + --from-literal=token="$(openssl rand -hex 24)" +``` + +Send the token on every request: + +```sh +export TOKEN= +curl -s -H "Authorization: Bearer $TOKEN" localhost:8090/v1/proxies | jq +``` + +A missing or wrong token gets `401 {"error":"unauthorized",...}`. +`GET /healthz` is always exempt. + +The curl examples below omit the `-H "Authorization: Bearer $TOKEN"` flag +for brevity — add it to every call when auth is enabled. + +## Conventions + +- Requests and responses are JSON. Errors share one envelope: + + ```json + {"error": "", "message": ""} + ``` + +- Request bodies are capped at **64 KiB** (larger bodies fail the JSON + decode with `400 invalid_body`). +- Proxies with a deletion timestamp (being finalized) are excluded from + every response and never offered for lease. + +## Configuration + +| Setting | Default | Meaning | +|---|---|---| +| `--discovery-addr` | `:8090` | Listen address of the API | +| `--max-lease-ttl` | `1h` | Maximum `ttlSeconds` a client may request | +| `--lease-cooldown` | `15m` | Cooldown window applied on `rate_limited`/`banned` reports | +| `DISCOVERY_TOKEN` (env) | empty | Bearer token; empty disables auth | + +The shipped Deployment passes none of these flags, so the defaults apply. + +## Endpoints + +### `GET /healthz` + +Liveness check. Unauthenticated, always `200` with body `ok`. + +```sh +curl -s localhost:8090/healthz +``` + +### `GET /v1/proxies` — list proxies + +Query parameters (all optional): + +| Parameter | Values | Effect | +|---|---|---| +| `healthy` | `true` \| `false` | Keep only proxies whose `Healthy` condition matches. Any other value → `400 invalid_query`. | +| `attr.` | any string | Exact match on `spec.attributes[]`. Repeatable; **all** given pairs must match. | + +List everything: + +```sh +curl -s localhost:8090/v1/proxies | jq +``` + +List healthy proxies in a given geo: + +```sh +curl -s 'localhost:8090/v1/proxies?healthy=true&attr.geo=eu' | jq +``` + +Response — `200`, proxies sorted by `id`, an empty match is `200` with +`"count": 0` (never `404`): + +```json +{ + "proxies": [ + { + "id": "default/proxy-kubernetes-sample", + "ip": "10.244.1.7", + "port": 3128, + "attributes": {"geo": "local"}, + "phase": "Ready", + "healthy": true, + "latencyMillis": 42, + "activeLeases": 1, + "maxLeases": 5 + } + ], + "count": 1 +} +``` + +Proxy object fields (the same shape appears inside lease responses): + +| Field | Meaning | +|---|---| +| `id` | `namespace/name` of the `Proxy` resource; used as the stable key everywhere | +| `ip` | Effective host — `spec.endpoint.host` for `External` proxies, `status.ip` for `Managed` (empty until the backing VM/pod is up) | +| `port` | Effective port (default `3128`) | +| `attributes` | `spec.attributes` — free-form selection labels (`geo`, `asn`, `purpose`, …); omitted when empty | +| `phase` | `Pending` \| `Provisioning` \| `Ready` \| `Unhealthy` \| `Deleting` \| `Failed` | +| `healthy` | `true` iff the `Healthy` condition is `True` (the through-the-proxy health probe passes) | +| `latencyMillis` | Latency of the last status-affecting health probe | +| `activeLeases` | Currently active leases on this proxy | +| `maxLeases` | Lease capacity (default `5`; an explicit `0` means unleasable) | + +### `POST /v1/leases` — acquire a lease + +Picks a healthy proxy with free capacity matching the selector and grants +an exclusive-slot, TTL-based lease on it. + +Request body (every field optional; `{}` is valid): + +```json +{ + "selector": {"geo": "eu"}, + "ttlSeconds": 300, + "target": "example.com" +} +``` + +| Field | Default | Meaning | +|---|---|---| +| `selector` | none | Attribute equality filter, same semantics as `attr.` above | +| `ttlSeconds` | `300` (5 min) | Lease lifetime; must be ≤ `--max-lease-ttl` (default 1 h), else `400 invalid_ttl` | +| `target` | none | The site you intend to crawl; enables per-target cooldowns (see below) | + +```sh +curl -s -XPOST localhost:8090/v1/leases \ + -d '{"selector":{"geo":"eu"},"ttlSeconds":300,"target":"example.com"}' | jq +``` + +Success — `201`: + +```json +{ + "leaseID": "P3X6HHQTPCM5UTGVGE3B5UPS3A", + "proxy": { + "id": "default/proxy-eu-1", + "ip": "34.88.10.20", + "port": 3128, + "attributes": {"geo": "eu"}, + "phase": "Ready", + "healthy": true, + "latencyMillis": 42, + "activeLeases": 1, + "maxLeases": 5 + }, + "expiresAt": "2026-08-11T22:05:00Z", + "ttlSeconds": 300 +} +``` + +Use `proxy.ip` and `proxy.port` as an HTTP proxy for the lease's lifetime: + +```sh +curl -x http://34.88.10.20:3128 https://example.com +``` + +No match — `409` with diagnostic counts explaining why nothing qualified: + +```json +{ + "error": "no_match", + "message": "no healthy proxy with free capacity matched the selector", + "considered": 3, + "atCapacity": 1, + "inCooldown": 1, + "unhealthy": 1 +} +``` + +| Count | Meaning | +|---|---| +| `considered` | Proxies that matched the selector (before health/capacity checks) | +| `atCapacity` | Skipped because `activeLeases >= maxLeases` | +| `inCooldown` | Skipped because of an active cooldown for this target (or a global one) | +| `unhealthy` | Skipped because the `Healthy` condition is not `True` | + +Leases expire on their own — releasing is only needed to free the slot +early. There is no renew/extend endpoint; acquire a new lease instead. + +### `DELETE /v1/leases/{id}` — release early + +Frees the lease's capacity slot immediately. Idempotent: always `204`, +including for unknown or already-expired lease IDs. + +```sh +curl -si -XDELETE localhost:8090/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A +``` + +### `POST /v1/leases/{id}/report` — report an outcome + +Tell the operator how the target site treated the proxy. This is the +feedback signal that drives cooldowns. + +Request body: + +```json +{"result": "rate_limited", "target": "example.com"} +``` + +| Field | Values | Meaning | +|---|---|---| +| `result` | `ok` \| `rate_limited` \| `banned` | Anything else → `400 invalid_result` | +| `target` | optional | Which site produced the result; falls back to the lease's `target`, then to global | + +```sh +curl -si -XPOST localhost:8090/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A/report \ + -d '{"result":"rate_limited","target":"example.com"}' +``` + +Responses: `204` on success, `404 unknown_lease` if the lease ID was never +issued or has aged out. + +Semantics: + +- `ok` is a pure acknowledgement — nothing is recorded. +- `rate_limited` and `banned` currently behave **identically**: both put + the proxy in one cooldown window (default 15 min, `--lease-cooldown`) + for the resolved target. +- An expired lease remains reportable for one cooldown window past its + TTL, so a late "we got rate-limited" still lands. + +## Proxy selection and cooldowns + +How `POST /v1/leases` picks among eligible proxies (healthy, matching the +selector, not being deleted, not at capacity, not in cooldown), in order: + +1. fewest `activeLeases` (least-loaded), +2. lowest `latencyMillis`, +3. lexicographic `id` (deterministic tie-break). + +Cooldowns are keyed by **(proxy, target)**: + +- A report **with a target** blocks that proxy only for lease requests + naming the **same target**. Other targets — and requests with no + target — still get the proxy. +- A report **without a target**, on a lease that also had no target, + creates a **global** cooldown: the proxy is blocked for *all* lease + requests until the window passes. Always pass `target` on leases and + reports unless you really mean "this proxy is bad for everyone". + +## End-to-end example + +```sh +# 1. Acquire a lease for crawling example.com through an EU proxy +LEASE=$(curl -s -XPOST localhost:8090/v1/leases \ + -H "Authorization: Bearer $TOKEN" \ + -d '{"selector":{"geo":"eu"},"ttlSeconds":600,"target":"example.com"}') +LEASE_ID=$(echo "$LEASE" | jq -r .leaseID) +PROXY=$(echo "$LEASE" | jq -r '"\(.proxy.ip):\(.proxy.port)"') + +# 2. Crawl through the leased proxy +curl -x "http://$PROXY" https://example.com/some/page + +# 3. Got a 429? Report it — example.com-bound leases will avoid this +# proxy for the next 15 minutes +curl -s -XPOST "localhost:8090/v1/leases/$LEASE_ID/report" \ + -H "Authorization: Bearer $TOKEN" \ + -d '{"result":"rate_limited","target":"example.com"}' + +# 4. Done early? Release the slot (otherwise the TTL frees it) +curl -s -XDELETE "localhost:8090/v1/leases/$LEASE_ID" \ + -H "Authorization: Bearer $TOKEN" +``` + +## Caveats + +- **Lease and cooldown state is in-memory and per-process.** An operator + restart drops all active leases and cooldowns. Clients must tolerate a + granted lease disappearing (a subsequent report returns `404`). +- **Run a single replica.** The API is served by every manager replica but + is not leader-elected, and lease state is not shared between replicas; + the shipped Deployment pins `replicas: 1`. diff --git a/docs/architecture.md b/docs/architecture.md index 619cd98..7946caf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -213,6 +213,8 @@ Kubernetes interaction is reading Proxies from the manager's cache. The server is a non-leader-elected Runnable (all replicas would serve, but the deployment ships `replicas: 1` because lease state is per-process — an operator restart drops all leases and cooldowns, a documented caveat). +Client-facing reference with request/response schemas and curl examples: +[api.md](api.md). ```text crawler client diff --git a/docs/plans-executions/2026-08-11-2152-discovery-api-docs.md b/docs/plans-executions/2026-08-11-2152-discovery-api-docs.md new file mode 100644 index 0000000..b878d0a --- /dev/null +++ b/docs/plans-executions/2026-08-11-2152-discovery-api-docs.md @@ -0,0 +1,38 @@ +# Execution: Discovery API documentation + +Plan: [2026-08-11-2152-discovery-api-docs.md](../plans/2026-08-11-2152-discovery-api-docs.md) + +- [x] Step 1 — Write `docs/api.md` full API reference +- [x] Step 2 — Add pointers in README and architecture.md + +## Step 1 — docs/api.md + +Wrote the full reference: base URL (in-cluster FQDN + port-forward), bearer +auth, error envelope, configuration table, all five routes with schemas, +status codes and curl examples, the selection/cooldown semantics section, +an end-to-end curl walkthrough, and the in-memory/single-replica caveats. +All facts were taken from the code, not from memory of prior docs. + +Worth noting: the doc explicitly calls out two things no earlier doc +stated for clients — that a report **without** a target on a targetless +lease creates a *global* cooldown (blocking the proxy for everyone), and +that `rate_limited` and `banned` currently behave identically. Both came +straight from `internal/lease/store.go` and are easy to trip over. + +## Step 2 — Pointers + verification + +Added one-line links to the new doc in README's quickstart (above the curl +block) and in `docs/architecture.md` §7. Verified the documented behavior +against the tree rather than trusting the write-up: + +```sh +go build ./... && go test -short ./internal/discovery/ ./internal/lease/ +``` + +Both pass; a grep of `server_test.go` confirmed every documented status +code and error code (`invalid_ttl`, `invalid_query`, `invalid_result`, +`no_match`, `unknown_lease`, 201/204/401/404/409) is asserted by tests. + +Worth noting: CHANGELOG entry deliberately deferred until the user +confirms the docs read well, per the CHANGELOG convention's +"once the user confirms it works" clause.