kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Document the discovery API in docs/api.md

Browse Source 
Full client-facing reference: auth, all routes with schemas and curl
examples, selection/cooldown semantics, caveats. README and
architecture.md link to it.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Jan Novak
2026-08-11 21:55:16 +02:00
parent e7fdae0859
commit 09845e4eaf
4 changed files with 359 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
README.md
View File

@@ -59,7 +59,8 @@ kubectl get px -w
# proxy-kubernetes-sample Managed kubernetes Ready 10.244.x.x True # proxy-kubernetes-sample Managed kubernetes Ready 10.244.x.x True
``` ```
Once it's `Ready`, port-forward the discovery API and use it: Once it's `Ready`, port-forward the discovery API and use it (full
reference with schemas and error codes: [docs/api.md](docs/api.md)):
```sh ```sh
kubectl -n egress-proxies-operator-system port-forward \ kubectl -n egress-proxies-operator-system port-forward \

317
docs/api.md Normal file
View File

@@ -0,0 +1,317 @@
# Discovery API reference
The operator serves an HTTP API (the *discovery API*) that crawler clients
use to find and lease egress proxies: list healthy proxies filtered by
attributes, acquire a TTL-based lease on one, release it early, and report
how a target site treated the proxy. It is implemented in
[`internal/discovery`](../internal/discovery/) with lease state in
[`internal/lease`](../internal/lease/); the only Kubernetes interaction is
reading `Proxy` resources from the manager's cache.
## Base URL
The API listens on `:8090` (`--discovery-addr`) inside the manager pod and
is exposed by a Service
([config/default/discovery_service.yaml](../config/default/discovery_service.yaml)).
In-cluster:
```text
http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090
```
From a workstation, port-forward:
```sh
kubectl -n egress-proxies-operator-system port-forward \
svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090 &
```
All examples below assume `localhost:8090` via that port-forward.
## Authentication
A single static bearer token, read from the `DISCOVERY_TOKEN` environment
variable at startup. The shipped Deployment populates it from the
`discovery-token` Secret (key `token`), which is **optional** — if the
Secret is absent or the token is empty, the API serves **unauthenticated**
(the manager logs a loud warning at startup). Create the Secret:
```sh
kubectl -n egress-proxies-operator-system create secret generic discovery-token \
--from-literal=token="$(openssl rand -hex 24)"
```
Send the token on every request:
```sh
export TOKEN=<the token>
curl -s -H "Authorization: Bearer $TOKEN" localhost:8090/v1/proxies | jq
```
A missing or wrong token gets `401 {"error":"unauthorized",...}`.
`GET /healthz` is always exempt.
The curl examples below omit the `-H "Authorization: Bearer $TOKEN"` flag
for brevity — add it to every call when auth is enabled.
## Conventions
- Requests and responses are JSON. Errors share one envelope:
```json
{"error": "<machine_code>", "message": "<human-readable text>"}
```
- Request bodies are capped at **64 KiB** (larger bodies fail the JSON
decode with `400 invalid_body`).
- Proxies with a deletion timestamp (being finalized) are excluded from
every response and never offered for lease.
## Configuration
| Setting | Default | Meaning |
|---|---|---|
| `--discovery-addr` | `:8090` | Listen address of the API |
| `--max-lease-ttl` | `1h` | Maximum `ttlSeconds` a client may request |
| `--lease-cooldown` | `15m` | Cooldown window applied on `rate_limited`/`banned` reports |
| `DISCOVERY_TOKEN` (env) | empty | Bearer token; empty disables auth |
The shipped Deployment passes none of these flags, so the defaults apply.
## Endpoints
### `GET /healthz`
Liveness check. Unauthenticated, always `200` with body `ok`.
```sh
curl -s localhost:8090/healthz
```
### `GET /v1/proxies` — list proxies
Query parameters (all optional):
| Parameter | Values | Effect |
|---|---|---|
| `healthy` | `true` \| `false` | Keep only proxies whose `Healthy` condition matches. Any other value → `400 invalid_query`. |
| `attr.<key>` | any string | Exact match on `spec.attributes[<key>]`. Repeatable; **all** given pairs must match. |
List everything:
```sh
curl -s localhost:8090/v1/proxies | jq
```
List healthy proxies in a given geo:
```sh
curl -s 'localhost:8090/v1/proxies?healthy=true&attr.geo=eu' | jq
```
Response — `200`, proxies sorted by `id`, an empty match is `200` with
`"count": 0` (never `404`):
```json
{
"proxies": [
{
"id": "default/proxy-kubernetes-sample",
"ip": "10.244.1.7",
"port": 3128,
"attributes": {"geo": "local"},
"phase": "Ready",
"healthy": true,
"latencyMillis": 42,
"activeLeases": 1,
"maxLeases": 5
}
],
"count": 1
}
```
Proxy object fields (the same shape appears inside lease responses):
| Field | Meaning |
|---|---|
| `id` | `namespace/name` of the `Proxy` resource; used as the stable key everywhere |
| `ip` | Effective host — `spec.endpoint.host` for `External` proxies, `status.ip` for `Managed` (empty until the backing VM/pod is up) |
| `port` | Effective port (default `3128`) |
| `attributes` | `spec.attributes` — free-form selection labels (`geo`, `asn`, `purpose`, …); omitted when empty |
| `phase` | `Pending` \| `Provisioning` \| `Ready` \| `Unhealthy` \| `Deleting` \| `Failed` |
| `healthy` | `true` iff the `Healthy` condition is `True` (the through-the-proxy health probe passes) |
| `latencyMillis` | Latency of the last status-affecting health probe |
| `activeLeases` | Currently active leases on this proxy |
| `maxLeases` | Lease capacity (default `5`; an explicit `0` means unleasable) |
### `POST /v1/leases` — acquire a lease
Picks a healthy proxy with free capacity matching the selector and grants
an exclusive-slot, TTL-based lease on it.
Request body (every field optional; `{}` is valid):
```json
{
"selector": {"geo": "eu"},
"ttlSeconds": 300,
"target": "example.com"
}
```
| Field | Default | Meaning |
|---|---|---|
| `selector` | none | Attribute equality filter, same semantics as `attr.<key>` above |
| `ttlSeconds` | `300` (5 min) | Lease lifetime; must be ≤ `--max-lease-ttl` (default 1 h), else `400 invalid_ttl` |
| `target` | none | The site you intend to crawl; enables per-target cooldowns (see below) |
```sh
curl -s -XPOST localhost:8090/v1/leases \
-d '{"selector":{"geo":"eu"},"ttlSeconds":300,"target":"example.com"}' | jq
```
Success — `201`:
```json
{
"leaseID": "P3X6HHQTPCM5UTGVGE3B5UPS3A",
"proxy": {
"id": "default/proxy-eu-1",
"ip": "34.88.10.20",
"port": 3128,
"attributes": {"geo": "eu"},
"phase": "Ready",
"healthy": true,
"latencyMillis": 42,
"activeLeases": 1,
"maxLeases": 5
},
"expiresAt": "2026-08-11T22:05:00Z",
"ttlSeconds": 300
}
```
Use `proxy.ip` and `proxy.port` as an HTTP proxy for the lease's lifetime:
```sh
curl -x http://34.88.10.20:3128 https://example.com
```
No match — `409` with diagnostic counts explaining why nothing qualified:
```json
{
"error": "no_match",
"message": "no healthy proxy with free capacity matched the selector",
"considered": 3,
"atCapacity": 1,
"inCooldown": 1,
"unhealthy": 1
}
```
| Count | Meaning |
|---|---|
| `considered` | Proxies that matched the selector (before health/capacity checks) |
| `atCapacity` | Skipped because `activeLeases >= maxLeases` |
| `inCooldown` | Skipped because of an active cooldown for this target (or a global one) |
| `unhealthy` | Skipped because the `Healthy` condition is not `True` |
Leases expire on their own — releasing is only needed to free the slot
early. There is no renew/extend endpoint; acquire a new lease instead.
### `DELETE /v1/leases/{id}` — release early
Frees the lease's capacity slot immediately. Idempotent: always `204`,
including for unknown or already-expired lease IDs.
```sh
curl -si -XDELETE localhost:8090/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A
```
### `POST /v1/leases/{id}/report` — report an outcome
Tell the operator how the target site treated the proxy. This is the
feedback signal that drives cooldowns.
Request body:
```json
{"result": "rate_limited", "target": "example.com"}
```
| Field | Values | Meaning |
|---|---|---|
| `result` | `ok` \| `rate_limited` \| `banned` | Anything else → `400 invalid_result` |
| `target` | optional | Which site produced the result; falls back to the lease's `target`, then to global |
```sh
curl -si -XPOST localhost:8090/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A/report \
-d '{"result":"rate_limited","target":"example.com"}'
```
Responses: `204` on success, `404 unknown_lease` if the lease ID was never
issued or has aged out.
Semantics:
- `ok` is a pure acknowledgement — nothing is recorded.
- `rate_limited` and `banned` currently behave **identically**: both put
the proxy in one cooldown window (default 15 min, `--lease-cooldown`)
for the resolved target.
- An expired lease remains reportable for one cooldown window past its
TTL, so a late "we got rate-limited" still lands.
## Proxy selection and cooldowns
How `POST /v1/leases` picks among eligible proxies (healthy, matching the
selector, not being deleted, not at capacity, not in cooldown), in order:
1. fewest `activeLeases` (least-loaded),
2. lowest `latencyMillis`,
3. lexicographic `id` (deterministic tie-break).
Cooldowns are keyed by **(proxy, target)**:
- A report **with a target** blocks that proxy only for lease requests
naming the **same target**. Other targets — and requests with no
target — still get the proxy.
- A report **without a target**, on a lease that also had no target,
creates a **global** cooldown: the proxy is blocked for *all* lease
requests until the window passes. Always pass `target` on leases and
reports unless you really mean "this proxy is bad for everyone".
## End-to-end example
```sh
# 1. Acquire a lease for crawling example.com through an EU proxy
LEASE=$(curl -s -XPOST localhost:8090/v1/leases \
-H "Authorization: Bearer $TOKEN" \
-d '{"selector":{"geo":"eu"},"ttlSeconds":600,"target":"example.com"}')
LEASE_ID=$(echo "$LEASE" | jq -r .leaseID)
PROXY=$(echo "$LEASE" | jq -r '"\(.proxy.ip):\(.proxy.port)"')
# 2. Crawl through the leased proxy
curl -x "http://$PROXY" https://example.com/some/page
# 3. Got a 429? Report it — example.com-bound leases will avoid this
# proxy for the next 15 minutes
curl -s -XPOST "localhost:8090/v1/leases/$LEASE_ID/report" \
-H "Authorization: Bearer $TOKEN" \
-d '{"result":"rate_limited","target":"example.com"}'
# 4. Done early? Release the slot (otherwise the TTL frees it)
curl -s -XDELETE "localhost:8090/v1/leases/$LEASE_ID" \
-H "Authorization: Bearer $TOKEN"
```
## Caveats
- **Lease and cooldown state is in-memory and per-process.** An operator
restart drops all active leases and cooldowns. Clients must tolerate a
granted lease disappearing (a subsequent report returns `404`).
- **Run a single replica.** The API is served by every manager replica but
is not leader-elected, and lease state is not shared between replicas;
the shipped Deployment pins `replicas: 1`.

2
docs/architecture.md
View File

@@ -213,6 +213,8 @@ Kubernetes interaction is reading Proxies from the manager's cache. The
server is a non-leader-elected Runnable (all replicas would serve, but the server is a non-leader-elected Runnable (all replicas would serve, but the
deployment ships `replicas: 1` because lease state is per-process — an deployment ships `replicas: 1` because lease state is per-process — an
operator restart drops all leases and cooldowns, a documented caveat). operator restart drops all leases and cooldowns, a documented caveat).
Client-facing reference with request/response schemas and curl examples:
[api.md](api.md).
```text ```text
crawler client crawler client

38
docs/plans-executions/2026-08-11-2152-discovery-api-docs.md Normal file
View File

@@ -0,0 +1,38 @@
# Execution: Discovery API documentation
Plan: [2026-08-11-2152-discovery-api-docs.md](../plans/2026-08-11-2152-discovery-api-docs.md)
- [x] Step 1 — Write `docs/api.md` full API reference
- [x] Step 2 — Add pointers in README and architecture.md
## Step 1 — docs/api.md
Wrote the full reference: base URL (in-cluster FQDN + port-forward), bearer
auth, error envelope, configuration table, all five routes with schemas,
status codes and curl examples, the selection/cooldown semantics section,
an end-to-end curl walkthrough, and the in-memory/single-replica caveats.
All facts were taken from the code, not from memory of prior docs.
Worth noting: the doc explicitly calls out two things no earlier doc
stated for clients — that a report **without** a target on a targetless
lease creates a *global* cooldown (blocking the proxy for everyone), and
that `rate_limited` and `banned` currently behave identically. Both came
straight from `internal/lease/store.go` and are easy to trip over.
## Step 2 — Pointers + verification
Added one-line links to the new doc in README's quickstart (above the curl
block) and in `docs/architecture.md` §7. Verified the documented behavior
against the tree rather than trusting the write-up:
```sh
go build ./... && go test -short ./internal/discovery/ ./internal/lease/
```
Both pass; a grep of `server_test.go` confirmed every documented status
code and error code (`invalid_ttl`, `invalid_query`, `invalid_result`,
`no_match`, `unknown_lease`, 201/204/401/404/409) is asserted by tests.
Worth noting: CHANGELOG entry deliberately deferred until the user
confirms the docs read well, per the CHANGELOG convention's
"once the user confirms it works" clause.