kacerrandClaude Fable 5
1610cd69e0
limiter/ package (ex03 bucket + ex09 sharded keyed store + idle sweep, race-tested), server/ with /allow /stats /healthz and X-RateLimit headers, client/ demo replica with fail-open/closed and latency percentiles. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GbnJj9ZqEx9nUvzkjdwfMq
119 lines
4.9 KiB
Markdown
119 lines
4.9 KiB
Markdown
# Experiment — a global rate limiter as a service
|
||
|
||
Exercise 03 built an in-process token bucket. That limits **one process**.
|
||
When N replicas of a service must share one quota ("tenant-a gets 10 req/s,
|
||
total"), the bucket has to live somewhere they can all reach. This is the
|
||
smallest honest version of that: an HTTP service holding per-key buckets, and
|
||
a demo client that plays one replica.
|
||
|
||
```
|
||
99-experiments/global-ratelimit/
|
||
limiter/ token bucket (ex03) + sharded keyed store (ex09) + idle eviction go test -race
|
||
server/ HTTP service: /allow, /stats, /healthz
|
||
client/ demo replica: asks /allow before each unit of work
|
||
```
|
||
|
||
## Run it
|
||
|
||
```
|
||
# terminal 1 — the limiter
|
||
go run ./99-experiments/global-ratelimit/server -rate 10 -burst 10
|
||
|
||
# quick check
|
||
for i in $(seq 12); do curl -s -o /dev/null -w '%{http_code} ' 'localhost:8100/allow?key=t'; done
|
||
# 200 200 200 200 200 200 200 200 200 200 429 429
|
||
curl -si 'localhost:8100/allow?key=t' | grep -iE 'HTTP/|retry|ratelimit'
|
||
|
||
# terminals 2–4 — three replicas, each WANTS 20/s, all on key tenant-a
|
||
go run ./99-experiments/global-ratelimit/client -id inst-1 -rps 20
|
||
go run ./99-experiments/global-ratelimit/client -id inst-2 -rps 20
|
||
go run ./99-experiments/global-ratelimit/client -id inst-3 -rps 20
|
||
|
||
curl -s localhost:8100/stats
|
||
```
|
||
|
||
Observed (3 × 20 rps attempted, limit 10/s, 8 s):
|
||
|
||
```
|
||
inst-1 summary: attempts=159 allowed=31 denied=127 → work done=32
|
||
inst-2 summary: attempts=159 allowed=28 denied=130 → work done=29
|
||
inst-3 summary: attempts=160 allowed=29 denied=130 → work done=30
|
||
───
|
||
88 ≈ 10/s × 8 s + burst 10
|
||
inst-1 /allow latency: p50=0.6ms p99=11ms
|
||
```
|
||
|
||
The **sum** stays at the limit, not each instance — that's the whole point of
|
||
centralising it. Each replica gets roughly a third, with no coordination
|
||
between replicas and no per-replica configuration.
|
||
|
||
## API
|
||
|
||
```
|
||
POST|GET /allow?key=K[&n=1]
|
||
200 {"allowed":true,"remaining":9,"limit":10,"rate":10}
|
||
429 {"allowed":false,"remaining":0.3,"limit":10,"rate":10,"retry_after_ms":70}
|
||
headers: X-RateLimit-Limit, X-RateLimit-Remaining, Retry-After (429; whole seconds, rounded up)
|
||
400 missing key / bad n
|
||
GET /stats {"keys":2,"allowed_total":99,"denied_total":391}
|
||
GET /healthz ok
|
||
```
|
||
|
||
Server flags: `-addr`, `-rate`/`-burst` (defaults for unknown keys),
|
||
`-limits "tenant-a=100:200,tenant-b=5:5"` (per-key overrides), `-sweep 1m`.
|
||
|
||
Client flags: `-id`, `-limiter`, `-key`, `-rps`, `-duration`, `-timeout 200ms`,
|
||
`-fail open|closed`.
|
||
|
||
## What to observe
|
||
|
||
**The hop.** Every decision is an HTTP round trip: p50 ≈ 0.6 ms on loopback,
|
||
1–2 ms across an AZ. The in-process limiter is ~100 ns. That's the price of
|
||
"global", and why you only pay it at the edge (per-tenant quota), not in inner
|
||
loops.
|
||
|
||
**Fail-open vs fail-closed.** Start a client with each mode, then Ctrl-C the
|
||
server mid-run:
|
||
|
||
```
|
||
open allowed=0 denied=0 errors=10 → work done keeps going (49 of 59)
|
||
closed allowed=0 denied=0 errors=10 → work stops (18 of 59)
|
||
```
|
||
|
||
The limiter is now a dependency on the request path. Fail-open protects your
|
||
users and risks overloading what the limiter guards; fail-closed protects the
|
||
backend and turns a limiter outage into *your* outage. Pick deliberately, per
|
||
call site, and alert on the error rate either way. `-timeout` matters for the
|
||
same reason: a *slow* limiter must not make you slow.
|
||
|
||
**Idle eviction.** `-sweep` removes buckets untouched for that long. An idle
|
||
bucket is full anyway (once `idle ≥ burst/rate`), so recreating it later is
|
||
indistinguishable from keeping it — `last` doubles as the idle timestamp,
|
||
nothing extra to track. Without this, one bucket per API key ever seen lives
|
||
forever.
|
||
|
||
**Burst semantics across replicas.** The first second admits ~20 (burst 10 +
|
||
10/s). Replicas that start first eat the burst. If that's unfair for your use
|
||
case, lower `burst` — global buckets make the burst global too.
|
||
|
||
## What this is not
|
||
|
||
- **Not HA.** One process, in memory. Restart = all buckets full again
|
||
(a free burst for everyone) and, for the duration, fail-open or fail-closed
|
||
as configured. Real deployments put the bucket in Redis and run the
|
||
refill+consume as one Lua script so it's atomic — same arithmetic as
|
||
`limiter.Bucket.AllowN`, different storage.
|
||
- **Not accurate under Redis Cluster / multiple limiter nodes** unless keys are
|
||
routed consistently to one node. Keys → node is what makes a "global" bucket
|
||
global.
|
||
- **No local tier.** Production usually also keeps a coarse in-process limiter
|
||
(ex03) as a pre-filter, so a flooding client is cut off before it costs a
|
||
network hop per request, and caches "denied until T" locally.
|
||
|
||
## References
|
||
|
||
- Stripe, "Scaling your API with rate limiters" — the token bucket + Redis
|
||
pattern and why to shed load with it
|
||
- Envoy `ratelimit` service / `local_ratelimit` filter — the two-tier design
|
||
- `golang.org/x/time/rate` — the in-process reference implementation
|