learning-go-for-sre-interview/99-experiments/global-ratelimit

Experiment — a global rate limiter as a service

Exercise 03 built an in-process token bucket. That limits one process. When N replicas of a service must share one quota ("tenant-a gets 10 req/s, total"), the bucket has to live somewhere they can all reach. This is the smallest honest version of that: an HTTP service holding per-key buckets, and a demo client that plays one replica.

99-experiments/global-ratelimit/
  limiter/   token bucket (ex03) + sharded keyed store (ex09) + idle eviction   go test -race
  server/    HTTP service: /allow, /stats, /healthz
  client/    demo replica: asks /allow before each unit of work

Run it

# terminal 1 — the limiter
go run ./99-experiments/global-ratelimit/server -rate 10 -burst 10

# quick check
for i in $(seq 12); do curl -s -o /dev/null -w '%{http_code} ' 'localhost:8100/allow?key=t'; done
#   200 200 200 200 200 200 200 200 200 200 429 429
curl -si 'localhost:8100/allow?key=t' | grep -iE 'HTTP/|retry|ratelimit'

# terminals 2–4 — three replicas, each WANTS 20/s, all on key tenant-a
go run ./99-experiments/global-ratelimit/client -id inst-1 -rps 20
go run ./99-experiments/global-ratelimit/client -id inst-2 -rps 20
go run ./99-experiments/global-ratelimit/client -id inst-3 -rps 20

curl -s localhost:8100/stats

Observed (3 × 20 rps attempted, limit 10/s, 8 s):

inst-1 summary: attempts=159 allowed=31 denied=127 → work done=32
inst-2 summary: attempts=159 allowed=28 denied=130 → work done=29
inst-3 summary: attempts=160 allowed=29 denied=130 → work done=30
                                     ───
                                      88 ≈ 10/s × 8 s + burst 10
inst-1 /allow latency: p50=0.6ms p99=11ms

The sum stays at the limit, not each instance — that's the whole point of centralising it. Each replica gets roughly a third, with no coordination between replicas and no per-replica configuration.

API

POST|GET /allow?key=K[&n=1]
  200 {"allowed":true,"remaining":9,"limit":10,"rate":10}
  429 {"allowed":false,"remaining":0.3,"limit":10,"rate":10,"retry_after_ms":70}
  headers: X-RateLimit-Limit, X-RateLimit-Remaining, Retry-After (429; whole seconds, rounded up)
  400 missing key / bad n
GET /stats    {"keys":2,"allowed_total":99,"denied_total":391}
GET /healthz  ok

Server flags: -addr, -rate/-burst (defaults for unknown keys), -limits "tenant-a=100:200,tenant-b=5:5" (per-key overrides), -sweep 1m.

Client flags: -id, -limiter, -key, -rps, -duration, -timeout 200ms, -fail open|closed.

What to observe

The hop. Every decision is an HTTP round trip: p50 ≈ 0.6 ms on loopback, 1–2 ms across an AZ. The in-process limiter is ~100 ns. That's the price of "global", and why you only pay it at the edge (per-tenant quota), not in inner loops.

Fail-open vs fail-closed. Start a client with each mode, then Ctrl-C the server mid-run:

open    allowed=0 denied=0 errors=10   → work done keeps going (49 of 59)
closed  allowed=0 denied=0 errors=10   → work stops             (18 of 59)

The limiter is now a dependency on the request path. Fail-open protects your users and risks overloading what the limiter guards; fail-closed protects the backend and turns a limiter outage into your outage. Pick deliberately, per call site, and alert on the error rate either way. -timeout matters for the same reason: a slow limiter must not make you slow.

Idle eviction. -sweep removes buckets untouched for that long. An idle bucket is full anyway (once idle ≥ burst/rate), so recreating it later is indistinguishable from keeping it — last doubles as the idle timestamp, nothing extra to track. Without this, one bucket per API key ever seen lives forever.

Burst semantics across replicas. The first second admits ~20 (burst 10 + 10/s). Replicas that start first eat the burst. If that's unfair for your use case, lower burst — global buckets make the burst global too.

What this is not

• Not HA. One process, in memory. Restart = all buckets full again (a free burst for everyone) and, for the duration, fail-open or fail-closed as configured. Real deployments put the bucket in Redis and run the refill+consume as one Lua script so it's atomic — same arithmetic as limiter.Bucket.AllowN, different storage.
• Not accurate under Redis Cluster / multiple limiter nodes unless keys are routed consistently to one node. Keys → node is what makes a "global" bucket global.
• No local tier. Production usually also keeps a coarse in-process limiter (ex03) as a pre-filter, so a flooding client is cut off before it costs a network hop per request, and caches "denied until T" locally.

References

• Stripe, "Scaling your API with rate limiters" — the token bucket + Redis pattern and why to shed load with it
• Envoy ratelimit service / local_ratelimit filter — the two-tier design
• golang.org/x/time/rate — the in-process reference implementation