Add global-ratelimit experiment: token-bucket limiter as an HTTP service

limiter/ package (ex03 bucket + ex09 sharded keyed store + idle sweep,
race-tested), server/ with /allow /stats /healthz and X-RateLimit headers,
client/ demo replica with fail-open/closed and latency percentiles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GbnJj9ZqEx9nUvzkjdwfMq

99-experiments/global-ratelimit/README.md

@@ -0,0 +1,118 @@
+# Experiment — a global rate limiter as a service
+
+Exercise 03 built an in-process token bucket. That limits **one process**.
+When N replicas of a service must share one quota ("tenant-a gets 10 req/s,
+total"), the bucket has to live somewhere they can all reach. This is the
+smallest honest version of that: an HTTP service holding per-key buckets, and
+a demo client that plays one replica.
+
+```
+99-experiments/global-ratelimit/
+  limiter/   token bucket (ex03) + sharded keyed store (ex09) + idle eviction   go test -race
+  server/    HTTP service: /allow, /stats, /healthz
+  client/    demo replica: asks /allow before each unit of work
+```
+
+## Run it
+
+```
+# terminal 1 — the limiter
+go run ./99-experiments/global-ratelimit/server -rate 10 -burst 10
+
+# quick check
+for i in $(seq 12); do curl -s -o /dev/null -w '%{http_code} ' 'localhost:8100/allow?key=t'; done
+#   200 200 200 200 200 200 200 200 200 200 429 429
+curl -si 'localhost:8100/allow?key=t' | grep -iE 'HTTP/|retry|ratelimit'
+
+# terminals 2–4 — three replicas, each WANTS 20/s, all on key tenant-a
+go run ./99-experiments/global-ratelimit/client -id inst-1 -rps 20
+go run ./99-experiments/global-ratelimit/client -id inst-2 -rps 20
+go run ./99-experiments/global-ratelimit/client -id inst-3 -rps 20
+
+curl -s localhost:8100/stats
+```
+
+Observed (3 × 20 rps attempted, limit 10/s, 8 s):
+
+```
+inst-1 summary: attempts=159 allowed=31 denied=127 → work done=32
+inst-2 summary: attempts=159 allowed=28 denied=130 → work done=29
+inst-3 summary: attempts=160 allowed=29 denied=130 → work done=30
+                                     ───
+                                      88 ≈ 10/s × 8 s + burst 10
+inst-1 /allow latency: p50=0.6ms p99=11ms
+```
+
+The **sum** stays at the limit, not each instance — that's the whole point of
+centralising it. Each replica gets roughly a third, with no coordination
+between replicas and no per-replica configuration.
+
+## API
+
+```
+POST|GET /allow?key=K[&n=1]
+  200 {"allowed":true,"remaining":9,"limit":10,"rate":10}
+  429 {"allowed":false,"remaining":0.3,"limit":10,"rate":10,"retry_after_ms":70}
+  headers: X-RateLimit-Limit, X-RateLimit-Remaining, Retry-After (429; whole seconds, rounded up)
+  400 missing key / bad n
+GET /stats    {"keys":2,"allowed_total":99,"denied_total":391}
+GET /healthz  ok
+```
+
+Server flags: `-addr`, `-rate`/`-burst` (defaults for unknown keys),
+`-limits "tenant-a=100:200,tenant-b=5:5"` (per-key overrides), `-sweep 1m`.
+
+Client flags: `-id`, `-limiter`, `-key`, `-rps`, `-duration`, `-timeout 200ms`,
+`-fail open|closed`.
+
+## What to observe
+
+**The hop.** Every decision is an HTTP round trip: p50 ≈ 0.6 ms on loopback,
+1–2 ms across an AZ. The in-process limiter is ~100 ns. That's the price of
+"global", and why you only pay it at the edge (per-tenant quota), not in inner
+loops.
+
+**Fail-open vs fail-closed.** Start a client with each mode, then Ctrl-C the
+server mid-run:
+
+```
+open    allowed=0 denied=0 errors=10   → work done keeps going (49 of 59)
+closed  allowed=0 denied=0 errors=10   → work stops             (18 of 59)
+```
+
+The limiter is now a dependency on the request path. Fail-open protects your
+users and risks overloading what the limiter guards; fail-closed protects the
+backend and turns a limiter outage into *your* outage. Pick deliberately, per
+call site, and alert on the error rate either way. `-timeout` matters for the
+same reason: a *slow* limiter must not make you slow.
+
+**Idle eviction.** `-sweep` removes buckets untouched for that long. An idle
+bucket is full anyway (once `idle ≥ burst/rate`), so recreating it later is
+indistinguishable from keeping it — `last` doubles as the idle timestamp,
+nothing extra to track. Without this, one bucket per API key ever seen lives
+forever.
+
+**Burst semantics across replicas.** The first second admits ~20 (burst 10 +
+10/s). Replicas that start first eat the burst. If that's unfair for your use
+case, lower `burst` — global buckets make the burst global too.
+
+## What this is not
+
+- **Not HA.** One process, in memory. Restart = all buckets full again
+  (a free burst for everyone) and, for the duration, fail-open or fail-closed
+  as configured. Real deployments put the bucket in Redis and run the
+  refill+consume as one Lua script so it's atomic — same arithmetic as
+  `limiter.Bucket.AllowN`, different storage.
+- **Not accurate under Redis Cluster / multiple limiter nodes** unless keys are
+  routed consistently to one node. Keys → node is what makes a "global" bucket
+  global.
+- **No local tier.** Production usually also keeps a coarse in-process limiter
+  (ex03) as a pre-filter, so a flooding client is cut off before it costs a
+  network hop per request, and caches "denied until T" locally.
+
+## References
+
+- Stripe, "Scaling your API with rate limiters" — the token bucket + Redis
+  pattern and why to shed load with it
+- Envoy `ratelimit` service / `local_ratelimit` filter — the two-tier design
+- `golang.org/x/time/rate` — the in-process reference implementation