kacerrandClaude Fable 5
7219eb5a23
Includes solved 02-worker-pool and 09-safe-counter (mutex, atomic, sharded). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GbnJj9ZqEx9nUvzkjdwfMq
59 lines
2.7 KiB
Markdown
59 lines
2.7 KiB
Markdown
# 06 — Retry with Backoff + Jitter
|
||
|
||
Retries are the most dangerous reliability primitive: applied naively they
|
||
convert partial outages into total ones. Interviewers use this task to check
|
||
whether you've internalized that or just memorized the loop.
|
||
|
||
## Why jitter is non-negotiable (the thundering herd)
|
||
|
||
A dependency blips for 2 seconds. 10,000 clients fail *simultaneously*.
|
||
Without jitter they all retry at t+1s, t+3s, t+7s — as one synchronized
|
||
battering ram. The service recovers, gets hit by a perfectly-phased wave of
|
||
10k requests, dies again. Congratulations, your retry policy is now a DDoS
|
||
with extra steps — this is a **metastable failure**, and the system can stay
|
||
wedged even after the original trigger is gone.
|
||
|
||
The AWS Architecture Blog's classic analysis ("Exponential Backoff and
|
||
Jitter") compared strategies and **full jitter** — `sleep = rand(0, backoff)`
|
||
— came out best: maximal desynchronization, nearly minimal total work. That's
|
||
what you implement here. "Equal jitter" (`backoff/2 + rand(0, backoff/2)`) is
|
||
the runner-up if you want a floor.
|
||
|
||
## `time.Sleep` vs `select` — the test that catches everyone
|
||
|
||
```go
|
||
select {
|
||
case <-time.After(sleep):
|
||
case <-ctx.Done():
|
||
return ctx.Err()
|
||
}
|
||
```
|
||
|
||
`time.Sleep` is uninterruptible. In a shutdown, a worker mid-backoff on a
|
||
10-minute sleep holds your rolling deploy hostage until SIGKILL. Selecting on
|
||
`ctx.Done()` is what makes retries composable with graceful shutdown
|
||
(exercise 08). This single line is the difference between "knows Go" and
|
||
"has been paged."
|
||
|
||
## Error classification — retrying the unretryable
|
||
|
||
Retrying a 400 or a permission error is pure waste; retrying a *non-idempotent*
|
||
operation (POST /charge) is a career event. The pattern here uses a sentinel
|
||
(`errors.Is(err, ErrRetryable)`); production code often classifies by type
|
||
(`errors.As(&netErr)`), by status code, or via an `interface{ Temporary() bool }`.
|
||
Bring up **idempotency keys** as the real fix for retrying writes.
|
||
|
||
## The complete picture (say this at the whiteboard)
|
||
|
||
Retries alone are not a resilience strategy. The full stack:
|
||
|
||
1. **Cap the backoff** (`min(backoff, maxBackoff)`) — 2^20 seconds is not a plan.
|
||
2. **Retry budgets** (e.g., retries ≤ 10% of requests, à la the SRE book / Finagle)
|
||
beat per-request counts — they bound *aggregate* amplification.
|
||
3. **Circuit breaker** — stop calling a dead dependency entirely; probe
|
||
half-open. Backoff protects *them*, the breaker protects *you*.
|
||
4. **Deadline propagation** — retrying past the caller's deadline serves
|
||
responses nobody is waiting for. Notice the ctx check does this for free.
|
||
5. **Only retry at one layer.** Client retries × mesh retries × app retries =
|
||
exponential amplification. Pick an owner.
|