Initial commit: Go SRE interview exercises
Includes solved 02-worker-pool and 09-safe-counter (mutex, atomic, sharded). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GbnJj9ZqEx9nUvzkjdwfMq
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
# 06 — Retry with Backoff + Jitter
|
||||
|
||||
Retries are the most dangerous reliability primitive: applied naively they
|
||||
convert partial outages into total ones. Interviewers use this task to check
|
||||
whether you've internalized that or just memorized the loop.
|
||||
|
||||
## Why jitter is non-negotiable (the thundering herd)
|
||||
|
||||
A dependency blips for 2 seconds. 10,000 clients fail *simultaneously*.
|
||||
Without jitter they all retry at t+1s, t+3s, t+7s — as one synchronized
|
||||
battering ram. The service recovers, gets hit by a perfectly-phased wave of
|
||||
10k requests, dies again. Congratulations, your retry policy is now a DDoS
|
||||
with extra steps — this is a **metastable failure**, and the system can stay
|
||||
wedged even after the original trigger is gone.
|
||||
|
||||
The AWS Architecture Blog's classic analysis ("Exponential Backoff and
|
||||
Jitter") compared strategies and **full jitter** — `sleep = rand(0, backoff)`
|
||||
— came out best: maximal desynchronization, nearly minimal total work. That's
|
||||
what you implement here. "Equal jitter" (`backoff/2 + rand(0, backoff/2)`) is
|
||||
the runner-up if you want a floor.
|
||||
|
||||
## `time.Sleep` vs `select` — the test that catches everyone
|
||||
|
||||
```go
|
||||
select {
|
||||
case <-time.After(sleep):
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
}
|
||||
```
|
||||
|
||||
`time.Sleep` is uninterruptible. In a shutdown, a worker mid-backoff on a
|
||||
10-minute sleep holds your rolling deploy hostage until SIGKILL. Selecting on
|
||||
`ctx.Done()` is what makes retries composable with graceful shutdown
|
||||
(exercise 08). This single line is the difference between "knows Go" and
|
||||
"has been paged."
|
||||
|
||||
## Error classification — retrying the unretryable
|
||||
|
||||
Retrying a 400 or a permission error is pure waste; retrying a *non-idempotent*
|
||||
operation (POST /charge) is a career event. The pattern here uses a sentinel
|
||||
(`errors.Is(err, ErrRetryable)`); production code often classifies by type
|
||||
(`errors.As(&netErr)`), by status code, or via an `interface{ Temporary() bool }`.
|
||||
Bring up **idempotency keys** as the real fix for retrying writes.
|
||||
|
||||
## The complete picture (say this at the whiteboard)
|
||||
|
||||
Retries alone are not a resilience strategy. The full stack:
|
||||
|
||||
1. **Cap the backoff** (`min(backoff, maxBackoff)`) — 2^20 seconds is not a plan.
|
||||
2. **Retry budgets** (e.g., retries ≤ 10% of requests, à la the SRE book / Finagle)
|
||||
beat per-request counts — they bound *aggregate* amplification.
|
||||
3. **Circuit breaker** — stop calling a dead dependency entirely; probe
|
||||
half-open. Backoff protects *them*, the breaker protects *you*.
|
||||
4. **Deadline propagation** — retrying past the caller's deadline serves
|
||||
responses nobody is waiting for. Notice the ctx check does this for free.
|
||||
5. **Only retry at one layer.** Client retries × mesh retries × app retries =
|
||||
exponential amplification. Pick an owner.
|
||||
Reference in New Issue
Block a user