learning-go-for-sre-interview/06-retry-backoff/README.md

06 — Retry with Backoff + Jitter

Retries are the most dangerous reliability primitive: applied naively they convert partial outages into total ones. Interviewers use this task to check whether you've internalized that or just memorized the loop.

Why jitter is non-negotiable (the thundering herd)

A dependency blips for 2 seconds. 10,000 clients fail simultaneously. Without jitter they all retry at t+1s, t+3s, t+7s — as one synchronized battering ram. The service recovers, gets hit by a perfectly-phased wave of 10k requests, dies again. Congratulations, your retry policy is now a DDoS with extra steps — this is a metastable failure, and the system can stay wedged even after the original trigger is gone.

The AWS Architecture Blog's classic analysis ("Exponential Backoff and Jitter") compared strategies and full jitter — sleep = rand(0, backoff) — came out best: maximal desynchronization, nearly minimal total work. That's what you implement here. "Equal jitter" (backoff/2 + rand(0, backoff/2)) is the runner-up if you want a floor.

time.Sleep vs select — the test that catches everyone

select {
case <-time.After(sleep):
case <-ctx.Done():
    return ctx.Err()
}

time.Sleep is uninterruptible. In a shutdown, a worker mid-backoff on a 10-minute sleep holds your rolling deploy hostage until SIGKILL. Selecting on ctx.Done() is what makes retries composable with graceful shutdown (exercise 08). This single line is the difference between "knows Go" and "has been paged."

Error classification — retrying the unretryable

Retrying a 400 or a permission error is pure waste; retrying a non-idempotent operation (POST /charge) is a career event. The pattern here uses a sentinel (errors.Is(err, ErrRetryable)); production code often classifies by type (errors.As(&netErr)), by status code, or via an interface{ Temporary() bool }. Bring up idempotency keys as the real fix for retrying writes.

The complete picture (say this at the whiteboard)

Retries alone are not a resilience strategy. The full stack:

1. Cap the backoff (min(backoff, maxBackoff)) — 2^20 seconds is not a plan.
2. Retry budgets (e.g., retries ≤ 10% of requests, à la the SRE book / Finagle) beat per-request counts — they bound aggregate amplification.
3. Circuit breaker — stop calling a dead dependency entirely; probe half-open. Backoff protects them, the breaker protects you.
4. Deadline propagation — retrying past the caller's deadline serves responses nobody is waiting for. Notice the ctx check does this for free.
5. Only retry at one layer. Client retries × mesh retries × app retries = exponential amplification. Pick an owner.