kacerrandClaude Fable 5
7219eb5a23
Includes solved 02-worker-pool and 09-safe-counter (mutex, atomic, sharded). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GbnJj9ZqEx9nUvzkjdwfMq
67 lines
3.0 KiB
Markdown
67 lines
3.0 KiB
Markdown
# 08 — Graceful Shutdown
|
|
|
|
Pure SRE bread and butter: every rolling deploy, node drain, and spot
|
|
eviction ends with this code path. Interviewers love it because it chains
|
|
signals, contexts, and HTTP internals into one small program.
|
|
|
|
## The fresh-context trick (the whole exercise in one line)
|
|
|
|
```go
|
|
<-ctx.Done() // SIGTERM arrived; ctx is DEAD
|
|
shutCtx, cancel := context.WithTimeout(context.Background(), drain)
|
|
srv.Shutdown(shutCtx)
|
|
```
|
|
|
|
`Shutdown`'s context bounds *how long draining may take*. Pass the
|
|
already-cancelled signal ctx and Shutdown gives up instantly — you built
|
|
"graceful" shutdown that guillotines every in-flight request. It has to derive
|
|
from `Background()`. This inverted-context moment is the most commonly botched
|
|
line in the whole genre; the `TestDrainDeadlineExceeded`/`TestInFlight` pair
|
|
exists to catch both failure modes.
|
|
|
|
## `http.ErrServerClosed` is a success
|
|
|
|
`ListenAndServe` *always* returns non-nil. After `Shutdown` it returns
|
|
`http.ErrServerClosed`, which means "you asked me to stop and I did." Filter
|
|
it with `errors.Is`. Logging it as fatal is the "error: success" log line
|
|
every fleet has somewhere.
|
|
|
|
Also: send it through a **buffered** channel (`make(chan error, 1)`) — if you
|
|
exit `Run` via the ctx path first, an unbuffered send would strand the
|
|
ListenAndServe goroutine forever. Tiny buffer, whole leak class gone.
|
|
|
|
## What Shutdown actually does
|
|
|
|
1. Closes listeners → new connects get refused (RST).
|
|
2. Closes *idle* keep-alive connections.
|
|
3. Waits for active requests to finish, polling, until its ctx expires.
|
|
4. Returns `ctx.Err()` if the drain deadline passed with requests still going.
|
|
|
|
Note what it does **not** do: notify your load balancer. Which brings us to…
|
|
|
|
## The Kubernetes choreography (senior talking points)
|
|
|
|
SIGTERM and "stop receiving traffic" are **not ordered** the way you'd hope.
|
|
Kubelet sends SIGTERM *and* starts removing the pod from Endpoints
|
|
concurrently — traffic can arrive *after* SIGTERM for however long it takes
|
|
every kube-proxy and LB in the fleet to converge. Hence the cargo-cult that
|
|
is actually correct:
|
|
|
|
- **`preStop: sleep 5-15s`** — not superstition; it's buying time for
|
|
endpoint propagation *before* the process even sees SIGTERM.
|
|
- Fail readiness on SIGTERM, keep serving while draining.
|
|
- `terminationGracePeriodSeconds` (default 30s) must exceed
|
|
preStop + drainTimeout, or SIGKILL wins and no code of yours runs at all.
|
|
- If you're PID 1 in a container without an init, you get no default signal
|
|
handlers — `signal.NotifyContext` is doing real work there. (Related deep
|
|
cut: PID 1 doesn't reap zombies unless you make it — `tini` exists for a
|
|
reason.)
|
|
|
|
## `signal.NotifyContext` fine print
|
|
|
|
Second signal = default action (instant death). That's a *feature*: hung
|
|
drain + impatient operator's second Ctrl+C = immediate exit. Calling the
|
|
returned `stop()` after `ctx.Done()` restores default handling explicitly for
|
|
the same effect. Mentioning "first SIGTERM drains, second kills" is a nice
|
|
operational garnish.
|