Initial commit: Go SRE interview exercises
Includes solved 02-worker-pool and 09-safe-counter (mutex, atomic, sharded). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GbnJj9ZqEx9nUvzkjdwfMq
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
# 08 — Graceful Shutdown
|
||||
|
||||
Pure SRE bread and butter: every rolling deploy, node drain, and spot
|
||||
eviction ends with this code path. Interviewers love it because it chains
|
||||
signals, contexts, and HTTP internals into one small program.
|
||||
|
||||
## The fresh-context trick (the whole exercise in one line)
|
||||
|
||||
```go
|
||||
<-ctx.Done() // SIGTERM arrived; ctx is DEAD
|
||||
shutCtx, cancel := context.WithTimeout(context.Background(), drain)
|
||||
srv.Shutdown(shutCtx)
|
||||
```
|
||||
|
||||
`Shutdown`'s context bounds *how long draining may take*. Pass the
|
||||
already-cancelled signal ctx and Shutdown gives up instantly — you built
|
||||
"graceful" shutdown that guillotines every in-flight request. It has to derive
|
||||
from `Background()`. This inverted-context moment is the most commonly botched
|
||||
line in the whole genre; the `TestDrainDeadlineExceeded`/`TestInFlight` pair
|
||||
exists to catch both failure modes.
|
||||
|
||||
## `http.ErrServerClosed` is a success
|
||||
|
||||
`ListenAndServe` *always* returns non-nil. After `Shutdown` it returns
|
||||
`http.ErrServerClosed`, which means "you asked me to stop and I did." Filter
|
||||
it with `errors.Is`. Logging it as fatal is the "error: success" log line
|
||||
every fleet has somewhere.
|
||||
|
||||
Also: send it through a **buffered** channel (`make(chan error, 1)`) — if you
|
||||
exit `Run` via the ctx path first, an unbuffered send would strand the
|
||||
ListenAndServe goroutine forever. Tiny buffer, whole leak class gone.
|
||||
|
||||
## What Shutdown actually does
|
||||
|
||||
1. Closes listeners → new connects get refused (RST).
|
||||
2. Closes *idle* keep-alive connections.
|
||||
3. Waits for active requests to finish, polling, until its ctx expires.
|
||||
4. Returns `ctx.Err()` if the drain deadline passed with requests still going.
|
||||
|
||||
Note what it does **not** do: notify your load balancer. Which brings us to…
|
||||
|
||||
## The Kubernetes choreography (senior talking points)
|
||||
|
||||
SIGTERM and "stop receiving traffic" are **not ordered** the way you'd hope.
|
||||
Kubelet sends SIGTERM *and* starts removing the pod from Endpoints
|
||||
concurrently — traffic can arrive *after* SIGTERM for however long it takes
|
||||
every kube-proxy and LB in the fleet to converge. Hence the cargo-cult that
|
||||
is actually correct:
|
||||
|
||||
- **`preStop: sleep 5-15s`** — not superstition; it's buying time for
|
||||
endpoint propagation *before* the process even sees SIGTERM.
|
||||
- Fail readiness on SIGTERM, keep serving while draining.
|
||||
- `terminationGracePeriodSeconds` (default 30s) must exceed
|
||||
preStop + drainTimeout, or SIGKILL wins and no code of yours runs at all.
|
||||
- If you're PID 1 in a container without an init, you get no default signal
|
||||
handlers — `signal.NotifyContext` is doing real work there. (Related deep
|
||||
cut: PID 1 doesn't reap zombies unless you make it — `tini` exists for a
|
||||
reason.)
|
||||
|
||||
## `signal.NotifyContext` fine print
|
||||
|
||||
Second signal = default action (instant death). That's a *feature*: hung
|
||||
drain + impatient operator's second Ctrl+C = immediate exit. Calling the
|
||||
returned `stop()` after `ctx.Done()` restores default handling explicitly for
|
||||
the same effect. Mentioning "first SIGTERM drains, second kills" is a nice
|
||||
operational garnish.
|
||||
Reference in New Issue
Block a user