# 08 — Graceful Shutdown

Pure SRE bread and butter: every rolling deploy, node drain, and spot
eviction ends with this code path. Interviewers love it because it chains
signals, contexts, and HTTP internals into one small program.

## The fresh-context trick (the whole exercise in one line)

```go
<-ctx.Done()                                   // SIGTERM arrived; ctx is DEAD
shutCtx, cancel := context.WithTimeout(context.Background(), drain)
srv.Shutdown(shutCtx)
```

`Shutdown`'s context bounds *how long draining may take*. Pass the
already-cancelled signal ctx and Shutdown gives up instantly — you built
"graceful" shutdown that guillotines every in-flight request. It has to derive
from `Background()`. This inverted-context moment is the most commonly botched
line in the whole genre; the `TestDrainDeadlineExceeded`/`TestInFlight` pair
exists to catch both failure modes.

## `http.ErrServerClosed` is a success

`ListenAndServe` *always* returns non-nil. After `Shutdown` it returns
`http.ErrServerClosed`, which means "you asked me to stop and I did." Filter
it with `errors.Is`. Logging it as fatal is the "error: success" log line
every fleet has somewhere.

Also: send it through a **buffered** channel (`make(chan error, 1)`) — if you
exit `Run` via the ctx path first, an unbuffered send would strand the
ListenAndServe goroutine forever. Tiny buffer, whole leak class gone.

## What Shutdown actually does

1. Closes listeners → new connects get refused (RST).
2. Closes *idle* keep-alive connections.
3. Waits for active requests to finish, polling, until its ctx expires.
4. Returns `ctx.Err()` if the drain deadline passed with requests still going.

Note what it does **not** do: notify your load balancer. Which brings us to…

## The Kubernetes choreography (senior talking points)

SIGTERM and "stop receiving traffic" are **not ordered** the way you'd hope.
Kubelet sends SIGTERM *and* starts removing the pod from Endpoints
concurrently — traffic can arrive *after* SIGTERM for however long it takes
every kube-proxy and LB in the fleet to converge. Hence the cargo-cult that
is actually correct:

- **`preStop: sleep 5-15s`** — not superstition; it's buying time for
  endpoint propagation *before* the process even sees SIGTERM.
- Fail readiness on SIGTERM, keep serving while draining.
- `terminationGracePeriodSeconds` (default 30s) must exceed
  preStop + drainTimeout, or SIGKILL wins and no code of yours runs at all.
- If you're PID 1 in a container without an init, you get no default signal
  handlers — `signal.NotifyContext` is doing real work there. (Related deep
  cut: PID 1 doesn't reap zombies unless you make it — `tini` exists for a
  reason.)

## `signal.NotifyContext` fine print

Second signal = default action (instant death). That's a *feature*: hung
drain + impatient operator's second Ctrl+C = immediate exit. Calling the
returned `stop()` after `ctx.Done()` restores default handling explicitly for
the same effect. Mentioning "first SIGTERM drains, second kills" is a nice
operational garnish.