learning-go-for-sre-interview/08-graceful-shutdown

08 — Graceful Shutdown

Pure SRE bread and butter: every rolling deploy, node drain, and spot eviction ends with this code path. Interviewers love it because it chains signals, contexts, and HTTP internals into one small program.

The fresh-context trick (the whole exercise in one line)

<-ctx.Done()                                   // SIGTERM arrived; ctx is DEAD
shutCtx, cancel := context.WithTimeout(context.Background(), drain)
srv.Shutdown(shutCtx)

Shutdown's context bounds how long draining may take. Pass the already-cancelled signal ctx and Shutdown gives up instantly — you built "graceful" shutdown that guillotines every in-flight request. It has to derive from Background(). This inverted-context moment is the most commonly botched line in the whole genre; the TestDrainDeadlineExceeded/TestInFlight pair exists to catch both failure modes.

http.ErrServerClosed is a success

ListenAndServe always returns non-nil. After Shutdown it returns http.ErrServerClosed, which means "you asked me to stop and I did." Filter it with errors.Is. Logging it as fatal is the "error: success" log line every fleet has somewhere.

Also: send it through a buffered channel (make(chan error, 1)) — if you exit Run via the ctx path first, an unbuffered send would strand the ListenAndServe goroutine forever. Tiny buffer, whole leak class gone.

What Shutdown actually does

1. Closes listeners → new connects get refused (RST).
2. Closes idle keep-alive connections.
3. Waits for active requests to finish, polling, until its ctx expires.
4. Returns ctx.Err() if the drain deadline passed with requests still going.

Note what it does not do: notify your load balancer. Which brings us to…

The Kubernetes choreography (senior talking points)

SIGTERM and "stop receiving traffic" are not ordered the way you'd hope. Kubelet sends SIGTERM and starts removing the pod from Endpoints concurrently — traffic can arrive after SIGTERM for however long it takes every kube-proxy and LB in the fleet to converge. Hence the cargo-cult that is actually correct:

• preStop: sleep 5-15s — not superstition; it's buying time for endpoint propagation before the process even sees SIGTERM.
• Fail readiness on SIGTERM, keep serving while draining.
• terminationGracePeriodSeconds (default 30s) must exceed preStop + drainTimeout, or SIGKILL wins and no code of yours runs at all.
• If you're PID 1 in a container without an init, you get no default signal handlers — signal.NotifyContext is doing real work there. (Related deep cut: PID 1 doesn't reap zombies unless you make it — tini exists for a reason.)

signal.NotifyContext fine print

Second signal = default action (instant death). That's a feature: hung drain + impatient operator's second Ctrl+C = immediate exit. Calling the returned stop() after ctx.Done() restores default handling explicitly for the same effect. Mentioning "first SIGTERM drains, second kills" is a nice operational garnish.