learning-go-for-sre-interview/99-experiments/body-drain

Experiment — not draining / closing HTTP response bodies

Companion to exercise 01, assignment point 5 ("reuse a single http.Client and properly drain+close response bodies so connections are reused").

go run ./99-experiments/body-drain -mode all
go run ./99-experiments/body-drain -mode leak -n 200
ulimit -n 64 && go run ./99-experiments/body-drain -mode leak -n 200   # watch it die

What it does

A local httptest.Server returns a 1 MiB body and counts accepted TCP connections via Server.ConnState. The client does N sequential GETs in one of three modes, with a fresh Transport per mode:

mode	client does
drain	io.Copy(io.Discard, body) then body.Close()
close	body.Close() only
leak	nothing

What to watch

• server new conns — TCP connections the server accepted. Ideal: 1.
• httptrace GotConn reused/fresh — the client's own view: did client.Do pull a conn from the idle pool or dial a new one.
• goroutines delta — every connection the transport owns runs two goroutines (readLoop + writeLoop). Pooled conns show up here too, but they disappear after CloseIdleConnections(). Leaked ones don't.
• server active conns — conns the server still considers open after the client is "done".

Typical output

== mode=drain ==
  httptrace GotConn  : reused=19 fresh=1
  goroutines         : before=2 after=5 (delta +3)
  after CloseIdleConnections: goroutines=2
  server new conns   : 1 (ideal: 1)
  server active conns: 0

== mode=close ==
  httptrace GotConn  : reused=0 fresh=20
  goroutines         : before=2 after=2
  server new conns   : 20 (ideal: 1)
  server active conns: 0

== mode=leak ==
  httptrace GotConn  : reused=0 fresh=20
  goroutines         : before=2 after=73 (delta +71)
  after CloseIdleConnections: goroutines=73        <- nothing reclaimed
  server new conns   : 20 (ideal: 1)
  server active conns: 20                          <- server stuck too

Reading it

close without drain is not a leak, but it defeats the pool. The transport sees unread bytes on the wire; reusing that TCP stream would corrupt the next response, so it closes the conn. Every request pays DNS + TCP + TLS again. On loopback that's microseconds; against a real HTTPS host it's 3–4 RTTs per request and a TLS handshake the server has to do CPU work for. Correct output, silently several times slower, and a TIME_WAIT pile-up on your host.

leak leaks three things per request: a file descriptor, two goroutines, and the read buffer. CloseIdleConnections can't help because the conn is not idle — the transport thinks a response is still in progress. Under any real request rate this ends in dial tcp: socket: too many open files.

The server suffers as well. Its handler is blocked in Write() because the client never reads and the kernel socket buffers are full. Without a WriteTimeout those handler goroutines live until the client process dies. This is why httptest.Server.Close hangs and logs "blocked in Close after 5 seconds" if you remove the CloseClientConnections() call — try it.

Why doesn't a Client.Timeout hide this? It does, partially: when the timer fires, the transport force-closes the conn and the goroutines go away. So a leaky client with a 5 s timeout leaks for 5 s per request instead of forever — enough to exhaust fds at a few hundred req/s, and no connection reuse either way. Add Timeout: 5 * time.Second to the client in run() and re-run leak with a time.Sleep(6 * time.Second) before the goroutine count to see it.

Rules that follow

resp, err := client.Do(req)
if err != nil { return err }
defer resp.Body.Close()
io.Copy(io.Discard, resp.Body)   // or io.CopyN(..., 64<<10) if bodies may be huge

• Always Close(), always via defer right after the error check.
• Read to EOF if you want the conn back. If you don't need the body and it may be large, cap the drain — past some size a new handshake is cheaper than the download.
• Consider HEAD for health checks: nothing to drain.
• Transport.MaxIdleConnsPerHost defaults to 2; with limit concurrent checkers to one host, raise it to limit or the pool thrashes anyway.

References

• net/http docs, Client.Do: "If the Body is not both read to EOF and closed, the Client's underlying RoundTripper may not be able to re-use a persistent TCP connection" — https://pkg.go.dev/net/http#Client.Do
• net/http/httptrace — https://pkg.go.dev/net/http/httptrace
• http.Server.ConnState — https://pkg.go.dev/net/http#Server.ConnState
• golang/go#4677 — the issue where this behaviour was settled.