learning-go-for-sre-interview/99-experiments/body-drain

Experiment — not draining / closing HTTP response bodies

Companion to exercise 01, assignment point 5 ("reuse a single http.Client and properly drain+close response bodies so connections are reused").

go run ./99-experiments/body-drain -mode all
go run ./99-experiments/body-drain -mode leak -n 200

Concurrency: -c and -idle-per-host

-c N keeps up to N requests in flight (semaphore + WaitGroup, the same shape as exercise 01). -idle-per-host sets Transport.MaxIdleConnsPerHost (default 10; Go's own default is only 2).

go run ./99-experiments/body-drain -mode drain -n 200 -c 8
#   httptrace GotConn  : reused=192 fresh=8      ← one conn per worker, then reused
#   server new conns   : 8

go run ./99-experiments/body-drain -mode drain -n 200 -c 8 -idle-per-host 2
#   server new conns   : 13+                     ← pool too small: conns returned
#                                                   while 2 are already idle get CLOSED

That second run is the trap in exercise 01: limit goroutines against one host with the default transport means only 2 conns are ever kept, the rest are dialed and thrown away on every round — even though you drained the body correctly. On loopback the effect is small; against a TLS host each throwaway is a full handshake. Raise MaxIdleConnsPerHost to your concurrency.

With -c the leak modes also exhaust fds N times faster, and failures no longer stop in-flight requests: cancelling a request context closes its body and would clean up the very leaks we want to see, so the client only stops launching new requests after the first error.

Two processes: watch each side run out of fds on its own

In one process the client and server share one fd limit and each leaked request costs two fds (client socket + accepted server socket). The server usually loses first — accept fails with too many open files, its accept loop retries with backoff forever, and the client silently hangs waiting for headers on a connection nobody will ever accept. Realistic, but confusing. So split them:

# terminal 1
go run ./99-experiments/body-drain -role server

# terminal 2 — client is the one out of fds
ulimit -Hn 64; ulimit -Sn 64
go run ./99-experiments/body-drain -role client -mode leak -n 200
#   request 60 failed: dial tcp 127.0.0.1:8099: socket: too many open files

Now swap: limit the server terminal instead. It logs http: Accept error: accept tcp ...: too many open files; retrying in 5ms, while the client (unlimited) either hangs (Linux — pass -header-timeout 3s to turn that into timeout awaiting response headers) or gets read: connection reset by peer (macOS).

Which error string you see tells you which side is exhausted:

error	who ran out
dial tcp ...: socket: too many open files	the client (this process)
http: Accept error: accept ...: too many open files	the server
timeout awaiting response headers / connection reset by peer	the client is fine; the server can't accept

Note on ulimit: since Go 1.19 the runtime raises the soft RLIMIT_NOFILE to the hard limit at startup, so lowering only the soft limit does nothing. bash's plain ulimit -n 64 sets both; zsh's sets only the soft one, hence the explicit -Hn.

What it does

A local httptest.Server returns a 1 MiB body and counts accepted TCP connections via Server.ConnState. The client does N sequential GETs in one of three modes, with a fresh Transport per mode:

mode	client does
drain	io.Copy(io.Discard, body) then body.Close()
close	body.Close() only
leak	nothing

What to watch

• server new conns — TCP connections the server accepted. Ideal: 1.
• httptrace GotConn reused/fresh — the client's own view: did client.Do pull a conn from the idle pool or dial a new one.
• goroutines delta — every connection the transport owns runs two goroutines (readLoop + writeLoop). Pooled conns show up here too, but they disappear after CloseIdleConnections(). Leaked ones don't.
• server active conns — conns the server still considers open after the client is "done".

Typical output

== mode=drain ==
  httptrace GotConn  : reused=19 fresh=1
  goroutines         : before=2 after=5 (delta +3)
  after CloseIdleConnections: goroutines=2
  server new conns   : 1 (ideal: 1)
  server active conns: 0

== mode=close ==
  httptrace GotConn  : reused=0 fresh=20
  goroutines         : before=2 after=2
  server new conns   : 20 (ideal: 1)
  server active conns: 0

== mode=leak ==
  httptrace GotConn  : reused=0 fresh=20
  goroutines         : before=2 after=73 (delta +71)
  after CloseIdleConnections: goroutines=73        <- nothing reclaimed
  server new conns   : 20 (ideal: 1)
  server active conns: 20                          <- server stuck too

Reading it

close without drain is not a leak, but it defeats the pool. The transport sees unread bytes on the wire; reusing that TCP stream would corrupt the next response, so it closes the conn. Every request pays DNS + TCP + TLS again. On loopback that's microseconds; against a real HTTPS host it's 3–4 RTTs per request and a TLS handshake the server has to do CPU work for. Correct output, silently several times slower, and a TIME_WAIT pile-up on your host.

leak leaks three things per request: a file descriptor, two goroutines, and the read buffer. CloseIdleConnections can't help because the conn is not idle — the transport thinks a response is still in progress. Under any real request rate this ends in dial tcp: socket: too many open files.

The server suffers as well. Its handler is blocked in Write() because the client never reads and the kernel socket buffers are full. Without a WriteTimeout those handler goroutines live until the client process dies. This is why httptest.Server.Close hangs and logs "blocked in Close after 5 seconds" if you remove the CloseClientConnections() call — try it.

Why doesn't a Client.Timeout hide this? It does, partially: when the timer fires, the transport force-closes the conn and the goroutines go away. So a leaky client with a 5 s timeout leaks for 5 s per request instead of forever — enough to exhaust fds at a few hundred req/s, and no connection reuse either way. Add Timeout: 5 * time.Second to the client in run() and re-run leak with a time.Sleep(6 * time.Second) before the goroutine count to see it.

Rules that follow

resp, err := client.Do(req)
if err != nil { return err }
defer resp.Body.Close()
io.Copy(io.Discard, resp.Body)   // or io.CopyN(..., 64<<10) if bodies may be huge

• Always Close(), always via defer right after the error check.
• Read to EOF if you want the conn back. If you don't need the body and it may be large, cap the drain — past some size a new handshake is cheaper than the download.
• Consider HEAD for health checks: nothing to drain.
• Transport.MaxIdleConnsPerHost defaults to 2; with limit concurrent checkers to one host, raise it to limit or the pool thrashes anyway.

References

• net/http docs, Client.Do: "If the Body is not both read to EOF and closed, the Client's underlying RoundTripper may not be able to re-use a persistent TCP connection" — https://pkg.go.dev/net/http#Client.Do
• net/http/httptrace — https://pkg.go.dev/net/http/httptrace
• http.Server.ConnState — https://pkg.go.dev/net/http#Server.ConnState
• golang/go#4677 — the issue where this behaviour was settled.