kacerrandClaude Fable 5
9df64837dd
Local counting server + httptrace/goroutine metrics showing connection reuse (drain), pool bypass (close only) and fd/goroutine leak (no close). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GbnJj9ZqEx9nUvzkjdwfMq
110 lines
4.8 KiB
Markdown
110 lines
4.8 KiB
Markdown
# Experiment — not draining / closing HTTP response bodies
|
||
|
||
Companion to exercise 01, assignment point 5 ("reuse a single http.Client and
|
||
properly drain+close response bodies so connections are reused").
|
||
|
||
```
|
||
go run ./99-experiments/body-drain -mode all
|
||
go run ./99-experiments/body-drain -mode leak -n 200
|
||
ulimit -n 64 && go run ./99-experiments/body-drain -mode leak -n 200 # watch it die
|
||
```
|
||
|
||
## What it does
|
||
|
||
A local `httptest.Server` returns a 1 MiB body and counts accepted TCP
|
||
connections via `Server.ConnState`. The client does N sequential GETs in one of
|
||
three modes, with a fresh `Transport` per mode:
|
||
|
||
| mode | client does |
|
||
|---------|-----------------------------------------------|
|
||
| `drain` | `io.Copy(io.Discard, body)` then `body.Close()` |
|
||
| `close` | `body.Close()` only |
|
||
| `leak` | nothing |
|
||
|
||
## What to watch
|
||
|
||
- **`server new conns`** — TCP connections the server accepted. Ideal: 1.
|
||
- **`httptrace GotConn reused/fresh`** — the client's own view: did `client.Do`
|
||
pull a conn from the idle pool or dial a new one.
|
||
- **`goroutines delta`** — every connection the transport owns runs two
|
||
goroutines (`readLoop` + `writeLoop`). Pooled conns show up here too, but
|
||
they disappear after `CloseIdleConnections()`. Leaked ones don't.
|
||
- **`server active conns`** — conns the server still considers open after the
|
||
client is "done".
|
||
|
||
## Typical output
|
||
|
||
```
|
||
== mode=drain ==
|
||
httptrace GotConn : reused=19 fresh=1
|
||
goroutines : before=2 after=5 (delta +3)
|
||
after CloseIdleConnections: goroutines=2
|
||
server new conns : 1 (ideal: 1)
|
||
server active conns: 0
|
||
|
||
== mode=close ==
|
||
httptrace GotConn : reused=0 fresh=20
|
||
goroutines : before=2 after=2
|
||
server new conns : 20 (ideal: 1)
|
||
server active conns: 0
|
||
|
||
== mode=leak ==
|
||
httptrace GotConn : reused=0 fresh=20
|
||
goroutines : before=2 after=73 (delta +71)
|
||
after CloseIdleConnections: goroutines=73 <- nothing reclaimed
|
||
server new conns : 20 (ideal: 1)
|
||
server active conns: 20 <- server stuck too
|
||
```
|
||
|
||
## Reading it
|
||
|
||
**`close` without drain is not a leak, but it defeats the pool.** The transport
|
||
sees unread bytes on the wire; reusing that TCP stream would corrupt the next
|
||
response, so it closes the conn. Every request pays DNS + TCP + TLS again. On
|
||
loopback that's microseconds; against a real HTTPS host it's 3–4 RTTs per
|
||
request and a TLS handshake the server has to do CPU work for. Correct output,
|
||
silently several times slower, and a `TIME_WAIT` pile-up on your host.
|
||
|
||
**`leak` leaks three things per request:** a file descriptor, two goroutines,
|
||
and the read buffer. `CloseIdleConnections` can't help because the conn is not
|
||
idle — the transport thinks a response is still in progress. Under any real
|
||
request rate this ends in `dial tcp: socket: too many open files`.
|
||
|
||
**The server suffers as well.** Its handler is blocked in `Write()` because the
|
||
client never reads and the kernel socket buffers are full. Without a
|
||
`WriteTimeout` those handler goroutines live until the client process dies.
|
||
This is why `httptest.Server.Close` hangs and logs "blocked in Close after 5
|
||
seconds" if you remove the `CloseClientConnections()` call — try it.
|
||
|
||
**Why doesn't a `Client.Timeout` hide this?** It does, partially: when the timer
|
||
fires, the transport force-closes the conn and the goroutines go away. So a
|
||
leaky client with a 5 s timeout leaks for 5 s per request instead of forever —
|
||
enough to exhaust fds at a few hundred req/s, and no connection reuse either
|
||
way. Add `Timeout: 5 * time.Second` to the client in `run()` and re-run `leak`
|
||
with a `time.Sleep(6 * time.Second)` before the goroutine count to see it.
|
||
|
||
## Rules that follow
|
||
|
||
```go
|
||
resp, err := client.Do(req)
|
||
if err != nil { return err }
|
||
defer resp.Body.Close()
|
||
io.Copy(io.Discard, resp.Body) // or io.CopyN(..., 64<<10) if bodies may be huge
|
||
```
|
||
|
||
- Always `Close()`, always via `defer` right after the error check.
|
||
- Read to EOF if you want the conn back. If you don't need the body and it may
|
||
be large, cap the drain — past some size a new handshake is cheaper than the
|
||
download.
|
||
- Consider `HEAD` for health checks: nothing to drain.
|
||
- `Transport.MaxIdleConnsPerHost` defaults to 2; with `limit` concurrent
|
||
checkers to one host, raise it to `limit` or the pool thrashes anyway.
|
||
|
||
## References
|
||
|
||
- `net/http` docs, `Client.Do`: "If the Body is not both read to EOF and closed,
|
||
the Client's underlying RoundTripper may not be able to re-use a persistent
|
||
TCP connection" — https://pkg.go.dev/net/http#Client.Do
|
||
- `net/http/httptrace` — https://pkg.go.dev/net/http/httptrace
|
||
- `http.Server.ConnState` — https://pkg.go.dev/net/http#Server.ConnState
|
||
- golang/go#4677 — the issue where this behaviour was settled. |