learning-go-for-sre-interview/99-experiments/body-drain/README.md

# Experiment — not draining / closing HTTP response bodies

Companion to exercise 01, assignment point 5 ("reuse a single http.Client and
properly drain+close response bodies so connections are reused").

```
go run ./99-experiments/body-drain -mode all
go run ./99-experiments/body-drain -mode leak -n 200
```

## Concurrency: `-c` and `-idle-per-host`

`-c N` keeps up to N requests in flight (semaphore + WaitGroup, the same
shape as exercise 01). `-idle-per-host` sets `Transport.MaxIdleConnsPerHost`
(default 10; Go's own default is only **2**).

```
go run ./99-experiments/body-drain -mode drain -n 200 -c 8
#   httptrace GotConn  : reused=192 fresh=8      ← one conn per worker, then reused
#   server new conns   : 8

go run ./99-experiments/body-drain -mode drain -n 200 -c 8 -idle-per-host 2
#   server new conns   : 13+                     ← pool too small: conns returned
#                                                   while 2 are already idle get CLOSED
```

That second run is the trap in exercise 01: `limit` goroutines against one
host with the default transport means only 2 conns are ever kept, the rest are
dialed and thrown away on every round — even though you drained the body
correctly. On loopback the effect is small; against a TLS host each throwaway
is a full handshake. Raise `MaxIdleConnsPerHost` to your concurrency.

With `-c` the leak modes also exhaust fds N times faster, and failures no
longer stop in-flight requests: cancelling a request context closes its body
and would clean up the very leaks we want to see, so the client only stops
*launching* new requests after the first error.

## Two processes: watch each side run out of fds on its own

In one process the client and server share one fd limit and each leaked
request costs *two* fds (client socket + accepted server socket). The server
usually loses first — `accept` fails with `too many open files`, its accept
loop retries with backoff forever, and the client silently hangs waiting for
headers on a connection nobody will ever accept. Realistic, but confusing. So
split them:

```
# terminal 1
go run ./99-experiments/body-drain -role server

# terminal 2 — client is the one out of fds
ulimit -Hn 64; ulimit -Sn 64
go run ./99-experiments/body-drain -role client -mode leak -n 200
#   request 60 failed: dial tcp 127.0.0.1:8099: socket: too many open files
```

Now swap: limit the *server* terminal instead. It logs
`http: Accept error: accept tcp ...: too many open files; retrying in 5ms`,
while the client (unlimited) either hangs (Linux — pass `-header-timeout 3s`
to turn that into `timeout awaiting response headers`) or gets
`read: connection reset by peer` (macOS).

Which error string you see tells you which side is exhausted:

| error | who ran out |
|---|---|
| `dial tcp ...: socket: too many open files` | the client (this process) |
| `http: Accept error: accept ...: too many open files` | the server |
| `timeout awaiting response headers` / `connection reset by peer` | the client is fine; the *server* can't accept |

Note on `ulimit`: since Go 1.19 the runtime raises the soft `RLIMIT_NOFILE`
to the hard limit at startup, so lowering only the soft limit does nothing.
bash's plain `ulimit -n 64` sets both; zsh's sets only the soft one, hence
the explicit `-Hn`.

## What it does

A local `httptest.Server` returns a 1 MiB body and counts accepted TCP
connections via `Server.ConnState`. The client does N sequential GETs in one of
three modes, with a fresh `Transport` per mode:

| mode    | client does                                   |
|---------|-----------------------------------------------|
| `drain` | `io.Copy(io.Discard, body)` then `body.Close()` |
| `close` | `body.Close()` only                             |
| `leak`  | nothing                                         |

## What to watch

- **`server new conns`** — TCP connections the server accepted. Ideal: 1.
- **`httptrace GotConn reused/fresh`** — the client's own view: did `client.Do`
  pull a conn from the idle pool or dial a new one.
- **`goroutines delta`** — every connection the transport owns runs two
  goroutines (`readLoop` + `writeLoop`). Pooled conns show up here too, but
  they disappear after `CloseIdleConnections()`. Leaked ones don't.
- **`server active conns`** — conns the server still considers open after the
  client is "done".

## Typical output

```
== mode=drain ==
  httptrace GotConn  : reused=19 fresh=1
  goroutines         : before=2 after=5 (delta +3)
  after CloseIdleConnections: goroutines=2
  server new conns   : 1 (ideal: 1)
  server active conns: 0

== mode=close ==
  httptrace GotConn  : reused=0 fresh=20
  goroutines         : before=2 after=2
  server new conns   : 20 (ideal: 1)
  server active conns: 0

== mode=leak ==
  httptrace GotConn  : reused=0 fresh=20
  goroutines         : before=2 after=73 (delta +71)
  after CloseIdleConnections: goroutines=73        <- nothing reclaimed
  server new conns   : 20 (ideal: 1)
  server active conns: 20                          <- server stuck too
```

## Reading it

**`close` without drain is not a leak, but it defeats the pool.** The transport
sees unread bytes on the wire; reusing that TCP stream would corrupt the next
response, so it closes the conn. Every request pays DNS + TCP + TLS again. On
loopback that's microseconds; against a real HTTPS host it's 3–4 RTTs per
request and a TLS handshake the server has to do CPU work for. Correct output,
silently several times slower, and a `TIME_WAIT` pile-up on your host.

**`leak` leaks three things per request:** a file descriptor, two goroutines,
and the read buffer. `CloseIdleConnections` can't help because the conn is not
idle — the transport thinks a response is still in progress. Under any real
request rate this ends in `dial tcp: socket: too many open files`.

**The server suffers as well.** Its handler is blocked in `Write()` because the
client never reads and the kernel socket buffers are full. Without a
`WriteTimeout` those handler goroutines live until the client process dies.
This is why `httptest.Server.Close` hangs and logs "blocked in Close after 5
seconds" if you remove the `CloseClientConnections()` call — try it.

**Why doesn't a `Client.Timeout` hide this?** It does, partially: when the timer
fires, the transport force-closes the conn and the goroutines go away. So a
leaky client with a 5 s timeout leaks for 5 s per request instead of forever —
enough to exhaust fds at a few hundred req/s, and no connection reuse either
way. Add `Timeout: 5 * time.Second` to the client in `run()` and re-run `leak`
with a `time.Sleep(6 * time.Second)` before the goroutine count to see it.

## Rules that follow

```go
resp, err := client.Do(req)
if err != nil { return err }
defer resp.Body.Close()
io.Copy(io.Discard, resp.Body)   // or io.CopyN(..., 64<<10) if bodies may be huge
```

- Always `Close()`, always via `defer` right after the error check.
- Read to EOF if you want the conn back. If you don't need the body and it may
  be large, cap the drain — past some size a new handshake is cheaper than the
  download.
- Consider `HEAD` for health checks: nothing to drain.
- `Transport.MaxIdleConnsPerHost` defaults to 2; with `limit` concurrent
  checkers to one host, raise it to `limit` or the pool thrashes anyway.

## References

- `net/http` docs, `Client.Do`: "If the Body is not both read to EOF and closed,
  the Client's underlying RoundTripper may not be able to re-use a persistent
  TCP connection" — https://pkg.go.dev/net/http#Client.Do
- `net/http/httptrace` — https://pkg.go.dev/net/http/httptrace
- `http.Server.ConnState` — https://pkg.go.dev/net/http#Server.ConnState
- golang/go#4677 — the issue where this behaviour was settled.