learning-go-for-sre-interview/98-http-server-experiments/README.md

# 98 — HTTP server connection experiments

How many connections can a Go HTTP server hold, and what gives out first? This
folder finds the **order of magnitude** (not a portable constant) for four
different failure modes, and — just as important — teaches you to tell a real
server ceiling from your own load generator running out first.

One simple server is the subject under test; one load client drives it in four
modes. Run them in two terminals so each side has its own fd budget.

```
98-http-server-experiments/
  server/main.go     subject under test: /static /compute /sleep /stats, ConnState + in_flight counters
  client/main.go     load generator: -mode throughput | idle | slow | slowloris
  loadlib/           tested core: Burn (CPU), Percentile, Classify (which side ran out)
  README.md          this file — decisions log + pasted evidence
```

Pure stdlib, zero dependencies. Part of the `gosre-exercises` module, so:

```sh
go run ./98-http-server-experiments/server        # terminal 1
go run ./98-http-server-experiments/client -mode idle -n 2000   # terminal 2
go test -race ./98-http-server-experiments/loadlib/
```

## Run it

**Terminal 1 — the server.** Give it its own fd budget so *you* decide when it
runs out, not the OS default. bash sets both soft and hard limits:

```sh
ulimit -Hn 4096; ulimit -Sn 4096
go run ./98-http-server-experiments/server                       # naive: no timeouts
go run ./98-http-server-experiments/server -read-header-timeout 3s -idle-timeout 30s   # hardened
```

(Since Go 1.19 the runtime raises the soft `RLIMIT_NOFILE` to the hard limit at
startup, so only the **hard** limit really caps you. bash's `ulimit -n` sets
both; zsh's sets only the soft one — use `-Hn`.)

The server prints one line per second:

```
19:12:20  accepted_total=10001 active=10001 in_flight=1 goroutines=10006 heap=107MiB
```

Three numbers that are easy to conflate:

| number | meaning | via |
|---|---|---|
| `accepted_total` | every TCP conn ever accepted (only grows) | `ConnState` StateNew |
| `active` | conns open **right now** — idle keep-alive *or* busy | `ConnState` new − closed |
| `in_flight` | handlers **actually running** right now | middleware around the mux |

An idle keep-alive counts in `active` but not `in_flight`; a running `/sleep`
counts in both. That gap is the whole point of the idle-vs-slow contrast.

**Terminal 2 — the client.** Give it *its own* fd budget, separate from the server:

```sh
ulimit -Hn 64; ulimit -Sn 64
go run ./98-http-server-experiments/client -mode idle -n 200
```

## The four dimensions

### 1. Idle connection ceiling — `-mode idle`

Open `-n` keep-alive conns and hold them idle. Uses **raw `net.Conn`**, not an
`http.Client` — with a fast endpoint the client's connection pool reuses conns
faster than you dial, so N concurrent GETs do *not* reliably produce N
simultaneous conns. Dialing raw and keeping each fd open guarantees exactly N.

```sh
go run ./98-http-server-experiments/client -mode idle -n 10000 -duration 30s
```

```
== mode=idle target=http://127.0.0.1:8098 ==
  server before      : active=1 in_flight=1 goroutines=6 heap=0MiB
  opened             : 10000/10000 idle conns held, 0 failed
  client goroutines  : before=3 after=3 (delta +0; raw fds, no per-conn goroutine)
  server after open  : active=10001 in_flight=1 goroutines=10006 heap=107MiB  ← active≈n, in_flight≈0
```

**Reading it:** 10 000 idle conns cost the server ~107 MiB (from a <1 MiB
baseline) and one goroutine each → **order of magnitude ~10 KiB per idle
connection**. A million idle keep-alives would be ~10 GiB and a million
goroutines — the server holds them fine until it runs out of fds or RAM, not CPU.
Note the client's goroutine delta is **0**: raw fds have no per-conn goroutine,
unlike `http.Transport`'s pool (2 goroutines per pooled conn).

### 2. Active slow requests — `-mode slow`

Fire `-c` concurrent `/sleep` requests and hold them *all in flight* for the
duration. Same conn count as idle, but now each pins a **running** server
goroutine. Prints the server's `in_flight` each second.

```sh
go run ./98-http-server-experiments/client -mode slow -c 500 -duration 10s
```

```
== mode=slow target=http://127.0.0.1:8098 ==
  firing 500 concurrent /sleep?d=10s ...
    server 01:13:13  active=501 in_flight=501 goroutines=1006 heap=25MiB
  requests           : 500 ok, 0 failed (of 500 held in flight)
  errors             : none
```

**Reading it:** `in_flight=501` (the 500 sleepers + the stats poll) vs idle's
`in_flight=1` — that is the difference between "conn open" and "handler running."
Each in-flight request is one goroutine on a goroutine-per-request server; 500
held costs ~1000 goroutines and a few MiB. Push `-c` up in orders of magnitude to
find where the *server* (goroutine stacks, fds) — not the client — gives out.
`/sleep` honors `r.Context()`, so a client that hangs up or a server timeout
frees the goroutine immediately rather than sleeping to term.

### 3. Timeouts & slowloris — `-mode slowloris`

Open `-c` raw TCP conns, send a request line + one header on each, then dribble a
bogus header every `-drip` and **never** send the terminating blank line. The
handler never starts. The only thing that reclaims these conns is the server's
`ReadHeaderTimeout`.

**Naive server** (`go run .../server`, no timeouts):

```
== mode=slowloris target=http://127.0.0.1:8098 ==
  opening 200 slowloris conns ...
    server 01:13:26  active=201 in_flight=1 goroutines=206 heap=26MiB   ← stuck, handler never runs
  reclaimed by server: 0
  (0 reclaimed: the server has no -read-header-timeout — these conns would stick until fds run out)
```

**Hardened server** (`go run .../server -read-header-timeout 3s`):

```
    server 01:13:40  active=201 in_flight=1 goroutines=206 heap=2MiB
    server 01:13:42  active=1   in_flight=1 goroutines=6   heap=3MiB    ← reclaimed ~3s after opening
  reclaimed by server: 200 (server closed the conn mid-headers → -read-header-timeout worked)
```

**Reading it:** with no `ReadHeaderTimeout`, 200 clients doing *nothing* consume
200 connections indefinitely — a handful of hosts can exhaust a server's fds
without ever completing a request. Setting `ReadHeaderTimeout=3s` drops each conn
~3 s after its last progress and `active` stays flat. The same logic applies to
`IdleTimeout` for *completed* keep-alives that then go quiet. **Every production
`http.Server` should set these; the stdlib defaults are all zero (off).**

### 4. Client-side limits — when the measurement is lying

The load generator hits its own walls, often *before* the server does. Push idle
past the ephemeral-port ceiling and watch the **client** fail while the server is
fine:

```sh
go run ./98-http-server-experiments/client -mode idle -n 20000 -duration 5s
```

```
  opened             : 16312/20000 idle conns held, 3688 failed
  errors             : client-port-limit=3688
  server after open  : active=16313 in_flight=1 goroutines=16318 heap=199MiB
```

**Reading it:** this host has 16 384 ephemeral ports (`49152–65535`), and every
conn to the same server `ip:port` needs a distinct one. At 16 312 the client got
`connect: can't assign requested address` — a **client-side** wall. The server
was untroubled (it held 16 313 conns on ~199 MiB). If you didn't cross-check the
server's `/stats`, you might wrongly conclude "the server maxes out at ~16 k."
Every mode fetches `/stats` at start and end for exactly this reason.

## What to watch

- **`active` vs `in_flight`** — open connections vs running handlers (dimensions 1 & 2).
- **`goroutines`** — ~1 per server conn, ~2 per pooled *client* conn, ~0 per raw client conn.
- **`heap`** — server memory; ~10 KiB per idle conn on this host.
- **`reused` vs `fresh`** (throughput) — is the pool actually reusing conns, or dialing every time?
- **`p50/p99/max`** latency and **error histogram** classified by which side ran out.

The throughput control shows how fast it goes when nothing is broken, and how a
CPU-bound endpoint plateaus near `GOMAXPROCS`:

```sh
go run ./98-http-server-experiments/client -mode throughput -path /static -c 50 -duration 3s
#   requests : 445013 ok, 0 failed  → 148338 req/s   latency p50=259µs p99=1.4ms   reused=444971 fresh=42

go run ./98-http-server-experiments/client -mode throughput -path "/compute?difficulty=200" -c 50 -duration 3s
#   requests : 370 ok, 0 failed  → 123 req/s   latency p50=393ms p99=1.24s
```

150 k req/s on `/static` collapses to ~120 req/s on `/compute?difficulty=200`:
that's CPU-bound, not connection-bound — 8 cores fully busy, more concurrency
just deepens the queue.

## Reading it — symptom → cause

| symptom | cause |
|---|---|
| client `dial: socket: too many open files` | **client** fd/ulimit — raise the client's limit; the measurement was client-bound |
| client `connect: can't assign requested address` | **client** ephemeral ports exhausted — reuse conns, or use multiple client IPs |
| server log `accept: too many open files` | **server** fd ceiling — the real thing |
| `timeout awaiting response headers` / `connection reset` | server can't accept/service; client is fine |
| slowloris conns stick forever | server has no `ReadHeaderTimeout` |
| idle keep-alives never drop | server has no `IdleTimeout` |
| throughput plateaus near `GOMAXPROCS` on `/compute` | CPU-bound, not conn-bound |

The `Classify` helper in `loadlib` turns these strings into counted classes so
the client's error histogram tells you which side gave out.

## Client-side pitfalls (know when the number is yours, not the server's)

- **Ephemeral port exhaustion.** One client → one server `ip:port` gets ~16 k–28 k
  ephemeral ports (`net.inet.ip.portrange.first/last` on macOS,
  `net.ipv4.ip_local_port_range` on Linux). Above that, `connect` fails. Fix for
  the *test*: reuse keep-alive conns (don't open a fresh 4-tuple per request), or
  drive from several client IPs/hosts.
- **TIME_WAIT.** Every actively-closed conn sits in TIME_WAIT ~60 s holding its
  4-tuple; a close-per-request pattern piles them up
  (`netstat -an | grep -c TIME_WAIT`) and starves ephemeral ports. Keep-alive
  reuse avoids it. `SO_REUSEADDR` does not help the client here.
- **`MaxIdleConnsPerHost` (default 2).** If it's below your concurrency, the
  client dials fresh every round and you measure TCP handshakes, not the server.
  The client auto-sizes it to `-c`; override with `-idle-per-host`.
- **`Client.Timeout` / `ResponseHeaderTimeout`.** A timeout force-closes a stuck
  conn, which *hides* a hold or a leak. The client defaults `-timeout 0`; `slow`
  mode disables the header timeout entirely because its responses are slow **by
  design** (an early version misreported held requests as "server-cant-accept").
- **Always cross-check `/stats`.** If the client says "failed" but the server
  never saw the conn, the client ran out first.

## Host caveats (numbers here are not portable)

Measured on: Apple M-series (Mac15,12), 8 cores, macOS (darwin/arm64), go 1.27.
`kern.maxfilesperproc=92160`, ephemeral ports `49152–65535` (16 384).

- **macOS vs Linux differ.** fd caps (`kern.maxfilesperproc` / `ulimit`), TIME_WAIT
  and port-range sysctls, and even `accept`/`dial` error strings are not the same.
- **Listen backlog** caps the accept queue independently of your handler:
  `kern.ipc.somaxconn` (macOS) / `net.core.somaxconn` (Linux).
- **One process, shared fd limit.** If you run both roles in one process they
  share the fd budget and the server usually dies first — use two terminals.

## Decisions log (what we built, how, and why)

- **One server binary + one client binary; scenarios are `-mode`, not separate
  binaries.** The four dimensions stress the *same* server; splitting them would
  duplicate the transport/httptrace/reporting scaffolding four times. Two-process
  split so each side gets its own fd/ulimit (same rationale as `99-experiments/body-drain`'s `-role`).
- **Three live counters, not one.** `active` (ConnState) and `in_flight`
  (middleware) are tracked separately because "conn open" ≠ "handler running" —
  that distinction *is* the idle-vs-slow experiment.
- **`idle` mode uses raw `net.Conn`, not `http.Client`.** The pool reuses conns
  faster than we dial against a fast endpoint, so a pool-based idle mode
  under-counts non-deterministically (20 000 GETs held only ~8 700). Raw dials
  give exactly N — and, as a bonus, expose ephemeral-port exhaustion cleanly.
- **`slowloris` uses raw `net.Conn`.** The stdlib client won't send a deliberately
  incomplete request, so we speak HTTP by hand: request line + headers, never the
  terminating blank line.
- **`Burn` is deterministic and non-elidable.** A data-dependent integer hash of
  `difficulty × 200k` iterations; the handler writes the checksum so the compiler
  can't drop the loop. No `runtime.Gosched`, so it genuinely holds a P. Tested by
  iteration count, not wall-clock, to stay non-flaky.
- **Zero dependencies.** No `x/net/netutil.LimitListener` for `-max-conns`; a real
  server caps concurrency with a semaphore-guarded handler or a wrapped listener,
  and we'd rather observe the OS ceiling than hide it.
- **Server timeouts default to OFF.** So the slowloris vulnerability is
  demonstrable out of the box; turn `-read-header-timeout` / `-idle-timeout` on to
  watch the server reclaim.
- **Network ceilings are not unit-tested.** They're environmental (ulimit, sysctl,
  host); only `Burn`, `Percentile`, and `Classify` are tested. The ceilings live
  here as pasted evidence.

## References

- `net/http.Server` timeout fields — https://pkg.go.dev/net/http#Server
- Cloudflare, "The complete guide to Go net/http timeouts" — https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/
- `net/http/httptrace` — https://pkg.go.dev/net/http/httptrace
- `http.ConnState` — https://pkg.go.dev/net/http#ConnState
- Slowloris (CWE-400 / low-and-slow) — https://en.wikipedia.org/wiki/Slowloris_(cyber_attack)
- Ephemeral ports & TIME_WAIT — https://www.rfc-editor.org/rfc/rfc6335, `sysctl net.inet.ip.portrange` / `net.ipv4.ip_local_port_range`