Add 98-http-server-experiments: how many connections can a Go server hold

One stdlib server under test (/static /compute /sleep /stats, ConnState and in-flight counters) plus a load client with four modes: throughput, idle, slow, and slowloris. loadlib holds the tested core (Burn, Percentile, Classify). README documents the runs and how to tell a real server ceiling from the load generator giving out first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012MpK9niZm4jcs3mAJbxMfi
2026-09-02 09:34:12 +02:00
parent b290957fc9
commit f08d4f44d4
5 changed files with 1300 additions and 0 deletions
@@ -0,0 +1,283 @@
+# 98 — HTTP server connection experiments
+
+How many connections can a Go HTTP server hold, and what gives out first? This
+folder finds the **order of magnitude** (not a portable constant) for four
+different failure modes, and — just as important — teaches you to tell a real
+server ceiling from your own load generator running out first.
+
+One simple server is the subject under test; one load client drives it in four
+modes. Run them in two terminals so each side has its own fd budget.
+
+```
+98-http-server-experiments/
+  server/main.go     subject under test: /static /compute /sleep /stats, ConnState + in_flight counters
+  client/main.go     load generator: -mode throughput | idle | slow | slowloris
+  loadlib/           tested core: Burn (CPU), Percentile, Classify (which side ran out)
+  README.md          this file — decisions log + pasted evidence
+```
+
+Pure stdlib, zero dependencies. Part of the `gosre-exercises` module, so:
+
+```sh
+go run ./98-http-server-experiments/server        # terminal 1
+go run ./98-http-server-experiments/client -mode idle -n 2000   # terminal 2
+go test -race ./98-http-server-experiments/loadlib/
+```
+
+## Run it
+
+**Terminal 1 — the server.** Give it its own fd budget so *you* decide when it
+runs out, not the OS default. bash sets both soft and hard limits:
+
+```sh
+ulimit -Hn 4096; ulimit -Sn 4096
+go run ./98-http-server-experiments/server                       # naive: no timeouts
+go run ./98-http-server-experiments/server -read-header-timeout 3s -idle-timeout 30s   # hardened
+```
+
+(Since Go 1.19 the runtime raises the soft `RLIMIT_NOFILE` to the hard limit at
+startup, so only the **hard** limit really caps you. bash's `ulimit -n` sets
+both; zsh's sets only the soft one — use `-Hn`.)
+
+The server prints one line per second:
+
+```
+19:12:20  accepted_total=10001 active=10001 in_flight=1 goroutines=10006 heap=107MiB
+```
+
+Three numbers that are easy to conflate:
+
+| number | meaning | via |
+|---|---|---|
+| `accepted_total` | every TCP conn ever accepted (only grows) | `ConnState` StateNew |
+| `active` | conns open **right now** — idle keep-alive *or* busy | `ConnState` new − closed |
+| `in_flight` | handlers **actually running** right now | middleware around the mux |
+
+An idle keep-alive counts in `active` but not `in_flight`; a running `/sleep`
+counts in both. That gap is the whole point of the idle-vs-slow contrast.
+
+**Terminal 2 — the client.** Give it *its own* fd budget, separate from the server:
+
+```sh
+ulimit -Hn 64; ulimit -Sn 64
+go run ./98-http-server-experiments/client -mode idle -n 200
+```
+
+## The four dimensions
+
+### 1. Idle connection ceiling — `-mode idle`
+
+Open `-n` keep-alive conns and hold them idle. Uses **raw `net.Conn`**, not an
+`http.Client` — with a fast endpoint the client's connection pool reuses conns
+faster than you dial, so N concurrent GETs do *not* reliably produce N
+simultaneous conns. Dialing raw and keeping each fd open guarantees exactly N.
+
+```sh
+go run ./98-http-server-experiments/client -mode idle -n 10000 -duration 30s
+```
+
+```
+== mode=idle target=http://127.0.0.1:8098 ==
+  server before      : active=1 in_flight=1 goroutines=6 heap=0MiB
+  opened             : 10000/10000 idle conns held, 0 failed
+  client goroutines  : before=3 after=3 (delta +0; raw fds, no per-conn goroutine)
+  server after open  : active=10001 in_flight=1 goroutines=10006 heap=107MiB  ← active≈n, in_flight≈0
+```
+
+**Reading it:** 10 000 idle conns cost the server ~107 MiB (from a <1 MiB
+baseline) and one goroutine each → **order of magnitude ~10 KiB per idle
+connection**. A million idle keep-alives would be ~10 GiB and a million
+goroutines — the server holds them fine until it runs out of fds or RAM, not CPU.
+Note the client's goroutine delta is **0**: raw fds have no per-conn goroutine,
+unlike `http.Transport`'s pool (2 goroutines per pooled conn).
+
+### 2. Active slow requests — `-mode slow`
+
+Fire `-c` concurrent `/sleep` requests and hold them *all in flight* for the
+duration. Same conn count as idle, but now each pins a **running** server
+goroutine. Prints the server's `in_flight` each second.
+
+```sh
+go run ./98-http-server-experiments/client -mode slow -c 500 -duration 10s
+```
+
+```
+== mode=slow target=http://127.0.0.1:8098 ==
+  firing 500 concurrent /sleep?d=10s ...
+    server 01:13:13  active=501 in_flight=501 goroutines=1006 heap=25MiB
+  requests           : 500 ok, 0 failed (of 500 held in flight)
+  errors             : none
+```
+
+**Reading it:** `in_flight=501` (the 500 sleepers + the stats poll) vs idle's
+`in_flight=1` — that is the difference between "conn open" and "handler running."
+Each in-flight request is one goroutine on a goroutine-per-request server; 500
+held costs ~1000 goroutines and a few MiB. Push `-c` up in orders of magnitude to
+find where the *server* (goroutine stacks, fds) — not the client — gives out.
+`/sleep` honors `r.Context()`, so a client that hangs up or a server timeout
+frees the goroutine immediately rather than sleeping to term.
+
+### 3. Timeouts & slowloris — `-mode slowloris`
+
+Open `-c` raw TCP conns, send a request line + one header on each, then dribble a
+bogus header every `-drip` and **never** send the terminating blank line. The
+handler never starts. The only thing that reclaims these conns is the server's
+`ReadHeaderTimeout`.
+
+**Naive server** (`go run .../server`, no timeouts):
+
+```
+== mode=slowloris target=http://127.0.0.1:8098 ==
+  opening 200 slowloris conns ...
+    server 01:13:26  active=201 in_flight=1 goroutines=206 heap=26MiB   ← stuck, handler never runs
+  reclaimed by server: 0
+  (0 reclaimed: the server has no -read-header-timeout — these conns would stick until fds run out)
+```
+
+**Hardened server** (`go run .../server -read-header-timeout 3s`):
+
+```
+    server 01:13:40  active=201 in_flight=1 goroutines=206 heap=2MiB
+    server 01:13:42  active=1   in_flight=1 goroutines=6   heap=3MiB    ← reclaimed ~3s after opening
+  reclaimed by server: 200 (server closed the conn mid-headers → -read-header-timeout worked)
+```
+
+**Reading it:** with no `ReadHeaderTimeout`, 200 clients doing *nothing* consume
+200 connections indefinitely — a handful of hosts can exhaust a server's fds
+without ever completing a request. Setting `ReadHeaderTimeout=3s` drops each conn
+~3 s after its last progress and `active` stays flat. The same logic applies to
+`IdleTimeout` for *completed* keep-alives that then go quiet. **Every production
+`http.Server` should set these; the stdlib defaults are all zero (off).**
+
+### 4. Client-side limits — when the measurement is lying
+
+The load generator hits its own walls, often *before* the server does. Push idle
+past the ephemeral-port ceiling and watch the **client** fail while the server is
+fine:
+
+```sh
+go run ./98-http-server-experiments/client -mode idle -n 20000 -duration 5s
+```
+
+```
+  opened             : 16312/20000 idle conns held, 3688 failed
+  errors             : client-port-limit=3688
+  server after open  : active=16313 in_flight=1 goroutines=16318 heap=199MiB
+```
+
+**Reading it:** this host has 16 384 ephemeral ports (`49152–65535`), and every
+conn to the same server `ip:port` needs a distinct one. At 16 312 the client got
+`connect: can't assign requested address` — a **client-side** wall. The server
+was untroubled (it held 16 313 conns on ~199 MiB). If you didn't cross-check the
+server's `/stats`, you might wrongly conclude "the server maxes out at ~16 k."
+Every mode fetches `/stats` at start and end for exactly this reason.
+
+## What to watch
+
+- **`active` vs `in_flight`** — open connections vs running handlers (dimensions 1 & 2).
+- **`goroutines`** — ~1 per server conn, ~2 per pooled *client* conn, ~0 per raw client conn.
+- **`heap`** — server memory; ~10 KiB per idle conn on this host.
+- **`reused` vs `fresh`** (throughput) — is the pool actually reusing conns, or dialing every time?
+- **`p50/p99/max`** latency and **error histogram** classified by which side ran out.
+
+The throughput control shows how fast it goes when nothing is broken, and how a
+CPU-bound endpoint plateaus near `GOMAXPROCS`:
+
+```sh
+go run ./98-http-server-experiments/client -mode throughput -path /static -c 50 -duration 3s
+#   requests : 445013 ok, 0 failed  → 148338 req/s   latency p50=259µs p99=1.4ms   reused=444971 fresh=42
+
+go run ./98-http-server-experiments/client -mode throughput -path "/compute?difficulty=200" -c 50 -duration 3s
+#   requests : 370 ok, 0 failed  → 123 req/s   latency p50=393ms p99=1.24s
+```
+
+150 k req/s on `/static` collapses to ~120 req/s on `/compute?difficulty=200`:
+that's CPU-bound, not connection-bound — 8 cores fully busy, more concurrency
+just deepens the queue.
+
+## Reading it — symptom → cause
+
+| symptom | cause |
+|---|---|
+| client `dial: socket: too many open files` | **client** fd/ulimit — raise the client's limit; the measurement was client-bound |
+| client `connect: can't assign requested address` | **client** ephemeral ports exhausted — reuse conns, or use multiple client IPs |
+| server log `accept: too many open files` | **server** fd ceiling — the real thing |
+| `timeout awaiting response headers` / `connection reset` | server can't accept/service; client is fine |
+| slowloris conns stick forever | server has no `ReadHeaderTimeout` |
+| idle keep-alives never drop | server has no `IdleTimeout` |
+| throughput plateaus near `GOMAXPROCS` on `/compute` | CPU-bound, not conn-bound |
+
+The `Classify` helper in `loadlib` turns these strings into counted classes so
+the client's error histogram tells you which side gave out.
+
+## Client-side pitfalls (know when the number is yours, not the server's)
+
+- **Ephemeral port exhaustion.** One client → one server `ip:port` gets ~16 k–28 k
+  ephemeral ports (`net.inet.ip.portrange.first/last` on macOS,
+  `net.ipv4.ip_local_port_range` on Linux). Above that, `connect` fails. Fix for
+  the *test*: reuse keep-alive conns (don't open a fresh 4-tuple per request), or
+  drive from several client IPs/hosts.
+- **TIME_WAIT.** Every actively-closed conn sits in TIME_WAIT ~60 s holding its
+  4-tuple; a close-per-request pattern piles them up
+  (`netstat -an | grep -c TIME_WAIT`) and starves ephemeral ports. Keep-alive
+  reuse avoids it. `SO_REUSEADDR` does not help the client here.
+- **`MaxIdleConnsPerHost` (default 2).** If it's below your concurrency, the
+  client dials fresh every round and you measure TCP handshakes, not the server.
+  The client auto-sizes it to `-c`; override with `-idle-per-host`.
+- **`Client.Timeout` / `ResponseHeaderTimeout`.** A timeout force-closes a stuck
+  conn, which *hides* a hold or a leak. The client defaults `-timeout 0`; `slow`
+  mode disables the header timeout entirely because its responses are slow **by
+  design** (an early version misreported held requests as "server-cant-accept").
+- **Always cross-check `/stats`.** If the client says "failed" but the server
+  never saw the conn, the client ran out first.
+
+## Host caveats (numbers here are not portable)
+
+Measured on: Apple M-series (Mac15,12), 8 cores, macOS (darwin/arm64), go 1.27.
+`kern.maxfilesperproc=92160`, ephemeral ports `49152–65535` (16 384).
+
+- **macOS vs Linux differ.** fd caps (`kern.maxfilesperproc` / `ulimit`), TIME_WAIT
+  and port-range sysctls, and even `accept`/`dial` error strings are not the same.
+- **Listen backlog** caps the accept queue independently of your handler:
+  `kern.ipc.somaxconn` (macOS) / `net.core.somaxconn` (Linux).
+- **One process, shared fd limit.** If you run both roles in one process they
+  share the fd budget and the server usually dies first — use two terminals.
+
+## Decisions log (what we built, how, and why)
+
+- **One server binary + one client binary; scenarios are `-mode`, not separate
+  binaries.** The four dimensions stress the *same* server; splitting them would
+  duplicate the transport/httptrace/reporting scaffolding four times. Two-process
+  split so each side gets its own fd/ulimit (same rationale as `99-experiments/body-drain`'s `-role`).
+- **Three live counters, not one.** `active` (ConnState) and `in_flight`
+  (middleware) are tracked separately because "conn open" ≠ "handler running" —
+  that distinction *is* the idle-vs-slow experiment.
+- **`idle` mode uses raw `net.Conn`, not `http.Client`.** The pool reuses conns
+  faster than we dial against a fast endpoint, so a pool-based idle mode
+  under-counts non-deterministically (20 000 GETs held only ~8 700). Raw dials
+  give exactly N — and, as a bonus, expose ephemeral-port exhaustion cleanly.
+- **`slowloris` uses raw `net.Conn`.** The stdlib client won't send a deliberately
+  incomplete request, so we speak HTTP by hand: request line + headers, never the
+  terminating blank line.
+- **`Burn` is deterministic and non-elidable.** A data-dependent integer hash of
+  `difficulty × 200k` iterations; the handler writes the checksum so the compiler
+  can't drop the loop. No `runtime.Gosched`, so it genuinely holds a P. Tested by
+  iteration count, not wall-clock, to stay non-flaky.
+- **Zero dependencies.** No `x/net/netutil.LimitListener` for `-max-conns`; a real
+  server caps concurrency with a semaphore-guarded handler or a wrapped listener,
+  and we'd rather observe the OS ceiling than hide it.
+- **Server timeouts default to OFF.** So the slowloris vulnerability is
+  demonstrable out of the box; turn `-read-header-timeout` / `-idle-timeout` on to
+  watch the server reclaim.
+- **Network ceilings are not unit-tested.** They're environmental (ulimit, sysctl,
+  host); only `Burn`, `Percentile`, and `Classify` are tested. The ceilings live
+  here as pasted evidence.
+
+## References
+
+- `net/http.Server` timeout fields — https://pkg.go.dev/net/http#Server
+- Cloudflare, "The complete guide to Go net/http timeouts" — https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/
+- `net/http/httptrace` — https://pkg.go.dev/net/http/httptrace
+- `http.ConnState` — https://pkg.go.dev/net/http#ConnState
+- Slowloris (CWE-400 / low-and-slow) — https://en.wikipedia.org/wiki/Slowloris_(cyber_attack)
+- Ephemeral ports & TIME_WAIT — https://www.rfc-editor.org/rfc/rfc6335, `sysctl net.inet.ip.portrange` / `net.ipv4.ip_local_port_range`