learning-go-for-sre-interview/98-http-server-experiments/README.md

98 — HTTP server connection experiments

How many connections can a Go HTTP server hold, and what gives out first? This folder finds the order of magnitude (not a portable constant) for four different failure modes, and — just as important — teaches you to tell a real server ceiling from your own load generator running out first.

One simple server is the subject under test; one load client drives it in four modes. Run them in two terminals so each side has its own fd budget.

98-http-server-experiments/
  server/main.go     subject under test: /static /compute /sleep /stats, ConnState + in_flight counters
  client/main.go     load generator: -mode throughput | idle | slow | slowloris
  loadlib/           tested core: Burn (CPU), Percentile, Classify (which side ran out)
  README.md          this file — decisions log + pasted evidence

Pure stdlib, zero dependencies. Part of the gosre-exercises module, so:

go run ./98-http-server-experiments/server        # terminal 1
go run ./98-http-server-experiments/client -mode idle -n 2000   # terminal 2
go test -race ./98-http-server-experiments/loadlib/

Run it

Terminal 1 — the server. Give it its own fd budget so you decide when it runs out, not the OS default. bash sets both soft and hard limits:

ulimit -Hn 4096; ulimit -Sn 4096
go run ./98-http-server-experiments/server                       # naive: no timeouts
go run ./98-http-server-experiments/server -read-header-timeout 3s -idle-timeout 30s   # hardened

(Since Go 1.19 the runtime raises the soft RLIMIT_NOFILE to the hard limit at startup, so only the hard limit really caps you. bash's ulimit -n sets both; zsh's sets only the soft one — use -Hn.)

The server prints one line per second:

19:12:20  accepted_total=10001 active=10001 in_flight=1 goroutines=10006 heap=107MiB

Three numbers that are easy to conflate:

number	meaning	via
accepted_total	every TCP conn ever accepted (only grows)	ConnState StateNew
active	conns open right now — idle keep-alive or busy	ConnState new − closed
in_flight	handlers actually running right now	middleware around the mux

An idle keep-alive counts in active but not in_flight; a running /sleep counts in both. That gap is the whole point of the idle-vs-slow contrast.

Terminal 2 — the client. Give it its own fd budget, separate from the server:

ulimit -Hn 64; ulimit -Sn 64
go run ./98-http-server-experiments/client -mode idle -n 200

The four dimensions

1. Idle connection ceiling — -mode idle

Open -n keep-alive conns and hold them idle. Uses raw net.Conn, not an http.Client — with a fast endpoint the client's connection pool reuses conns faster than you dial, so N concurrent GETs do not reliably produce N simultaneous conns. Dialing raw and keeping each fd open guarantees exactly N.

go run ./98-http-server-experiments/client -mode idle -n 10000 -duration 30s

== mode=idle target=http://127.0.0.1:8098 ==
  server before      : active=1 in_flight=1 goroutines=6 heap=0MiB
  opened             : 10000/10000 idle conns held, 0 failed
  client goroutines  : before=3 after=3 (delta +0; raw fds, no per-conn goroutine)
  server after open  : active=10001 in_flight=1 goroutines=10006 heap=107MiB  ← active≈n, in_flight≈0

Reading it: 10 000 idle conns cost the server ~107 MiB (from a <1 MiB baseline) and one goroutine each → order of magnitude ~10 KiB per idle connection. A million idle keep-alives would be ~10 GiB and a million goroutines — the server holds them fine until it runs out of fds or RAM, not CPU. Note the client's goroutine delta is 0: raw fds have no per-conn goroutine, unlike http.Transport's pool (2 goroutines per pooled conn).

2. Active slow requests — -mode slow

Fire -c concurrent /sleep requests and hold them all in flight for the duration. Same conn count as idle, but now each pins a running server goroutine. Prints the server's in_flight each second.

go run ./98-http-server-experiments/client -mode slow -c 500 -duration 10s

== mode=slow target=http://127.0.0.1:8098 ==
  firing 500 concurrent /sleep?d=10s ...
    server 01:13:13  active=501 in_flight=501 goroutines=1006 heap=25MiB
  requests           : 500 ok, 0 failed (of 500 held in flight)
  errors             : none

Reading it: in_flight=501 (the 500 sleepers + the stats poll) vs idle's in_flight=1 — that is the difference between "conn open" and "handler running." Each in-flight request is one goroutine on a goroutine-per-request server; 500 held costs ~1000 goroutines and a few MiB. Push -c up in orders of magnitude to find where the server (goroutine stacks, fds) — not the client — gives out. /sleep honors r.Context(), so a client that hangs up or a server timeout frees the goroutine immediately rather than sleeping to term.

3. Timeouts & slowloris — -mode slowloris

Open -c raw TCP conns, send a request line + one header on each, then dribble a bogus header every -drip and never send the terminating blank line. The handler never starts. The only thing that reclaims these conns is the server's ReadHeaderTimeout.

Naive server (go run .../server, no timeouts):

== mode=slowloris target=http://127.0.0.1:8098 ==
  opening 200 slowloris conns ...
    server 01:13:26  active=201 in_flight=1 goroutines=206 heap=26MiB   ← stuck, handler never runs
  reclaimed by server: 0
  (0 reclaimed: the server has no -read-header-timeout — these conns would stick until fds run out)

Hardened server (go run .../server -read-header-timeout 3s):

    server 01:13:40  active=201 in_flight=1 goroutines=206 heap=2MiB
    server 01:13:42  active=1   in_flight=1 goroutines=6   heap=3MiB    ← reclaimed ~3s after opening
  reclaimed by server: 200 (server closed the conn mid-headers → -read-header-timeout worked)

Reading it: with no ReadHeaderTimeout, 200 clients doing nothing consume 200 connections indefinitely — a handful of hosts can exhaust a server's fds without ever completing a request. Setting ReadHeaderTimeout=3s drops each conn ~3 s after its last progress and active stays flat. The same logic applies to IdleTimeout for completed keep-alives that then go quiet. Every production http.Server should set these; the stdlib defaults are all zero (off).

4. Client-side limits — when the measurement is lying

The load generator hits its own walls, often before the server does. Push idle past the ephemeral-port ceiling and watch the client fail while the server is fine:

go run ./98-http-server-experiments/client -mode idle -n 20000 -duration 5s

  opened             : 16312/20000 idle conns held, 3688 failed
  errors             : client-port-limit=3688
  server after open  : active=16313 in_flight=1 goroutines=16318 heap=199MiB

Reading it: this host has 16 384 ephemeral ports (49152–65535), and every conn to the same server ip:port needs a distinct one. At 16 312 the client got connect: can't assign requested address — a client-side wall. The server was untroubled (it held 16 313 conns on ~199 MiB). If you didn't cross-check the server's /stats, you might wrongly conclude "the server maxes out at ~16 k." Every mode fetches /stats at start and end for exactly this reason.

What to watch

• active vs in_flight — open connections vs running handlers (dimensions 1 & 2).
• goroutines — ~1 per server conn, ~2 per pooled client conn, ~0 per raw client conn.
• heap — server memory; ~10 KiB per idle conn on this host.
• reused vs fresh (throughput) — is the pool actually reusing conns, or dialing every time?
• p50/p99/max latency and error histogram classified by which side ran out.

The throughput control shows how fast it goes when nothing is broken, and how a CPU-bound endpoint plateaus near GOMAXPROCS:

go run ./98-http-server-experiments/client -mode throughput -path /static -c 50 -duration 3s
#   requests : 445013 ok, 0 failed  → 148338 req/s   latency p50=259µs p99=1.4ms   reused=444971 fresh=42

go run ./98-http-server-experiments/client -mode throughput -path "/compute?difficulty=200" -c 50 -duration 3s
#   requests : 370 ok, 0 failed  → 123 req/s   latency p50=393ms p99=1.24s

150 k req/s on /static collapses to ~120 req/s on /compute?difficulty=200: that's CPU-bound, not connection-bound — 8 cores fully busy, more concurrency just deepens the queue.

Reading it — symptom → cause

symptom	cause
client dial: socket: too many open files	client fd/ulimit — raise the client's limit; the measurement was client-bound
client connect: can't assign requested address	client ephemeral ports exhausted — reuse conns, or use multiple client IPs
server log accept: too many open files	server fd ceiling — the real thing
timeout awaiting response headers / connection reset	server can't accept/service; client is fine
slowloris conns stick forever	server has no ReadHeaderTimeout
idle keep-alives never drop	server has no IdleTimeout
throughput plateaus near GOMAXPROCS on /compute	CPU-bound, not conn-bound

The Classify helper in loadlib turns these strings into counted classes so the client's error histogram tells you which side gave out.

Client-side pitfalls (know when the number is yours, not the server's)

• Ephemeral port exhaustion. One client → one server ip:port gets ~16 k–28 k ephemeral ports (net.inet.ip.portrange.first/last on macOS, net.ipv4.ip_local_port_range on Linux). Above that, connect fails. Fix for the test: reuse keep-alive conns (don't open a fresh 4-tuple per request), or drive from several client IPs/hosts.
• TIME_WAIT. Every actively-closed conn sits in TIME_WAIT ~60 s holding its 4-tuple; a close-per-request pattern piles them up (netstat -an | grep -c TIME_WAIT) and starves ephemeral ports. Keep-alive reuse avoids it. SO_REUSEADDR does not help the client here.
• MaxIdleConnsPerHost (default 2). If it's below your concurrency, the client dials fresh every round and you measure TCP handshakes, not the server. The client auto-sizes it to -c; override with -idle-per-host.
• Client.Timeout / ResponseHeaderTimeout. A timeout force-closes a stuck conn, which hides a hold or a leak. The client defaults -timeout 0; slow mode disables the header timeout entirely because its responses are slow by design (an early version misreported held requests as "server-cant-accept").
• Always cross-check /stats. If the client says "failed" but the server never saw the conn, the client ran out first.

Host caveats (numbers here are not portable)

Measured on: Apple M-series (Mac15,12), 8 cores, macOS (darwin/arm64), go 1.27. kern.maxfilesperproc=92160, ephemeral ports 49152–65535 (16 384).

• macOS vs Linux differ. fd caps (kern.maxfilesperproc / ulimit), TIME_WAIT and port-range sysctls, and even accept/dial error strings are not the same.
• Listen backlog caps the accept queue independently of your handler: kern.ipc.somaxconn (macOS) / net.core.somaxconn (Linux).
• One process, shared fd limit. If you run both roles in one process they share the fd budget and the server usually dies first — use two terminals.

Decisions log (what we built, how, and why)

• One server binary + one client binary; scenarios are -mode, not separate binaries. The four dimensions stress the same server; splitting them would duplicate the transport/httptrace/reporting scaffolding four times. Two-process split so each side gets its own fd/ulimit (same rationale as 99-experiments/body-drain's -role).
• Three live counters, not one. active (ConnState) and in_flight (middleware) are tracked separately because "conn open" ≠ "handler running" — that distinction is the idle-vs-slow experiment.
• idle mode uses raw net.Conn, not http.Client. The pool reuses conns faster than we dial against a fast endpoint, so a pool-based idle mode under-counts non-deterministically (20 000 GETs held only ~8 700). Raw dials give exactly N — and, as a bonus, expose ephemeral-port exhaustion cleanly.
• slowloris uses raw net.Conn. The stdlib client won't send a deliberately incomplete request, so we speak HTTP by hand: request line + headers, never the terminating blank line.
• Burn is deterministic and non-elidable. A data-dependent integer hash of difficulty × 200k iterations; the handler writes the checksum so the compiler can't drop the loop. No runtime.Gosched, so it genuinely holds a P. Tested by iteration count, not wall-clock, to stay non-flaky.
• Zero dependencies. No x/net/netutil.LimitListener for -max-conns; a real server caps concurrency with a semaphore-guarded handler or a wrapped listener, and we'd rather observe the OS ceiling than hide it.
• Server timeouts default to OFF. So the slowloris vulnerability is demonstrable out of the box; turn -read-header-timeout / -idle-timeout on to watch the server reclaim.
• Network ceilings are not unit-tested. They're environmental (ulimit, sysctl, host); only Burn, Percentile, and Classify are tested. The ceilings live here as pasted evidence.

References

• net/http.Server timeout fields — https://pkg.go.dev/net/http#Server
• Cloudflare, "The complete guide to Go net/http timeouts" — https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/
• net/http/httptrace — https://pkg.go.dev/net/http/httptrace
• http.ConnState — https://pkg.go.dev/net/http#ConnState
• Slowloris (CWE-400 / low-and-slow) — https://en.wikipedia.org/wiki/Slowloris_(cyber_attack)
• Ephemeral ports & TIME_WAIT — https://www.rfc-editor.org/rfc/rfc6335, sysctl net.inet.ip.portrange / net.ipv4.ip_local_port_range