Add 98-http-server-experiments: how many connections can a Go server hold

One stdlib server under test (/static /compute /sleep /stats, ConnState and in-flight counters) plus a load client with four modes: throughput, idle, slow, and slowloris. loadlib holds the tested core (Burn, Percentile, Classify). README documents the runs and how to tell a real server ceiling from the load generator giving out first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012MpK9niZm4jcs3mAJbxMfi
2026-09-02 09:34:12 +02:00
parent b290957fc9
commit f08d4f44d4
5 changed files with 1300 additions and 0 deletions
@@ -0,0 +1,283 @@
+# 98 — HTTP server connection experiments
+
+How many connections can a Go HTTP server hold, and what gives out first? This
+folder finds the **order of magnitude** (not a portable constant) for four
+different failure modes, and — just as important — teaches you to tell a real
+server ceiling from your own load generator running out first.
+
+One simple server is the subject under test; one load client drives it in four
+modes. Run them in two terminals so each side has its own fd budget.
+
+```
+98-http-server-experiments/
+  server/main.go     subject under test: /static /compute /sleep /stats, ConnState + in_flight counters
+  client/main.go     load generator: -mode throughput | idle | slow | slowloris
+  loadlib/           tested core: Burn (CPU), Percentile, Classify (which side ran out)
+  README.md          this file — decisions log + pasted evidence
+```
+
+Pure stdlib, zero dependencies. Part of the `gosre-exercises` module, so:
+
+```sh
+go run ./98-http-server-experiments/server        # terminal 1
+go run ./98-http-server-experiments/client -mode idle -n 2000   # terminal 2
+go test -race ./98-http-server-experiments/loadlib/
+```
+
+## Run it
+
+**Terminal 1 — the server.** Give it its own fd budget so *you* decide when it
+runs out, not the OS default. bash sets both soft and hard limits:
+
+```sh
+ulimit -Hn 4096; ulimit -Sn 4096
+go run ./98-http-server-experiments/server                       # naive: no timeouts
+go run ./98-http-server-experiments/server -read-header-timeout 3s -idle-timeout 30s   # hardened
+```
+
+(Since Go 1.19 the runtime raises the soft `RLIMIT_NOFILE` to the hard limit at
+startup, so only the **hard** limit really caps you. bash's `ulimit -n` sets
+both; zsh's sets only the soft one — use `-Hn`.)
+
+The server prints one line per second:
+
+```
+19:12:20  accepted_total=10001 active=10001 in_flight=1 goroutines=10006 heap=107MiB
+```
+
+Three numbers that are easy to conflate:
+
+| number | meaning | via |
+|---|---|---|
+| `accepted_total` | every TCP conn ever accepted (only grows) | `ConnState` StateNew |
+| `active` | conns open **right now** — idle keep-alive *or* busy | `ConnState` new − closed |
+| `in_flight` | handlers **actually running** right now | middleware around the mux |
+
+An idle keep-alive counts in `active` but not `in_flight`; a running `/sleep`
+counts in both. That gap is the whole point of the idle-vs-slow contrast.
+
+**Terminal 2 — the client.** Give it *its own* fd budget, separate from the server:
+
+```sh
+ulimit -Hn 64; ulimit -Sn 64
+go run ./98-http-server-experiments/client -mode idle -n 200
+```
+
+## The four dimensions
+
+### 1. Idle connection ceiling — `-mode idle`
+
+Open `-n` keep-alive conns and hold them idle. Uses **raw `net.Conn`**, not an
+`http.Client` — with a fast endpoint the client's connection pool reuses conns
+faster than you dial, so N concurrent GETs do *not* reliably produce N
+simultaneous conns. Dialing raw and keeping each fd open guarantees exactly N.
+
+```sh
+go run ./98-http-server-experiments/client -mode idle -n 10000 -duration 30s
+```
+
+```
+== mode=idle target=http://127.0.0.1:8098 ==
+  server before      : active=1 in_flight=1 goroutines=6 heap=0MiB
+  opened             : 10000/10000 idle conns held, 0 failed
+  client goroutines  : before=3 after=3 (delta +0; raw fds, no per-conn goroutine)
+  server after open  : active=10001 in_flight=1 goroutines=10006 heap=107MiB  ← active≈n, in_flight≈0
+```
+
+**Reading it:** 10 000 idle conns cost the server ~107 MiB (from a <1 MiB
+baseline) and one goroutine each → **order of magnitude ~10 KiB per idle
+connection**. A million idle keep-alives would be ~10 GiB and a million
+goroutines — the server holds them fine until it runs out of fds or RAM, not CPU.
+Note the client's goroutine delta is **0**: raw fds have no per-conn goroutine,
+unlike `http.Transport`'s pool (2 goroutines per pooled conn).
+
+### 2. Active slow requests — `-mode slow`
+
+Fire `-c` concurrent `/sleep` requests and hold them *all in flight* for the
+duration. Same conn count as idle, but now each pins a **running** server
+goroutine. Prints the server's `in_flight` each second.
+
+```sh
+go run ./98-http-server-experiments/client -mode slow -c 500 -duration 10s
+```
+
+```
+== mode=slow target=http://127.0.0.1:8098 ==
+  firing 500 concurrent /sleep?d=10s ...
+    server 01:13:13  active=501 in_flight=501 goroutines=1006 heap=25MiB
+  requests           : 500 ok, 0 failed (of 500 held in flight)
+  errors             : none
+```
+
+**Reading it:** `in_flight=501` (the 500 sleepers + the stats poll) vs idle's
+`in_flight=1` — that is the difference between "conn open" and "handler running."
+Each in-flight request is one goroutine on a goroutine-per-request server; 500
+held costs ~1000 goroutines and a few MiB. Push `-c` up in orders of magnitude to
+find where the *server* (goroutine stacks, fds) — not the client — gives out.
+`/sleep` honors `r.Context()`, so a client that hangs up or a server timeout
+frees the goroutine immediately rather than sleeping to term.
+
+### 3. Timeouts & slowloris — `-mode slowloris`
+
+Open `-c` raw TCP conns, send a request line + one header on each, then dribble a
+bogus header every `-drip` and **never** send the terminating blank line. The
+handler never starts. The only thing that reclaims these conns is the server's
+`ReadHeaderTimeout`.
+
+**Naive server** (`go run .../server`, no timeouts):
+
+```
+== mode=slowloris target=http://127.0.0.1:8098 ==
+  opening 200 slowloris conns ...
+    server 01:13:26  active=201 in_flight=1 goroutines=206 heap=26MiB   ← stuck, handler never runs
+  reclaimed by server: 0
+  (0 reclaimed: the server has no -read-header-timeout — these conns would stick until fds run out)
+```
+
+**Hardened server** (`go run .../server -read-header-timeout 3s`):
+
+```
+    server 01:13:40  active=201 in_flight=1 goroutines=206 heap=2MiB
+    server 01:13:42  active=1   in_flight=1 goroutines=6   heap=3MiB    ← reclaimed ~3s after opening
+  reclaimed by server: 200 (server closed the conn mid-headers → -read-header-timeout worked)
+```
+
+**Reading it:** with no `ReadHeaderTimeout`, 200 clients doing *nothing* consume
+200 connections indefinitely — a handful of hosts can exhaust a server's fds
+without ever completing a request. Setting `ReadHeaderTimeout=3s` drops each conn
+~3 s after its last progress and `active` stays flat. The same logic applies to
+`IdleTimeout` for *completed* keep-alives that then go quiet. **Every production
+`http.Server` should set these; the stdlib defaults are all zero (off).**
+
+### 4. Client-side limits — when the measurement is lying
+
+The load generator hits its own walls, often *before* the server does. Push idle
+past the ephemeral-port ceiling and watch the **client** fail while the server is
+fine:
+
+```sh
+go run ./98-http-server-experiments/client -mode idle -n 20000 -duration 5s
+```
+
+```
+  opened             : 16312/20000 idle conns held, 3688 failed
+  errors             : client-port-limit=3688
+  server after open  : active=16313 in_flight=1 goroutines=16318 heap=199MiB
+```
+
+**Reading it:** this host has 16 384 ephemeral ports (`49152–65535`), and every
+conn to the same server `ip:port` needs a distinct one. At 16 312 the client got
+`connect: can't assign requested address` — a **client-side** wall. The server
+was untroubled (it held 16 313 conns on ~199 MiB). If you didn't cross-check the
+server's `/stats`, you might wrongly conclude "the server maxes out at ~16 k."
+Every mode fetches `/stats` at start and end for exactly this reason.
+
+## What to watch
+
+- **`active` vs `in_flight`** — open connections vs running handlers (dimensions 1 & 2).
+- **`goroutines`** — ~1 per server conn, ~2 per pooled *client* conn, ~0 per raw client conn.
+- **`heap`** — server memory; ~10 KiB per idle conn on this host.
+- **`reused` vs `fresh`** (throughput) — is the pool actually reusing conns, or dialing every time?
+- **`p50/p99/max`** latency and **error histogram** classified by which side ran out.
+
+The throughput control shows how fast it goes when nothing is broken, and how a
+CPU-bound endpoint plateaus near `GOMAXPROCS`:
+
+```sh
+go run ./98-http-server-experiments/client -mode throughput -path /static -c 50 -duration 3s
+#   requests : 445013 ok, 0 failed  → 148338 req/s   latency p50=259µs p99=1.4ms   reused=444971 fresh=42
+
+go run ./98-http-server-experiments/client -mode throughput -path "/compute?difficulty=200" -c 50 -duration 3s
+#   requests : 370 ok, 0 failed  → 123 req/s   latency p50=393ms p99=1.24s
+```
+
+150 k req/s on `/static` collapses to ~120 req/s on `/compute?difficulty=200`:
+that's CPU-bound, not connection-bound — 8 cores fully busy, more concurrency
+just deepens the queue.
+
+## Reading it — symptom → cause
+
+| symptom | cause |
+|---|---|
+| client `dial: socket: too many open files` | **client** fd/ulimit — raise the client's limit; the measurement was client-bound |
+| client `connect: can't assign requested address` | **client** ephemeral ports exhausted — reuse conns, or use multiple client IPs |
+| server log `accept: too many open files` | **server** fd ceiling — the real thing |
+| `timeout awaiting response headers` / `connection reset` | server can't accept/service; client is fine |
+| slowloris conns stick forever | server has no `ReadHeaderTimeout` |
+| idle keep-alives never drop | server has no `IdleTimeout` |
+| throughput plateaus near `GOMAXPROCS` on `/compute` | CPU-bound, not conn-bound |
+
+The `Classify` helper in `loadlib` turns these strings into counted classes so
+the client's error histogram tells you which side gave out.
+
+## Client-side pitfalls (know when the number is yours, not the server's)
+
+- **Ephemeral port exhaustion.** One client → one server `ip:port` gets ~16 k–28 k
+  ephemeral ports (`net.inet.ip.portrange.first/last` on macOS,
+  `net.ipv4.ip_local_port_range` on Linux). Above that, `connect` fails. Fix for
+  the *test*: reuse keep-alive conns (don't open a fresh 4-tuple per request), or
+  drive from several client IPs/hosts.
+- **TIME_WAIT.** Every actively-closed conn sits in TIME_WAIT ~60 s holding its
+  4-tuple; a close-per-request pattern piles them up
+  (`netstat -an | grep -c TIME_WAIT`) and starves ephemeral ports. Keep-alive
+  reuse avoids it. `SO_REUSEADDR` does not help the client here.
+- **`MaxIdleConnsPerHost` (default 2).** If it's below your concurrency, the
+  client dials fresh every round and you measure TCP handshakes, not the server.
+  The client auto-sizes it to `-c`; override with `-idle-per-host`.
+- **`Client.Timeout` / `ResponseHeaderTimeout`.** A timeout force-closes a stuck
+  conn, which *hides* a hold or a leak. The client defaults `-timeout 0`; `slow`
+  mode disables the header timeout entirely because its responses are slow **by
+  design** (an early version misreported held requests as "server-cant-accept").
+- **Always cross-check `/stats`.** If the client says "failed" but the server
+  never saw the conn, the client ran out first.
+
+## Host caveats (numbers here are not portable)
+
+Measured on: Apple M-series (Mac15,12), 8 cores, macOS (darwin/arm64), go 1.27.
+`kern.maxfilesperproc=92160`, ephemeral ports `49152–65535` (16 384).
+
+- **macOS vs Linux differ.** fd caps (`kern.maxfilesperproc` / `ulimit`), TIME_WAIT
+  and port-range sysctls, and even `accept`/`dial` error strings are not the same.
+- **Listen backlog** caps the accept queue independently of your handler:
+  `kern.ipc.somaxconn` (macOS) / `net.core.somaxconn` (Linux).
+- **One process, shared fd limit.** If you run both roles in one process they
+  share the fd budget and the server usually dies first — use two terminals.
+
+## Decisions log (what we built, how, and why)
+
+- **One server binary + one client binary; scenarios are `-mode`, not separate
+  binaries.** The four dimensions stress the *same* server; splitting them would
+  duplicate the transport/httptrace/reporting scaffolding four times. Two-process
+  split so each side gets its own fd/ulimit (same rationale as `99-experiments/body-drain`'s `-role`).
+- **Three live counters, not one.** `active` (ConnState) and `in_flight`
+  (middleware) are tracked separately because "conn open" ≠ "handler running" —
+  that distinction *is* the idle-vs-slow experiment.
+- **`idle` mode uses raw `net.Conn`, not `http.Client`.** The pool reuses conns
+  faster than we dial against a fast endpoint, so a pool-based idle mode
+  under-counts non-deterministically (20 000 GETs held only ~8 700). Raw dials
+  give exactly N — and, as a bonus, expose ephemeral-port exhaustion cleanly.
+- **`slowloris` uses raw `net.Conn`.** The stdlib client won't send a deliberately
+  incomplete request, so we speak HTTP by hand: request line + headers, never the
+  terminating blank line.
+- **`Burn` is deterministic and non-elidable.** A data-dependent integer hash of
+  `difficulty × 200k` iterations; the handler writes the checksum so the compiler
+  can't drop the loop. No `runtime.Gosched`, so it genuinely holds a P. Tested by
+  iteration count, not wall-clock, to stay non-flaky.
+- **Zero dependencies.** No `x/net/netutil.LimitListener` for `-max-conns`; a real
+  server caps concurrency with a semaphore-guarded handler or a wrapped listener,
+  and we'd rather observe the OS ceiling than hide it.
+- **Server timeouts default to OFF.** So the slowloris vulnerability is
+  demonstrable out of the box; turn `-read-header-timeout` / `-idle-timeout` on to
+  watch the server reclaim.
+- **Network ceilings are not unit-tested.** They're environmental (ulimit, sysctl,
+  host); only `Burn`, `Percentile`, and `Classify` are tested. The ceilings live
+  here as pasted evidence.
+
+## References
+
+- `net/http.Server` timeout fields — https://pkg.go.dev/net/http#Server
+- Cloudflare, "The complete guide to Go net/http timeouts" — https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/
+- `net/http/httptrace` — https://pkg.go.dev/net/http/httptrace
+- `http.ConnState` — https://pkg.go.dev/net/http#ConnState
+- Slowloris (CWE-400 / low-and-slow) — https://en.wikipedia.org/wiki/Slowloris_(cyber_attack)
+- Ephemeral ports & TIME_WAIT — https://www.rfc-editor.org/rfc/rfc6335, `sysctl net.inet.ip.portrange` / `net.ipv4.ip_local_port_range`
@@ -0,0 +1,487 @@
+// Load generator for the experiment server. One binary, four modes, each a
+// different way to ask "how many connections, and what gives out first?".
+//
+// Modes (-mode):
+//
+//	throughput  closed loop: -c workers hammer -path for -duration, reusing conns.
+//	            The control — how fast when nothing is broken. Point -path at
+//	            /compute?difficulty=N to watch it plateau near GOMAXPROCS.
+//	idle        open -n keep-alive conns to /static and HOLD them (don't close).
+//	            Measures the cost of an idle connection: fds, goroutines, memory.
+//	slow        fire -c concurrent /sleep requests and hold them all in flight.
+//	            Same conn count as idle, but each pins a running server goroutine.
+//	slowloris   -c raw TCP conns that dribble headers forever and never finish
+//	            the request. Defeated only by the server's -read-header-timeout.
+//
+// Every mode cross-checks the server's own /stats so you can tell a real server
+// ceiling from your own client running out first. Which error you get tells you
+// which side gave out:
+//
+//	dial ...: socket: too many open files   → THIS process (raise the client's ulimit)
+//	timeout awaiting response headers        → the server can't accept
+//	connection reset by peer                 → the server closed on you (or a timeout fired)
+//	connection refused                       → nothing listening
+//
+// Give the client its own fd budget, separate from the server's:
+//
+//	ulimit -Hn 64; ulimit -Sn 64; go run ./98-http-server-experiments/client -mode idle -n 200
+//
+// (bash sets soft+hard; zsh only soft — use -Hn. Since Go 1.19 only the hard
+// limit really caps you.)
+package main
+
+import (
+	"bufio"
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"net/http/httptrace"
+	"os"
+	"os/signal"
+	"runtime"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"time"
+
+	"gosre-exercises/98-http-server-experiments/loadlib"
+)
+
+// serverStats mirrors the server's /stats JSON for the cross-check.
+type serverStats struct {
+	AcceptedTotal int64  `json:"accepted_total"`
+	ActiveConns   int64  `json:"active_conns"`
+	InFlight      int64  `json:"in_flight"`
+	Goroutines    int    `json:"goroutines"`
+	HeapBytes     uint64 `json:"heap_bytes"`
+}
+
+func (s serverStats) String() string {
+	return fmt.Sprintf("active=%d in_flight=%d goroutines=%d heap=%dMiB",
+		s.ActiveConns, s.InFlight, s.Goroutines, s.HeapBytes>>20)
+}
+
+// fetchStats reads the server's /stats over a throwaway conn (own transport so it
+// never borrows from or perturbs the pool we're measuring).
+func fetchStats(base string) (serverStats, error) {
+	var st serverStats
+	c := &http.Client{Timeout: 2 * time.Second}
+	resp, err := c.Get(base + "/stats")
+	if err != nil {
+		return st, err
+	}
+	defer resp.Body.Close()
+	err = json.NewDecoder(resp.Body).Decode(&st)
+	return st, err
+}
+
+// opts bundle the CLI knobs the modes read.
+type opts struct {
+	url, path     string
+	n, c          int
+	duration      time.Duration
+	idlePerHost   int
+	headerTimeout time.Duration
+	clientTimeout time.Duration
+	drip          time.Duration
+}
+
+// newTransport builds a fresh pool so runs don't share state. idlePerHost must be
+// >= the concurrency you want reused, or conns get dropped instead of pooled.
+func newTransport(o opts) *http.Transport {
+	tr := http.DefaultTransport.(*http.Transport).Clone()
+	tr.MaxIdleConns = o.n + o.c
+	tr.MaxIdleConnsPerHost = o.idlePerHost
+	tr.ResponseHeaderTimeout = o.headerTimeout
+	return tr
+}
+
+// errHistogram counts failures by which side they blame.
+type errHistogram struct {
+	mu sync.Mutex
+	m  map[loadlib.ErrClass]int
+}
+
+func newErrHistogram() *errHistogram { return &errHistogram{m: map[loadlib.ErrClass]int{}} }
+func (h *errHistogram) add(err error) {
+	h.mu.Lock()
+	h.m[loadlib.Classify(err)]++
+	h.mu.Unlock()
+}
+func (h *errHistogram) String() string {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	if len(h.m) == 0 {
+		return "none"
+	}
+	s := ""
+	for cls, n := range h.m {
+		s += fmt.Sprintf("%s=%d ", cls, n)
+	}
+	return s
+}
+
+// ---- mode: throughput ----
+
+// runThroughput drives a closed loop: c workers each loop Do()+drain for the
+// duration. Conns are reused, so this measures service speed, not dial cost.
+func runThroughput(o opts) {
+	tr := newTransport(o)
+	defer tr.CloseIdleConnections()
+
+	var reused, fresh, ok, failed atomic.Int64
+	trace := &httptrace.ClientTrace{GotConn: func(i httptrace.GotConnInfo) {
+		if i.Reused {
+			reused.Add(1)
+		} else {
+			fresh.Add(1)
+		}
+	}}
+	ctx := httptrace.WithClientTrace(context.Background(), trace)
+	deadline := time.Now().Add(o.duration)
+	client := &http.Client{Transport: tr, Timeout: o.clientTimeout}
+	hist := newErrHistogram()
+	target := o.url + o.path
+
+	var wg sync.WaitGroup
+	var lats []loadlib.Latencies
+	lats = make([]loadlib.Latencies, o.c)
+	for w := 0; w < o.c; w++ {
+		wg.Add(1)
+		go func(w int) {
+			defer wg.Done()
+			for time.Now().Before(deadline) {
+				start := time.Now()
+				req, _ := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
+				resp, err := client.Do(req)
+				if err != nil {
+					failed.Add(1)
+					hist.add(err)
+					continue
+				}
+				io.Copy(io.Discard, resp.Body)
+				resp.Body.Close()
+				lats[w].Add(time.Since(start))
+				ok.Add(1)
+			}
+		}(w)
+	}
+	wg.Wait()
+
+	var all loadlib.Latencies
+	for i := range lats {
+		all.Merge(&lats[i])
+	}
+	p50, p99, max := all.Snapshot()
+	rps := float64(ok.Load()) / o.duration.Seconds()
+	fmt.Printf("  target             : %s (concurrency %d, %v)\n", target, o.c, o.duration)
+	fmt.Printf("  requests           : %d ok, %d failed  → %.0f req/s\n", ok.Load(), failed.Load(), rps)
+	fmt.Printf("  latency            : p50=%v p99=%v max=%v\n", p50.Round(time.Microsecond), p99.Round(time.Microsecond), max.Round(time.Microsecond))
+	fmt.Printf("  conns              : reused=%d fresh=%d\n", reused.Load(), fresh.Load())
+	fmt.Printf("  errors             : %s\n", hist)
+}
+
+// ---- mode: idle ----
+
+// runIdle opens n keep-alive conns and HOLDS them idle, then reports the server's
+// view. It uses raw net.Conn rather than http.Client on purpose: with a fast
+// endpoint the client's connection pool reuses conns faster than we dial, so n
+// concurrent GETs do NOT reliably produce n simultaneous conns. Dialing raw and
+// keeping each fd open guarantees exactly n idle keep-alives — the number we want
+// to be honest about. Each does one real GET /static (so the server completes a
+// request and parks the conn as idle: active=n, in_flight≈0), then we sit on it.
+func runIdle(o opts) {
+	before, _ := fetchStats(o.url)
+	fmt.Printf("  server before      : %s\n", before)
+
+	host := hostPort(o.url)
+	grBefore := runtime.NumGoroutine()
+
+	var opened, failed atomic.Int64
+	hist := newErrHistogram()
+	conns := make([]net.Conn, 0, o.n)
+	var mu sync.Mutex
+
+	// Bounded dial concurrency: fast enough to open thousands quickly, without a
+	// thundering herd of goroutines. Each conn, once its response is drained,
+	// just sits open — no per-conn goroutine, unlike the transport's 2-per-conn.
+	sem := make(chan struct{}, 256)
+	var wg sync.WaitGroup
+	for i := 0; i < o.n; i++ {
+		sem <- struct{}{}
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			defer func() { <-sem }()
+			conn, err := net.DialTimeout("tcp", host, 5*time.Second)
+			if err != nil {
+				failed.Add(1)
+				hist.add(err)
+				return
+			}
+			// One real keep-alive request, response drained, conn left open.
+			if _, err := fmt.Fprintf(conn, "GET /static HTTP/1.1\r\nHost: %s\r\nConnection: keep-alive\r\n\r\n", host); err != nil {
+				failed.Add(1)
+				hist.add(err)
+				conn.Close()
+				return
+			}
+			br := bufio.NewReader(conn)
+			resp, err := http.ReadResponse(br, nil)
+			if err != nil {
+				failed.Add(1)
+				hist.add(err)
+				conn.Close()
+				return
+			}
+			io.Copy(io.Discard, resp.Body)
+			resp.Body.Close()
+			mu.Lock()
+			conns = append(conns, conn)
+			mu.Unlock()
+			opened.Add(1)
+		}()
+	}
+	wg.Wait()
+	time.Sleep(100 * time.Millisecond) // let the server finish parking them as idle
+
+	peak, _ := fetchStats(o.url)
+	fmt.Printf("  opened             : %d/%d idle conns held, %d failed\n", opened.Load(), o.n, failed.Load())
+	fmt.Printf("  errors             : %s\n", hist)
+	fmt.Printf("  client goroutines  : before=%d after=%d (delta %+d; raw fds, no per-conn goroutine)\n",
+		grBefore, runtime.NumGoroutine(), runtime.NumGoroutine()-grBefore)
+	fmt.Printf("  server after open  : %s  ← active≈n, in_flight≈0: these conns are OPEN but no handler runs\n", peak)
+	fmt.Printf("  holding %d idle conns for %v — watch the server's active= stay put. Ctrl-C to release early.\n", opened.Load(), o.duration)
+
+	holdFor(o.url, o.duration)
+
+	for _, c := range conns {
+		c.Close()
+	}
+	time.Sleep(200 * time.Millisecond)
+	after, _ := fetchStats(o.url)
+	fmt.Printf("  server after close : %s (active should fall back toward its baseline)\n", after)
+}
+
+// ---- mode: slow ----
+
+// runSlow fires c concurrent /sleep requests that each stay in flight for the
+// whole duration, so c server goroutines are pinned at once. Prints the server's
+// in_flight/goroutines each second while they're held.
+func runSlow(o opts) {
+	before, _ := fetchStats(o.url)
+	fmt.Printf("  server before      : %s\n", before)
+
+	// By design every response is slow (it arrives only after the sleep), so we
+	// must NOT apply a response-header timeout here — it would abort legitimate
+	// slow responses and lie that the server "can't accept". Instead we bound
+	// each request with a context deadline a bit past the sleep, so a truly
+	// stuck request still unblocks wg.Wait() and gets reported.
+	tr := newTransport(o)
+	tr.ResponseHeaderTimeout = 0
+	defer tr.CloseIdleConnections()
+	client := &http.Client{Transport: tr}
+	target := fmt.Sprintf("%s/sleep?d=%s", o.url, o.duration)
+	reqTimeout := o.duration + 10*time.Second
+
+	var ok, failed atomic.Int64
+	hist := newErrHistogram()
+
+	stopPoll := make(chan struct{})
+	go pollStats(o.url, stopPoll)
+
+	fmt.Printf("  firing %d concurrent /sleep?d=%v ...\n", o.c, o.duration)
+	var wg sync.WaitGroup
+	for i := 0; i < o.c; i++ {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			ctx, cancel := context.WithTimeout(context.Background(), reqTimeout)
+			defer cancel()
+			req, _ := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
+			resp, err := client.Do(req)
+			if err != nil {
+				failed.Add(1)
+				hist.add(err)
+				return
+			}
+			io.Copy(io.Discard, resp.Body)
+			resp.Body.Close()
+			ok.Add(1)
+		}()
+	}
+	wg.Wait()
+	close(stopPoll)
+
+	fmt.Printf("  requests           : %d ok, %d failed (of %d held in flight)\n", ok.Load(), failed.Load(), o.c)
+	fmt.Printf("  errors             : %s\n", hist)
+}
+
+// ---- mode: slowloris ----
+
+// runSlowloris opens c raw TCP conns, sends a request line + one header on each,
+// then dribbles a bogus header every -drip and NEVER sends the terminating blank
+// line. The handler never starts. A server with -read-header-timeout closes the
+// conn after that timeout; we detect it as a read returning EOF/reset and count
+// how many the server reclaimed.
+func runSlowloris(o opts) {
+	host := hostPort(o.url)
+	before, _ := fetchStats(o.url)
+	fmt.Printf("  server before      : %s\n", before)
+	fmt.Printf("  opening %d slowloris conns to %s (drip every %v) ...\n", o.c, host, o.drip)
+
+	var held, reclaimed, dialFailed atomic.Int64
+	hist := newErrHistogram()
+	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
+	defer stop()
+
+	var wg sync.WaitGroup
+	for i := 0; i < o.c; i++ {
+		conn, err := net.DialTimeout("tcp", host, 5*time.Second)
+		if err != nil {
+			dialFailed.Add(1)
+			hist.add(err)
+			continue
+		}
+		held.Add(1)
+		wg.Add(1)
+		go func(conn net.Conn) {
+			defer wg.Done()
+			defer conn.Close()
+			// Incomplete request: a valid start line and Host, but no blank line.
+			fmt.Fprintf(conn, "GET /static HTTP/1.1\r\nHost: %s\r\n", host)
+			dripTick := time.NewTicker(o.drip)
+			defer dripTick.Stop()
+			// A reader goroutine turns a server-side close into a signal.
+			closed := make(chan struct{})
+			go func() {
+				buf := make([]byte, 1)
+				conn.Read(buf) // blocks until the server closes → EOF/reset
+				close(closed)
+			}()
+			for {
+				select {
+				case <-ctx.Done():
+					return
+				case <-closed:
+					reclaimed.Add(1)
+					return
+				case <-dripTick.C:
+					// One more bogus header — still no terminating blank line.
+					if _, err := fmt.Fprint(conn, "X-Keep: alive\r\n"); err != nil {
+						reclaimed.Add(1)
+						return
+					}
+				}
+			}
+		}(conn)
+	}
+
+	// Report the server's view periodically while conns are held.
+	stopPoll := make(chan struct{})
+	go pollStats(o.url, stopPoll)
+	select {
+	case <-ctx.Done():
+	case <-time.After(o.duration):
+	}
+	stop()          // stop the drip loops
+	close(stopPoll) // stop polling
+	wg.Wait()
+
+	after, _ := fetchStats(o.url)
+	fmt.Printf("  held               : %d conns opened, %d dial-failed\n", held.Load(), dialFailed.Load())
+	fmt.Printf("  reclaimed by server: %d (server closed the conn mid-headers → -read-header-timeout worked)\n", reclaimed.Load())
+	fmt.Printf("  errors             : %s\n", hist)
+	fmt.Printf("  server after       : %s\n", after)
+	if reclaimed.Load() == 0 {
+		fmt.Println("  (0 reclaimed: the server has no -read-header-timeout — these conns would stick until fds run out)")
+	}
+}
+
+// ---- helpers ----
+
+// pollStats prints the server's /stats once a second until stop is closed.
+func pollStats(base string, stop <-chan struct{}) {
+	tick := time.NewTicker(time.Second)
+	defer tick.Stop()
+	for {
+		select {
+		case <-stop:
+			return
+		case <-tick.C:
+			if st, err := fetchStats(base); err == nil {
+				fmt.Printf("    server %s  %s\n", time.Now().Format("15:04:05"), st)
+			}
+		}
+	}
+}
+
+// holdFor blocks for d or until Ctrl-C, printing the server's view each second so
+// you can watch the held conns sit there.
+func holdFor(base string, d time.Duration) {
+	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
+	defer stop()
+	stopPoll := make(chan struct{})
+	go pollStats(base, stopPoll)
+	select {
+	case <-ctx.Done():
+	case <-time.After(d):
+	}
+	close(stopPoll)
+}
+
+// hostPort turns http://host:port[/...] into host:port for net.Dial.
+func hostPort(rawurl string) string {
+	s := strings.TrimPrefix(rawurl, "http://")
+	if i := strings.IndexByte(s, '/'); i >= 0 {
+		s = s[:i]
+	}
+	return s
+}
+
+func main() {
+	mode := flag.String("mode", "throughput", "throughput | idle | slow | slowloris")
+	url := flag.String("url", "http://127.0.0.1:8098", "server base URL")
+	path := flag.String("path", "/static", "throughput: path to hit, e.g. /static or /compute?difficulty=50")
+	n := flag.Int("n", 1000, "idle: number of keep-alive conns to open and hold")
+	c := flag.Int("c", 50, "throughput/slow/slowloris: concurrency (workers / held conns)")
+	duration := flag.Duration("duration", 10*time.Second, "how long to run / hold")
+	idlePerHost := flag.Int("idle-per-host", 0, "Transport.MaxIdleConnsPerHost (0 = auto: n for idle, c otherwise)")
+	headerTimeout := flag.Duration("header-timeout", 3*time.Second, "Transport.ResponseHeaderTimeout (0 = none)")
+	clientTimeout := flag.Duration("timeout", 0, "http.Client.Timeout (0 = none; a timeout force-closes stuck conns and hides holds)")
+	drip := flag.Duration("drip", 10*time.Second, "slowloris: interval between dribbled header bytes")
+	flag.Parse()
+
+	o := opts{
+		url: *url, path: *path, n: *n, c: *c, duration: *duration,
+		idlePerHost: *idlePerHost, headerTimeout: *headerTimeout,
+		clientTimeout: *clientTimeout, drip: *drip,
+	}
+	// Auto-size the idle pool so conns are actually reused/held, not dropped.
+	if o.idlePerHost == 0 {
+		if *mode == "idle" {
+			o.idlePerHost = o.n
+		} else {
+			o.idlePerHost = o.c
+		}
+	}
+
+	fmt.Printf("== mode=%s target=%s ==\n", *mode, o.url)
+	switch *mode {
+	case "throughput":
+		runThroughput(o)
+	case "idle":
+		runIdle(o)
+	case "slow":
+		runSlow(o)
+	case "slowloris":
+		runSlowloris(o)
+	default:
+		fmt.Fprintf(os.Stderr, "unknown -mode %q (throughput|idle|slow|slowloris)\n", *mode)
+		os.Exit(2)
+	}
+}
@@ -0,0 +1,164 @@
+// Package loadlib holds the small, pure pieces the server and load client share
+// and that are worth a unit test: a deterministic CPU burn, a percentile helper,
+// and an error classifier that says which side of a connection ran out.
+//
+// Everything network-shaped lives in the two binaries; those ceilings are
+// environmental (ulimit/sysctl/host) and are documented as pasted evidence in
+// the folder README rather than asserted in a test.
+package loadlib
+
+import (
+	"errors"
+	"net"
+	"sort"
+	"strings"
+	"time"
+)
+
+// burnIters is the inner-loop count per unit of difficulty. Kept as a named
+// constant so the test can reason about total iterations without a wall clock.
+const burnIters = 200_000
+
+// Iterations returns how many inner-loop steps Burn(difficulty) will run.
+// It is the seam the test asserts on: cost is linear in difficulty, no timing.
+func Iterations(difficulty int) int {
+	if difficulty < 1 {
+		difficulty = 1
+	}
+	return difficulty * burnIters
+}
+
+// Burn occupies a CPU for an amount of work proportional to difficulty and
+// returns a checksum. The caller must use the return value (write it to the
+// response) so the compiler cannot elide the loop.
+//
+// The mix is a cheap integer hash with a data dependency across iterations, so
+// each step depends on the last and the optimizer can't collapse the loop or
+// vectorize it away. There is deliberately no runtime.Gosched inside: Burn is
+// meant to genuinely hold a P, which is the whole point of the /compute probe
+// (how many CPU-bound requests before GOMAXPROCS saturates).
+func Burn(difficulty int) uint64 {
+	var x uint64 = 1469598103934665603 // FNV-1a offset basis, just a seed
+	n := Iterations(difficulty)
+	for i := 0; i < n; i++ {
+		x ^= uint64(i)
+		x *= 1099511628211 // FNV-1a prime
+		x = x<<13 | x>>51  // rotate so high bits feed back in
+	}
+	return x
+}
+
+// Percentile returns the p-th percentile (p in [0,1]) of an already-sorted
+// slice. Empty → 0, p<=0 → first, p>=1 → last. Nearest-rank, no interpolation —
+// good enough for "orders of magnitude" latency reporting.
+func Percentile(sorted []time.Duration, p float64) time.Duration {
+	if len(sorted) == 0 {
+		return 0
+	}
+	if p < 0 {
+		p = 0
+	}
+	if p > 1 {
+		p = 1
+	}
+	i := int(float64(len(sorted)-1) * p)
+	return sorted[i]
+}
+
+// Latencies accumulates request durations and reports percentiles, keeping the
+// client's main loop thin. Not safe for concurrent use — each worker should own
+// one and they get merged, or a single collector goroutine owns it.
+type Latencies struct {
+	d []time.Duration
+}
+
+func (l *Latencies) Add(d time.Duration) { l.d = append(l.d, d) }
+func (l *Latencies) Len() int            { return len(l.d) }
+
+// Merge folds another accumulator's samples in (for per-worker collection).
+func (l *Latencies) Merge(o *Latencies) { l.d = append(l.d, o.d...) }
+
+// Snapshot sorts and returns p50, p99, and max. Sorts in place; call once at end.
+func (l *Latencies) Snapshot() (p50, p99, max time.Duration) {
+	sort.Slice(l.d, func(i, j int) bool { return l.d[i] < l.d[j] })
+	return Percentile(l.d, 0.50), Percentile(l.d, 0.99), Percentile(l.d, 1)
+}
+
+// ErrClass names which side of the connection gave out. It turns the body-drain
+// "which error string means what" table into something the client can count.
+type ErrClass int
+
+const (
+	ErrNone             ErrClass = iota // no error
+	ErrClientFdLimit                    // this process ran out of fds (dial: too many open files)
+	ErrClientPortLimit                  // this process ran out of ephemeral ports (can't assign requested address)
+	ErrServerCantAccept                 // server can't accept (out of fds / backlog full) — we time out
+	ErrTimeout                          // request/response timed out (ambiguous, but usually the server)
+	ErrConnReset                        // peer reset/closed the connection
+	ErrRefused                          // nothing listening / server gone
+	ErrOther
+)
+
+func (c ErrClass) String() string {
+	switch c {
+	case ErrNone:
+		return "none"
+	case ErrClientFdLimit:
+		return "client-fd-limit"
+	case ErrClientPortLimit:
+		return "client-port-limit"
+	case ErrServerCantAccept:
+		return "server-cant-accept"
+	case ErrTimeout:
+		return "timeout"
+	case ErrConnReset:
+		return "conn-reset"
+	case ErrRefused:
+		return "conn-refused"
+	default:
+		return "other"
+	}
+}
+
+// Classify maps a net/http client error to the side that most likely caused it.
+// It leans on error strings because the kernel/runtime messages are the only
+// distinguishing signal for several of these; the checks are ordered most- to
+// least-specific so, e.g., "too many open files" wins over the generic timeout.
+func Classify(err error) ErrClass {
+	if err == nil {
+		return ErrNone
+	}
+	msg := strings.ToLower(err.Error())
+
+	switch {
+	case strings.Contains(msg, "too many open files"):
+		// A dial failing this way is THIS process out of fds. (A server out of
+		// fds fails on accept and never produces this string on the client.)
+		return ErrClientFdLimit
+	case strings.Contains(msg, "assign requested address"):
+		// EADDRNOTAVAIL: no ephemeral port left for a fresh 4-tuple to this
+		// server. A client-side wall, often hit before the fd limit.
+		return ErrClientPortLimit
+	case strings.Contains(msg, "connection refused"):
+		return ErrRefused
+	case strings.Contains(msg, "connection reset") || strings.Contains(msg, "broken pipe") ||
+		strings.Contains(msg, "eof"):
+		return ErrConnReset
+	}
+
+	// "awaiting response headers" / a dial timeout usually means the server
+	// accepted us into its backlog but can't service (or can't accept at all),
+	// so we sit waiting. Treat a timeout as a server-can't-keep-up signal, but
+	// keep a generic ErrTimeout for the plain deadline case.
+	var ne net.Error
+	if errors.As(err, &ne) && ne.Timeout() {
+		if strings.Contains(msg, "awaiting response headers") {
+			return ErrServerCantAccept
+		}
+		return ErrTimeout
+	}
+	if strings.Contains(msg, "timeout") || strings.Contains(msg, "deadline exceeded") {
+		return ErrTimeout
+	}
+	return ErrOther
+}
@@ -0,0 +1,147 @@
+package loadlib
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"testing"
+	"time"
+)
+
+func TestBurnDeterministic(t *testing.T) {
+	// Same difficulty must give the same checksum: the /compute endpoint returns
+	// it, and a flaky checksum would mean a non-deterministic loop.
+	for _, d := range []int{1, 3, 10} {
+		if a, b := Burn(d), Burn(d); a != b {
+			t.Errorf("Burn(%d) not deterministic: %d != %d", d, a, b)
+		}
+	}
+}
+
+func TestBurnDifficultyChangesResult(t *testing.T) {
+	// Different difficulties should not collide (they run different iteration
+	// counts); a collision here would suggest the loop body ignores i.
+	if Burn(1) == Burn(2) {
+		t.Error("Burn(1) == Burn(2): difficulty has no effect on the checksum")
+	}
+}
+
+func TestIterationsLinearAndFloored(t *testing.T) {
+	if got := Iterations(2); got != 2*burnIters {
+		t.Errorf("Iterations(2) = %d, want %d", got, 2*burnIters)
+	}
+	// difficulty < 1 is clamped to 1 so /compute?difficulty=0 still does work.
+	for _, d := range []int{0, -5} {
+		if got := Iterations(d); got != burnIters {
+			t.Errorf("Iterations(%d) = %d, want %d (floor of 1)", d, got, burnIters)
+		}
+	}
+	// Cost is monotonic in difficulty.
+	if Iterations(3) <= Iterations(2) {
+		t.Error("Iterations not increasing with difficulty")
+	}
+}
+
+func TestPercentileEdges(t *testing.T) {
+	if got := Percentile(nil, 0.5); got != 0 {
+		t.Errorf("Percentile(empty) = %v, want 0", got)
+	}
+	s := []time.Duration{10, 20, 30, 40, 50}
+	cases := []struct {
+		p    float64
+		want time.Duration
+	}{
+		{-1, 10},  // clamped to first
+		{0, 10},   // first
+		{0.5, 30}, // middle
+		{1, 50},   // last
+		{2, 50},   // clamped to last
+	}
+	for _, c := range cases {
+		if got := Percentile(s, c.p); got != c.want {
+			t.Errorf("Percentile(%v, %v) = %v, want %v", s, c.p, got, c.want)
+		}
+	}
+	if got := Percentile([]time.Duration{99}, 0.9); got != 99 {
+		t.Errorf("Percentile(single) = %v, want 99", got)
+	}
+}
+
+func TestLatenciesSnapshot(t *testing.T) {
+	var l Latencies
+	// Add out of order; Snapshot must sort before reporting.
+	for _, d := range []time.Duration{50, 10, 30, 40, 20} {
+		l.Add(d * time.Millisecond)
+	}
+	if l.Len() != 5 {
+		t.Fatalf("Len = %d, want 5", l.Len())
+	}
+	p50, p99, max := l.Snapshot()
+	if p50 != 30*time.Millisecond {
+		t.Errorf("p50 = %v, want 30ms", p50)
+	}
+	if max != 50*time.Millisecond {
+		t.Errorf("max = %v, want 50ms", max)
+	}
+	// Nearest-rank: index int(4*0.99)=3 → 40ms for this 5-sample set.
+	if p99 != 40*time.Millisecond {
+		t.Errorf("p99 = %v, want 40ms", p99)
+	}
+}
+
+func TestLatenciesMerge(t *testing.T) {
+	var a, b Latencies
+	a.Add(10 * time.Millisecond)
+	b.Add(20 * time.Millisecond)
+	a.Merge(&b)
+	if a.Len() != 2 {
+		t.Fatalf("after merge Len = %d, want 2", a.Len())
+	}
+}
+
+func TestClassify(t *testing.T) {
+	timeoutErr := &net.OpError{
+		Op:  "dial",
+		Err: timeoutError{},
+	}
+	headerTimeout := fmt.Errorf("net/http: timeout awaiting response headers")
+
+	cases := []struct {
+		name string
+		err  error
+		want ErrClass
+	}{
+		{"nil", nil, ErrNone},
+		{"client fds", errors.New("dial tcp 127.0.0.1:8098: socket: too many open files"), ErrClientFdLimit},
+		{"client ports", errors.New("dial tcp 127.0.0.1:8098: connect: can't assign requested address"), ErrClientPortLimit},
+		{"refused", errors.New("dial tcp 127.0.0.1:8098: connect: connection refused"), ErrRefused},
+		{"reset", errors.New("read tcp: connection reset by peer"), ErrConnReset},
+		{"eof", errors.New("unexpected EOF"), ErrConnReset},
+		{"plain net timeout", timeoutErr, ErrTimeout},
+		{"server cant accept", headerTimeoutNetErr{}, ErrServerCantAccept},
+		{"header timeout string", headerTimeout, ErrTimeout}, // not a net.Error, but has "timeout"
+		{"deadline", errors.New("context deadline exceeded"), ErrTimeout},
+		{"other", errors.New("something weird"), ErrOther},
+	}
+	for _, c := range cases {
+		if got := Classify(c.err); got != c.want {
+			t.Errorf("%s: Classify(%v) = %v, want %v", c.name, c.err, got, c.want)
+		}
+	}
+}
+
+// timeoutError is a minimal net.Error whose Timeout() reports true, used to build
+// a synthetic *net.OpError the way the runtime would on a real dial timeout.
+type timeoutError struct{}
+
+func (timeoutError) Error() string   { return "i/o timeout" }
+func (timeoutError) Timeout() bool   { return true }
+func (timeoutError) Temporary() bool { return true }
+
+// headerTimeoutNetErr is a net.Error whose message is the exact one net/http
+// produces when ResponseHeaderTimeout fires — the "server can't accept" signal.
+type headerTimeoutNetErr struct{}
+
+func (headerTimeoutNetErr) Error() string   { return "net/http: timeout awaiting response headers" }
+func (headerTimeoutNetErr) Timeout() bool   { return true }
+func (headerTimeoutNetErr) Temporary() bool { return true }
@@ -0,0 +1,219 @@
+// Experiment subject — a deliberately simple HTTP server whose only job is to
+// be measured. How many connections can it hold, and what breaks first?
+//
+// Endpoints:
+//
+//	GET /static              fixed tiny body, no per-request work   (the baseline)
+//	GET /compute?difficulty=N  burns CPU ∝ N, returns the checksum   (CPU-bound probe)
+//	GET /sleep?d=500ms       sleeps d, honoring client disconnect   (in-flight holder)
+//	GET /healthz             liveness
+//	GET /stats               JSON: accepted_total, active_conns, in_flight, goroutines, heap_bytes
+//
+// Observability. We separate three numbers that are easy to conflate:
+//
+//	accepted_total  every TCP conn ever accepted (only grows)
+//	active_conns    conns open right now (idle keep-alive OR busy)   — via ConnState
+//	in_flight       handlers actually running right now              — via middleware
+//
+// An idle keep-alive conn counts in active_conns but not in_flight; a running
+// /sleep counts in both. That gap is the whole point of the idle-vs-slow contrast.
+//
+// Timeout flags are the naive-vs-hardened knobs. They default to OFF so the
+// slowloris experiment works out of the box; turn them on to watch the server
+// reclaim connections:
+//
+//	-read-header-timeout 3s   drop a conn that dawdles sending headers (kills slowloris)
+//	-idle-timeout 30s         drop a keep-alive conn that goes quiet
+//	-read-timeout / -write-timeout   whole-request deadlines
+//
+// Run (terminal 1), then drive it with ./client (terminal 2):
+//
+//	go run ./98-http-server-experiments/server
+//	go run ./98-http-server-experiments/server -read-header-timeout 3s -idle-timeout 30s   # hardened
+//
+// Give the server its own fd budget so YOU decide when it runs out (not the OS
+// default of a few hundred / a few thousand). bash sets both soft and hard:
+//
+//	ulimit -Hn 4096; ulimit -Sn 4096; go run ./98-http-server-experiments/server
+//
+// (Since Go 1.19 the runtime raises the soft RLIMIT_NOFILE to the hard limit at
+// startup, so only the hard limit really caps you. zsh's `ulimit -n` sets only
+// the soft one — use `-Hn`.)
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"net"
+	"net/http"
+	"os"
+	"os/signal"
+	"runtime"
+	"strconv"
+	"sync/atomic"
+	"time"
+
+	"gosre-exercises/98-http-server-experiments/loadlib"
+)
+
+// server is the subject under test: an http.Server plus the three live counters.
+type server struct {
+	srv         *http.Server
+	ln          net.Listener
+	acceptedTot atomic.Int64 // every conn ever accepted
+	activeConns atomic.Int64 // conns open right now
+	inFlight    atomic.Int64 // handlers running right now
+}
+
+func (s *server) handleStatic(w http.ResponseWriter, r *http.Request) {
+	w.Header().Set("Content-Type", "text/plain")
+	fmt.Fprint(w, "ok\n")
+}
+
+// handleCompute burns CPU proportional to ?difficulty= and writes the checksum
+// back. Writing it matters: it forces the burn to actually happen (the compiler
+// can't prove the result is unused) and lets the client confirm the work ran.
+func (s *server) handleCompute(w http.ResponseWriter, r *http.Request) {
+	difficulty := 1
+	if v := r.URL.Query().Get("difficulty"); v != "" {
+		if n, err := strconv.Atoi(v); err == nil {
+			difficulty = n
+		}
+	}
+	sum := loadlib.Burn(difficulty)
+	w.Header().Set("Content-Type", "text/plain")
+	fmt.Fprintf(w, "difficulty=%d checksum=%d\n", difficulty, sum)
+}
+
+// handleSleep keeps the request in flight for ?d= but returns early if the
+// client disconnects or a server timeout fires — otherwise a client that hangs
+// up would leave the goroutine sleeping to full term, hiding the reclaim.
+func (s *server) handleSleep(w http.ResponseWriter, r *http.Request) {
+	d := 500 * time.Millisecond
+	if v := r.URL.Query().Get("d"); v != "" {
+		if parsed, err := time.ParseDuration(v); err == nil {
+			d = parsed
+		}
+	}
+	select {
+	case <-time.After(d):
+		w.Header().Set("Content-Type", "text/plain")
+		fmt.Fprintf(w, "slept %v\n", d)
+	case <-r.Context().Done():
+		// Client went away or a timeout fired; nothing to write.
+	}
+}
+
+type statsSnapshot struct {
+	AcceptedTotal int64  `json:"accepted_total"`
+	ActiveConns   int64  `json:"active_conns"`
+	InFlight      int64  `json:"in_flight"`
+	Goroutines    int    `json:"goroutines"`
+	HeapBytes     uint64 `json:"heap_bytes"`
+}
+
+func (s *server) snapshot() statsSnapshot {
+	var m runtime.MemStats
+	runtime.ReadMemStats(&m)
+	return statsSnapshot{
+		AcceptedTotal: s.acceptedTot.Load(),
+		ActiveConns:   s.activeConns.Load(),
+		InFlight:      s.inFlight.Load(),
+		Goroutines:    runtime.NumGoroutine(),
+		HeapBytes:     m.HeapInuse,
+	}
+}
+
+func (s *server) handleStats(w http.ResponseWriter, r *http.Request) {
+	w.Header().Set("Content-Type", "application/json")
+	json.NewEncoder(w).Encode(s.snapshot())
+}
+
+// countInFlight wraps a handler so in_flight tracks handlers actually running,
+// as distinct from conns merely open (which ConnState already counts).
+func (s *server) countInFlight(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		s.inFlight.Add(1)
+		defer s.inFlight.Add(-1)
+		next.ServeHTTP(w, r)
+	})
+}
+
+func main() {
+	addr := flag.String("addr", "127.0.0.1:8098", "address to listen on")
+	readHeaderTimeout := flag.Duration("read-header-timeout", 0, "Server.ReadHeaderTimeout (0 = off; set 3s to defeat slowloris)")
+	readTimeout := flag.Duration("read-timeout", 0, "Server.ReadTimeout (0 = off; whole-request read deadline)")
+	writeTimeout := flag.Duration("write-timeout", 0, "Server.WriteTimeout (0 = off)")
+	idleTimeout := flag.Duration("idle-timeout", 0, "Server.IdleTimeout (0 = off; set 30s to reclaim quiet keep-alives)")
+	maxHeaderBytes := flag.Int("max-header-bytes", http.DefaultMaxHeaderBytes, "Server.MaxHeaderBytes (1<<20 default)")
+	flag.Parse()
+
+	// NOTE: no -max-conns. Capping concurrent connections in the stdlib means
+	// wrapping the Listener (x/net/netutil.LimitListener) or a semaphore in the
+	// handler; we keep this zero-dependency and instead observe the ceiling the
+	// OS imposes (fds/backlog). See the README for the stdlib-only pattern.
+
+	ln, err := net.Listen("tcp", *addr)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "listen:", err)
+		os.Exit(1)
+	}
+	s := &server{ln: ln}
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/static", s.handleStatic)
+	mux.HandleFunc("/compute", s.handleCompute)
+	mux.HandleFunc("/sleep", s.handleSleep)
+	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "ok") })
+	mux.HandleFunc("/stats", s.handleStats)
+
+	s.srv = &http.Server{
+		Handler:           s.countInFlight(mux),
+		ReadHeaderTimeout: *readHeaderTimeout,
+		ReadTimeout:       *readTimeout,
+		WriteTimeout:      *writeTimeout,
+		IdleTimeout:       *idleTimeout,
+		MaxHeaderBytes:    *maxHeaderBytes,
+		ConnState: func(c net.Conn, st http.ConnState) {
+			switch st {
+			case http.StateNew:
+				s.acceptedTot.Add(1)
+				s.activeConns.Add(1)
+			case http.StateClosed, http.StateHijacked:
+				s.activeConns.Add(-1)
+			}
+		},
+	}
+
+	go func() {
+		if err := s.srv.Serve(ln); err != nil && err != http.ErrServerClosed {
+			fmt.Fprintln(os.Stderr, "serve:", err)
+		}
+	}()
+
+	fmt.Printf("serving on http://%s  (timeouts: readHeader=%v read=%v write=%v idle=%v)\n",
+		ln.Addr(), *readHeaderTimeout, *readTimeout, *writeTimeout, *idleTimeout)
+	fmt.Println("point the client at it:  go run ./98-http-server-experiments/client -mode idle -n 2000")
+
+	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
+	defer stop()
+	tick := time.NewTicker(time.Second)
+	defer tick.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			fmt.Println("\nshutting down")
+			shutCtx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+			defer cancel()
+			s.srv.Shutdown(shutCtx)
+			return
+		case <-tick.C:
+			snap := s.snapshot()
+			fmt.Printf("%s  accepted_total=%d active=%d in_flight=%d goroutines=%d heap=%dMiB\n",
+				time.Now().Format("15:04:05"), snap.AcceptedTotal, snap.ActiveConns,
+				snap.InFlight, snap.Goroutines, snap.HeapBytes>>20)
+		}
+	}
+}