Add 98-http-server-experiments: how many connections can a Go server hold

One stdlib server under test (/static /compute /sleep /stats, ConnState and in-flight counters) plus a load client with four modes: throughput, idle, slow, and slowloris. loadlib holds the tested core (Burn, Percentile, Classify). README documents the runs and how to tell a real server ceiling from the load generator giving out first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012MpK9niZm4jcs3mAJbxMfi
2026-09-02 09:34:12 +02:00
parent b290957fc9
commit f08d4f44d4
5 changed files with 1300 additions and 0 deletions
@@ -0,0 +1,283 @@
 # 98 — HTTP server connection experiments
 How many connections can a Go HTTP server hold, and what gives out first? This
 folder finds the **order of magnitude** (not a portable constant) for four
 different failure modes, and — just as important — teaches you to tell a real
 server ceiling from your own load generator running out first.
 One simple server is the subject under test; one load client drives it in four
 modes. Run them in two terminals so each side has its own fd budget.
 ```
 98-http-server-experiments/
  server/main.go     subject under test: /static /compute /sleep /stats, ConnState + in_flight counters
  client/main.go     load generator: -mode throughput | idle | slow | slowloris
  loadlib/           tested core: Burn (CPU), Percentile, Classify (which side ran out)
  README.md          this file — decisions log + pasted evidence
 ```
 Pure stdlib, zero dependencies. Part of the `gosre-exercises` module, so:
 ```sh
 go run ./98-http-server-experiments/server        # terminal 1
 go run ./98-http-server-experiments/client -mode idle -n 2000   # terminal 2
 go test -race ./98-http-server-experiments/loadlib/
 ```
 ## Run it
 **Terminal 1 — the server.** Give it its own fd budget so *you* decide when it
 runs out, not the OS default. bash sets both soft and hard limits:
 ```sh
 ulimit -Hn 4096; ulimit -Sn 4096
 go run ./98-http-server-experiments/server                       # naive: no timeouts
 go run ./98-http-server-experiments/server -read-header-timeout 3s -idle-timeout 30s   # hardened
 ```
 (Since Go 1.19 the runtime raises the soft `RLIMIT_NOFILE` to the hard limit at
 startup, so only the **hard** limit really caps you. bash's `ulimit -n` sets
 both; zsh's sets only the soft one — use `-Hn`.)
 The server prints one line per second:
 ```
 19:12:20  accepted_total=10001 active=10001 in_flight=1 goroutines=10006 heap=107MiB
 ```
 Three numbers that are easy to conflate:
 | number | meaning | via |
 |---|---|---|
 | `accepted_total` | every TCP conn ever accepted (only grows) | `ConnState` StateNew |
 | `active` | conns open **right now** — idle keep-alive *or* busy | `ConnState` new − closed |
 | `in_flight` | handlers **actually running** right now | middleware around the mux |
 An idle keep-alive counts in `active` but not `in_flight`; a running `/sleep`
 counts in both. That gap is the whole point of the idle-vs-slow contrast.
 **Terminal 2 — the client.** Give it *its own* fd budget, separate from the server:
 ```sh
 ulimit -Hn 64; ulimit -Sn 64
 go run ./98-http-server-experiments/client -mode idle -n 200
 ```
 ## The four dimensions
 ### 1. Idle connection ceiling — `-mode idle`
 Open `-n` keep-alive conns and hold them idle. Uses **raw `net.Conn`**, not an
 `http.Client` — with a fast endpoint the client's connection pool reuses conns
 faster than you dial, so N concurrent GETs do *not* reliably produce N
 simultaneous conns. Dialing raw and keeping each fd open guarantees exactly N.
 ```sh
 go run ./98-http-server-experiments/client -mode idle -n 10000 -duration 30s
 ```
 ```
 == mode=idle target=http://127.0.0.1:8098 ==
  server before      : active=1 in_flight=1 goroutines=6 heap=0MiB
  opened             : 10000/10000 idle conns held, 0 failed
  client goroutines  : before=3 after=3 (delta +0; raw fds, no per-conn goroutine)
  server after open  : active=10001 in_flight=1 goroutines=10006 heap=107MiB  ← active≈n, in_flight≈0
 ```
 **Reading it:** 10 000 idle conns cost the server ~107 MiB (from a <1 MiB
 baseline) and one goroutine each → **order of magnitude ~10 KiB per idle
 connection**. A million idle keep-alives would be ~10 GiB and a million
 goroutines — the server holds them fine until it runs out of fds or RAM, not CPU.
 Note the client's goroutine delta is **0**: raw fds have no per-conn goroutine,
 unlike `http.Transport`'s pool (2 goroutines per pooled conn).
 ### 2. Active slow requests — `-mode slow`
 Fire `-c` concurrent `/sleep` requests and hold them *all in flight* for the
 duration. Same conn count as idle, but now each pins a **running** server
 goroutine. Prints the server's `in_flight` each second.
 ```sh
 go run ./98-http-server-experiments/client -mode slow -c 500 -duration 10s
 ```
 ```
 == mode=slow target=http://127.0.0.1:8098 ==
  firing 500 concurrent /sleep?d=10s ...
    server 01:13:13  active=501 in_flight=501 goroutines=1006 heap=25MiB
  requests           : 500 ok, 0 failed (of 500 held in flight)
  errors             : none
 ```
 **Reading it:** `in_flight=501` (the 500 sleepers + the stats poll) vs idle's
 `in_flight=1` — that is the difference between "conn open" and "handler running."
 Each in-flight request is one goroutine on a goroutine-per-request server; 500
 held costs ~1000 goroutines and a few MiB. Push `-c` up in orders of magnitude to
 find where the *server* (goroutine stacks, fds) — not the client — gives out.
 `/sleep` honors `r.Context()`, so a client that hangs up or a server timeout
 frees the goroutine immediately rather than sleeping to term.
 ### 3. Timeouts & slowloris — `-mode slowloris`
 Open `-c` raw TCP conns, send a request line + one header on each, then dribble a
 bogus header every `-drip` and **never** send the terminating blank line. The
 handler never starts. The only thing that reclaims these conns is the server's
 `ReadHeaderTimeout`.
 **Naive server** (`go run .../server`, no timeouts):
 ```
 == mode=slowloris target=http://127.0.0.1:8098 ==
  opening 200 slowloris conns ...
    server 01:13:26  active=201 in_flight=1 goroutines=206 heap=26MiB   ← stuck, handler never runs
  reclaimed by server: 0
  (0 reclaimed: the server has no -read-header-timeout — these conns would stick until fds run out)
 ```
 **Hardened server** (`go run .../server -read-header-timeout 3s`):
 ```
    server 01:13:40  active=201 in_flight=1 goroutines=206 heap=2MiB
    server 01:13:42  active=1   in_flight=1 goroutines=6   heap=3MiB    ← reclaimed ~3s after opening
  reclaimed by server: 200 (server closed the conn mid-headers → -read-header-timeout worked)
 ```
 **Reading it:** with no `ReadHeaderTimeout`, 200 clients doing *nothing* consume
 200 connections indefinitely — a handful of hosts can exhaust a server's fds
 without ever completing a request. Setting `ReadHeaderTimeout=3s` drops each conn
 ~3 s after its last progress and `active` stays flat. The same logic applies to
 `IdleTimeout` for *completed* keep-alives that then go quiet. **Every production
 `http.Server` should set these; the stdlib defaults are all zero (off).**
 ### 4. Client-side limits — when the measurement is lying
 The load generator hits its own walls, often *before* the server does. Push idle
 past the ephemeral-port ceiling and watch the **client** fail while the server is
 fine:
 ```sh
 go run ./98-http-server-experiments/client -mode idle -n 20000 -duration 5s
 ```
 ```
  opened             : 16312/20000 idle conns held, 3688 failed
  errors             : client-port-limit=3688
  server after open  : active=16313 in_flight=1 goroutines=16318 heap=199MiB
 ```
 **Reading it:** this host has 16 384 ephemeral ports (`49152–65535`), and every
 conn to the same server `ip:port` needs a distinct one. At 16 312 the client got
 `connect: can't assign requested address` — a **client-side** wall. The server
 was untroubled (it held 16 313 conns on ~199 MiB). If you didn't cross-check the
 server's `/stats`, you might wrongly conclude "the server maxes out at ~16 k."
 Every mode fetches `/stats` at start and end for exactly this reason.
 ## What to watch
 - **`active` vs `in_flight`** — open connections vs running handlers (dimensions 1 & 2).
 - **`goroutines`** — ~1 per server conn, ~2 per pooled *client* conn, ~0 per raw client conn.
 - **`heap`** — server memory; ~10 KiB per idle conn on this host.
 - **`reused` vs `fresh`** (throughput) — is the pool actually reusing conns, or dialing every time?
 - **`p50/p99/max`** latency and **error histogram** classified by which side ran out.
 The throughput control shows how fast it goes when nothing is broken, and how a
 CPU-bound endpoint plateaus near `GOMAXPROCS`:
 ```sh
 go run ./98-http-server-experiments/client -mode throughput -path /static -c 50 -duration 3s
 #   requests : 445013 ok, 0 failed  → 148338 req/s   latency p50=259µs p99=1.4ms   reused=444971 fresh=42
 go run ./98-http-server-experiments/client -mode throughput -path "/compute?difficulty=200" -c 50 -duration 3s
 #   requests : 370 ok, 0 failed  → 123 req/s   latency p50=393ms p99=1.24s
 ```
 150 k req/s on `/static` collapses to ~120 req/s on `/compute?difficulty=200`:
 that's CPU-bound, not connection-bound — 8 cores fully busy, more concurrency
 just deepens the queue.
 ## Reading it — symptom → cause
 | symptom | cause |
 |---|---|
 | client `dial: socket: too many open files` | **client** fd/ulimit — raise the client's limit; the measurement was client-bound |
 | client `connect: can't assign requested address` | **client** ephemeral ports exhausted — reuse conns, or use multiple client IPs |
 | server log `accept: too many open files` | **server** fd ceiling — the real thing |
 | `timeout awaiting response headers` / `connection reset` | server can't accept/service; client is fine |
 | slowloris conns stick forever | server has no `ReadHeaderTimeout` |
 | idle keep-alives never drop | server has no `IdleTimeout` |
 | throughput plateaus near `GOMAXPROCS` on `/compute` | CPU-bound, not conn-bound |
 The `Classify` helper in `loadlib` turns these strings into counted classes so
 the client's error histogram tells you which side gave out.
 ## Client-side pitfalls (know when the number is yours, not the server's)
 - **Ephemeral port exhaustion.** One client → one server `ip:port` gets ~16 k–28 k
  ephemeral ports (`net.inet.ip.portrange.first/last` on macOS,
  `net.ipv4.ip_local_port_range` on Linux). Above that, `connect` fails. Fix for
  the *test*: reuse keep-alive conns (don't open a fresh 4-tuple per request), or
  drive from several client IPs/hosts.
 - **TIME_WAIT.** Every actively-closed conn sits in TIME_WAIT ~60 s holding its
  4-tuple; a close-per-request pattern piles them up
  (`netstat -an | grep -c TIME_WAIT`) and starves ephemeral ports. Keep-alive
  reuse avoids it. `SO_REUSEADDR` does not help the client here.
 - **`MaxIdleConnsPerHost` (default 2).** If it's below your concurrency, the
  client dials fresh every round and you measure TCP handshakes, not the server.
  The client auto-sizes it to `-c`; override with `-idle-per-host`.
 - **`Client.Timeout` / `ResponseHeaderTimeout`.** A timeout force-closes a stuck
  conn, which *hides* a hold or a leak. The client defaults `-timeout 0`; `slow`
  mode disables the header timeout entirely because its responses are slow **by
  design** (an early version misreported held requests as "server-cant-accept").
 - **Always cross-check `/stats`.** If the client says "failed" but the server
  never saw the conn, the client ran out first.
 ## Host caveats (numbers here are not portable)
 Measured on: Apple M-series (Mac15,12), 8 cores, macOS (darwin/arm64), go 1.27.
 `kern.maxfilesperproc=92160`, ephemeral ports `49152–65535` (16 384).
 - **macOS vs Linux differ.** fd caps (`kern.maxfilesperproc` / `ulimit`), TIME_WAIT
  and port-range sysctls, and even `accept`/`dial` error strings are not the same.
 - **Listen backlog** caps the accept queue independently of your handler:
  `kern.ipc.somaxconn` (macOS) / `net.core.somaxconn` (Linux).
 - **One process, shared fd limit.** If you run both roles in one process they
  share the fd budget and the server usually dies first — use two terminals.
 ## Decisions log (what we built, how, and why)
 - **One server binary + one client binary; scenarios are `-mode`, not separate
  binaries.** The four dimensions stress the *same* server; splitting them would
  duplicate the transport/httptrace/reporting scaffolding four times. Two-process
  split so each side gets its own fd/ulimit (same rationale as `99-experiments/body-drain`'s `-role`).
 - **Three live counters, not one.** `active` (ConnState) and `in_flight`
  (middleware) are tracked separately because "conn open" ≠ "handler running" —
  that distinction *is* the idle-vs-slow experiment.
 - **`idle` mode uses raw `net.Conn`, not `http.Client`.** The pool reuses conns
  faster than we dial against a fast endpoint, so a pool-based idle mode
  under-counts non-deterministically (20 000 GETs held only ~8 700). Raw dials
  give exactly N — and, as a bonus, expose ephemeral-port exhaustion cleanly.
 - **`slowloris` uses raw `net.Conn`.** The stdlib client won't send a deliberately
  incomplete request, so we speak HTTP by hand: request line + headers, never the
  terminating blank line.
 - **`Burn` is deterministic and non-elidable.** A data-dependent integer hash of
  `difficulty × 200k` iterations; the handler writes the checksum so the compiler
  can't drop the loop. No `runtime.Gosched`, so it genuinely holds a P. Tested by
  iteration count, not wall-clock, to stay non-flaky.
 - **Zero dependencies.** No `x/net/netutil.LimitListener` for `-max-conns`; a real
  server caps concurrency with a semaphore-guarded handler or a wrapped listener,
  and we'd rather observe the OS ceiling than hide it.
 - **Server timeouts default to OFF.** So the slowloris vulnerability is
  demonstrable out of the box; turn `-read-header-timeout` / `-idle-timeout` on to
  watch the server reclaim.
 - **Network ceilings are not unit-tested.** They're environmental (ulimit, sysctl,
  host); only `Burn`, `Percentile`, and `Classify` are tested. The ceilings live
  here as pasted evidence.
 ## References
 - `net/http.Server` timeout fields — https://pkg.go.dev/net/http#Server
 - Cloudflare, "The complete guide to Go net/http timeouts" — https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/
 - `net/http/httptrace` — https://pkg.go.dev/net/http/httptrace
 - `http.ConnState` — https://pkg.go.dev/net/http#ConnState
 - Slowloris (CWE-400 / low-and-slow) — https://en.wikipedia.org/wiki/Slowloris_(cyber_attack)
 - Ephemeral ports & TIME_WAIT — https://www.rfc-editor.org/rfc/rfc6335, `sysctl net.inet.ip.portrange` / `net.ipv4.ip_local_port_range`
@@ -0,0 +1,487 @@
 // Load generator for the experiment server. One binary, four modes, each a
 // different way to ask "how many connections, and what gives out first?".
 //
 // Modes (-mode):
 //
 //	throughput  closed loop: -c workers hammer -path for -duration, reusing conns.
 //	            The control — how fast when nothing is broken. Point -path at
 //	            /compute?difficulty=N to watch it plateau near GOMAXPROCS.
 //	idle        open -n keep-alive conns to /static and HOLD them (don't close).
 //	            Measures the cost of an idle connection: fds, goroutines, memory.
 //	slow        fire -c concurrent /sleep requests and hold them all in flight.
 //	            Same conn count as idle, but each pins a running server goroutine.
 //	slowloris   -c raw TCP conns that dribble headers forever and never finish
 //	            the request. Defeated only by the server's -read-header-timeout.
 //
 // Every mode cross-checks the server's own /stats so you can tell a real server
 // ceiling from your own client running out first. Which error you get tells you
 // which side gave out:
 //
 //	dial ...: socket: too many open files   → THIS process (raise the client's ulimit)
 //	timeout awaiting response headers        → the server can't accept
 //	connection reset by peer                 → the server closed on you (or a timeout fired)
 //	connection refused                       → nothing listening
 //
 // Give the client its own fd budget, separate from the server's:
 //
 //	ulimit -Hn 64; ulimit -Sn 64; go run ./98-http-server-experiments/client -mode idle -n 200
 //
 // (bash sets soft+hard; zsh only soft — use -Hn. Since Go 1.19 only the hard
 // limit really caps you.)
 package main
 import (
 	"bufio"
 	"context"
 	"encoding/json"
 	"flag"
 	"fmt"
 	"io"
 	"net"
 	"net/http"
 	"net/http/httptrace"
 	"os"
 	"os/signal"
 	"runtime"
 	"strings"
 	"sync"
 	"sync/atomic"
 	"time"
 	"gosre-exercises/98-http-server-experiments/loadlib"
 )
 // serverStats mirrors the server's /stats JSON for the cross-check.
 type serverStats struct {
 	AcceptedTotal int64  `json:"accepted_total"`
 	ActiveConns   int64  `json:"active_conns"`
 	InFlight      int64  `json:"in_flight"`
 	Goroutines    int    `json:"goroutines"`
 	HeapBytes     uint64 `json:"heap_bytes"`
 }
 func (s serverStats) String() string {
 	return fmt.Sprintf("active=%d in_flight=%d goroutines=%d heap=%dMiB",
 		s.ActiveConns, s.InFlight, s.Goroutines, s.HeapBytes>>20)
 }
 // fetchStats reads the server's /stats over a throwaway conn (own transport so it
 // never borrows from or perturbs the pool we're measuring).
 func fetchStats(base string) (serverStats, error) {
 	var st serverStats
 	c := &http.Client{Timeout: 2 * time.Second}
 	resp, err := c.Get(base + "/stats")
 	if err != nil {
 		return st, err
 	}
 	defer resp.Body.Close()
 	err = json.NewDecoder(resp.Body).Decode(&st)
 	return st, err
 }
 // opts bundle the CLI knobs the modes read.
 type opts struct {
 	url, path     string
 	n, c          int
 	duration      time.Duration
 	idlePerHost   int
 	headerTimeout time.Duration
 	clientTimeout time.Duration
 	drip          time.Duration
 }
 // newTransport builds a fresh pool so runs don't share state. idlePerHost must be
 // >= the concurrency you want reused, or conns get dropped instead of pooled.
 func newTransport(o opts) *http.Transport {
 	tr := http.DefaultTransport.(*http.Transport).Clone()
 	tr.MaxIdleConns = o.n + o.c
 	tr.MaxIdleConnsPerHost = o.idlePerHost
 	tr.ResponseHeaderTimeout = o.headerTimeout
 	return tr
 }
 // errHistogram counts failures by which side they blame.
 type errHistogram struct {
 	mu sync.Mutex
 	m  map[loadlib.ErrClass]int
 }
 func newErrHistogram() *errHistogram { return &errHistogram{m: map[loadlib.ErrClass]int{}} }
 func (h *errHistogram) add(err error) {
 	h.mu.Lock()
 	h.m[loadlib.Classify(err)]++
 	h.mu.Unlock()
 }
 func (h *errHistogram) String() string {
 	h.mu.Lock()
 	defer h.mu.Unlock()
 	if len(h.m) == 0 {
 		return "none"
 	}
 	s := ""
 	for cls, n := range h.m {
 		s += fmt.Sprintf("%s=%d ", cls, n)
 	}
 	return s
 }
 // ---- mode: throughput ----
 // runThroughput drives a closed loop: c workers each loop Do()+drain for the
 // duration. Conns are reused, so this measures service speed, not dial cost.
 func runThroughput(o opts) {
 	tr := newTransport(o)
 	defer tr.CloseIdleConnections()
 	var reused, fresh, ok, failed atomic.Int64
 	trace := &httptrace.ClientTrace{GotConn: func(i httptrace.GotConnInfo) {
 		if i.Reused {
 			reused.Add(1)
 		} else {
 			fresh.Add(1)
 		}
 	}}
 	ctx := httptrace.WithClientTrace(context.Background(), trace)
 	deadline := time.Now().Add(o.duration)
 	client := &http.Client{Transport: tr, Timeout: o.clientTimeout}
 	hist := newErrHistogram()
 	target := o.url + o.path
 	var wg sync.WaitGroup
 	var lats []loadlib.Latencies
 	lats = make([]loadlib.Latencies, o.c)
 	for w := 0; w < o.c; w++ {
 		wg.Add(1)
 		go func(w int) {
 			defer wg.Done()
 			for time.Now().Before(deadline) {
 				start := time.Now()
 				req, _ := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
 				resp, err := client.Do(req)
 				if err != nil {
 					failed.Add(1)
 					hist.add(err)
 					continue
 				}
 				io.Copy(io.Discard, resp.Body)
 				resp.Body.Close()
 				lats[w].Add(time.Since(start))
 				ok.Add(1)
 			}
 		}(w)
 	}
 	wg.Wait()
 	var all loadlib.Latencies
 	for i := range lats {
 		all.Merge(&lats[i])
 	}
 	p50, p99, max := all.Snapshot()
 	rps := float64(ok.Load()) / o.duration.Seconds()
 	fmt.Printf("  target             : %s (concurrency %d, %v)\n", target, o.c, o.duration)
 	fmt.Printf("  requests           : %d ok, %d failed  → %.0f req/s\n", ok.Load(), failed.Load(), rps)
 	fmt.Printf("  latency            : p50=%v p99=%v max=%v\n", p50.Round(time.Microsecond), p99.Round(time.Microsecond), max.Round(time.Microsecond))
 	fmt.Printf("  conns              : reused=%d fresh=%d\n", reused.Load(), fresh.Load())
 	fmt.Printf("  errors             : %s\n", hist)
 }
 // ---- mode: idle ----
 // runIdle opens n keep-alive conns and HOLDS them idle, then reports the server's
 // view. It uses raw net.Conn rather than http.Client on purpose: with a fast
 // endpoint the client's connection pool reuses conns faster than we dial, so n
 // concurrent GETs do NOT reliably produce n simultaneous conns. Dialing raw and
 // keeping each fd open guarantees exactly n idle keep-alives — the number we want
 // to be honest about. Each does one real GET /static (so the server completes a
 // request and parks the conn as idle: active=n, in_flight≈0), then we sit on it.
 func runIdle(o opts) {
 	before, _ := fetchStats(o.url)
 	fmt.Printf("  server before      : %s\n", before)
 	host := hostPort(o.url)
 	grBefore := runtime.NumGoroutine()
 	var opened, failed atomic.Int64
 	hist := newErrHistogram()
 	conns := make([]net.Conn, 0, o.n)
 	var mu sync.Mutex
 	// Bounded dial concurrency: fast enough to open thousands quickly, without a
 	// thundering herd of goroutines. Each conn, once its response is drained,
 	// just sits open — no per-conn goroutine, unlike the transport's 2-per-conn.
 	sem := make(chan struct{}, 256)
 	var wg sync.WaitGroup
 	for i := 0; i < o.n; i++ {
 		sem <- struct{}{}
 		wg.Add(1)
 		go func() {
 			defer wg.Done()
 			defer func() { <-sem }()
 			conn, err := net.DialTimeout("tcp", host, 5*time.Second)
 			if err != nil {
 				failed.Add(1)
 				hist.add(err)
 				return
 			}
 			// One real keep-alive request, response drained, conn left open.
 			if _, err := fmt.Fprintf(conn, "GET /static HTTP/1.1\r\nHost: %s\r\nConnection: keep-alive\r\n\r\n", host); err != nil {
 				failed.Add(1)
 				hist.add(err)
 				conn.Close()
 				return
 			}
 			br := bufio.NewReader(conn)
 			resp, err := http.ReadResponse(br, nil)
 			if err != nil {
 				failed.Add(1)
 				hist.add(err)
 				conn.Close()
 				return
 			}
 			io.Copy(io.Discard, resp.Body)
 			resp.Body.Close()
 			mu.Lock()
 			conns = append(conns, conn)
 			mu.Unlock()
 			opened.Add(1)
 		}()
 	}
 	wg.Wait()
 	time.Sleep(100 * time.Millisecond) // let the server finish parking them as idle
 	peak, _ := fetchStats(o.url)
 	fmt.Printf("  opened             : %d/%d idle conns held, %d failed\n", opened.Load(), o.n, failed.Load())
 	fmt.Printf("  errors             : %s\n", hist)
 	fmt.Printf("  client goroutines  : before=%d after=%d (delta %+d; raw fds, no per-conn goroutine)\n",
 		grBefore, runtime.NumGoroutine(), runtime.NumGoroutine()-grBefore)
 	fmt.Printf("  server after open  : %s  ← active≈n, in_flight≈0: these conns are OPEN but no handler runs\n", peak)
 	fmt.Printf("  holding %d idle conns for %v — watch the server's active= stay put. Ctrl-C to release early.\n", opened.Load(), o.duration)
 	holdFor(o.url, o.duration)
 	for _, c := range conns {
 		c.Close()
 	}
 	time.Sleep(200 * time.Millisecond)
 	after, _ := fetchStats(o.url)
 	fmt.Printf("  server after close : %s (active should fall back toward its baseline)\n", after)
 }
 // ---- mode: slow ----
 // runSlow fires c concurrent /sleep requests that each stay in flight for the
 // whole duration, so c server goroutines are pinned at once. Prints the server's
 // in_flight/goroutines each second while they're held.
 func runSlow(o opts) {
 	before, _ := fetchStats(o.url)
 	fmt.Printf("  server before      : %s\n", before)
 	// By design every response is slow (it arrives only after the sleep), so we
 	// must NOT apply a response-header timeout here — it would abort legitimate
 	// slow responses and lie that the server "can't accept". Instead we bound
 	// each request with a context deadline a bit past the sleep, so a truly
 	// stuck request still unblocks wg.Wait() and gets reported.
 	tr := newTransport(o)
 	tr.ResponseHeaderTimeout = 0
 	defer tr.CloseIdleConnections()
 	client := &http.Client{Transport: tr}
 	target := fmt.Sprintf("%s/sleep?d=%s", o.url, o.duration)
 	reqTimeout := o.duration + 10*time.Second
 	var ok, failed atomic.Int64
 	hist := newErrHistogram()
 	stopPoll := make(chan struct{})
 	go pollStats(o.url, stopPoll)
 	fmt.Printf("  firing %d concurrent /sleep?d=%v ...\n", o.c, o.duration)
 	var wg sync.WaitGroup
 	for i := 0; i < o.c; i++ {
 		wg.Add(1)
 		go func() {
 			defer wg.Done()
 			ctx, cancel := context.WithTimeout(context.Background(), reqTimeout)
 			defer cancel()
 			req, _ := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
 			resp, err := client.Do(req)
 			if err != nil {
 				failed.Add(1)
 				hist.add(err)
 				return
 			}
 			io.Copy(io.Discard, resp.Body)
 			resp.Body.Close()
 			ok.Add(1)
 		}()
 	}
 	wg.Wait()
 	close(stopPoll)
 	fmt.Printf("  requests           : %d ok, %d failed (of %d held in flight)\n", ok.Load(), failed.Load(), o.c)
 	fmt.Printf("  errors             : %s\n", hist)
 }
 // ---- mode: slowloris ----
 // runSlowloris opens c raw TCP conns, sends a request line + one header on each,
 // then dribbles a bogus header every -drip and NEVER sends the terminating blank
 // line. The handler never starts. A server with -read-header-timeout closes the
 // conn after that timeout; we detect it as a read returning EOF/reset and count
 // how many the server reclaimed.
 func runSlowloris(o opts) {
 	host := hostPort(o.url)
 	before, _ := fetchStats(o.url)
 	fmt.Printf("  server before      : %s\n", before)
 	fmt.Printf("  opening %d slowloris conns to %s (drip every %v) ...\n", o.c, host, o.drip)
 	var held, reclaimed, dialFailed atomic.Int64
 	hist := newErrHistogram()
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
 	defer stop()
 	var wg sync.WaitGroup
 	for i := 0; i < o.c; i++ {
 		conn, err := net.DialTimeout("tcp", host, 5*time.Second)
 		if err != nil {
 			dialFailed.Add(1)
 			hist.add(err)
 			continue
 		}
 		held.Add(1)
 		wg.Add(1)
 		go func(conn net.Conn) {
 			defer wg.Done()
 			defer conn.Close()
 			// Incomplete request: a valid start line and Host, but no blank line.
 			fmt.Fprintf(conn, "GET /static HTTP/1.1\r\nHost: %s\r\n", host)
 			dripTick := time.NewTicker(o.drip)
 			defer dripTick.Stop()
 			// A reader goroutine turns a server-side close into a signal.
 			closed := make(chan struct{})
 			go func() {
 				buf := make([]byte, 1)
 				conn.Read(buf) // blocks until the server closes → EOF/reset
 				close(closed)
 			}()
 			for {
 				select {
 				case <-ctx.Done():
 					return
 				case <-closed:
 					reclaimed.Add(1)
 					return
 				case <-dripTick.C:
 					// One more bogus header — still no terminating blank line.
 					if _, err := fmt.Fprint(conn, "X-Keep: alive\r\n"); err != nil {
 						reclaimed.Add(1)
 						return
 					}
 				}
 			}
 		}(conn)
 	}
 	// Report the server's view periodically while conns are held.
 	stopPoll := make(chan struct{})
 	go pollStats(o.url, stopPoll)
 	select {
 	case <-ctx.Done():
 	case <-time.After(o.duration):
 	}
 	stop()          // stop the drip loops
 	close(stopPoll) // stop polling
 	wg.Wait()
 	after, _ := fetchStats(o.url)
 	fmt.Printf("  held               : %d conns opened, %d dial-failed\n", held.Load(), dialFailed.Load())
 	fmt.Printf("  reclaimed by server: %d (server closed the conn mid-headers → -read-header-timeout worked)\n", reclaimed.Load())
 	fmt.Printf("  errors             : %s\n", hist)
 	fmt.Printf("  server after       : %s\n", after)
 	if reclaimed.Load() == 0 {
 		fmt.Println("  (0 reclaimed: the server has no -read-header-timeout — these conns would stick until fds run out)")
 	}
 }
 // ---- helpers ----
 // pollStats prints the server's /stats once a second until stop is closed.
 func pollStats(base string, stop <-chan struct{}) {
 	tick := time.NewTicker(time.Second)
 	defer tick.Stop()
 	for {
 		select {
 		case <-stop:
 			return
 		case <-tick.C:
 			if st, err := fetchStats(base); err == nil {
 				fmt.Printf("    server %s  %s\n", time.Now().Format("15:04:05"), st)
 			}
 		}
 	}
 }
 // holdFor blocks for d or until Ctrl-C, printing the server's view each second so
 // you can watch the held conns sit there.
 func holdFor(base string, d time.Duration) {
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
 	defer stop()
 	stopPoll := make(chan struct{})
 	go pollStats(base, stopPoll)
 	select {
 	case <-ctx.Done():
 	case <-time.After(d):
 	}
 	close(stopPoll)
 }
 // hostPort turns http://host:port[/...] into host:port for net.Dial.
 func hostPort(rawurl string) string {
 	s := strings.TrimPrefix(rawurl, "http://")
 	if i := strings.IndexByte(s, '/'); i >= 0 {
 		s = s[:i]
 	}
 	return s
 }
 func main() {
 	mode := flag.String("mode", "throughput", "throughput | idle | slow | slowloris")
 	url := flag.String("url", "http://127.0.0.1:8098", "server base URL")
 	path := flag.String("path", "/static", "throughput: path to hit, e.g. /static or /compute?difficulty=50")
 	n := flag.Int("n", 1000, "idle: number of keep-alive conns to open and hold")
 	c := flag.Int("c", 50, "throughput/slow/slowloris: concurrency (workers / held conns)")
 	duration := flag.Duration("duration", 10*time.Second, "how long to run / hold")
 	idlePerHost := flag.Int("idle-per-host", 0, "Transport.MaxIdleConnsPerHost (0 = auto: n for idle, c otherwise)")
 	headerTimeout := flag.Duration("header-timeout", 3*time.Second, "Transport.ResponseHeaderTimeout (0 = none)")
 	clientTimeout := flag.Duration("timeout", 0, "http.Client.Timeout (0 = none; a timeout force-closes stuck conns and hides holds)")
 	drip := flag.Duration("drip", 10*time.Second, "slowloris: interval between dribbled header bytes")
 	flag.Parse()
 	o := opts{
 		url: *url, path: *path, n: *n, c: *c, duration: *duration,
 		idlePerHost: *idlePerHost, headerTimeout: *headerTimeout,
 		clientTimeout: *clientTimeout, drip: *drip,
 	}
 	// Auto-size the idle pool so conns are actually reused/held, not dropped.
 	if o.idlePerHost == 0 {
 		if *mode == "idle" {
 			o.idlePerHost = o.n
 		} else {
 			o.idlePerHost = o.c
 		}
 	}
 	fmt.Printf("== mode=%s target=%s ==\n", *mode, o.url)
 	switch *mode {
 	case "throughput":
 		runThroughput(o)
 	case "idle":
 		runIdle(o)
 	case "slow":
 		runSlow(o)
 	case "slowloris":
 		runSlowloris(o)
 	default:
 		fmt.Fprintf(os.Stderr, "unknown -mode %q (throughput|idle|slow|slowloris)\n", *mode)
 		os.Exit(2)
 	}
 }
@@ -0,0 +1,164 @@
 // Package loadlib holds the small, pure pieces the server and load client share
 // and that are worth a unit test: a deterministic CPU burn, a percentile helper,
 // and an error classifier that says which side of a connection ran out.
 //
 // Everything network-shaped lives in the two binaries; those ceilings are
 // environmental (ulimit/sysctl/host) and are documented as pasted evidence in
 // the folder README rather than asserted in a test.
 package loadlib
 import (
 	"errors"
 	"net"
 	"sort"
 	"strings"
 	"time"
 )
 // burnIters is the inner-loop count per unit of difficulty. Kept as a named
 // constant so the test can reason about total iterations without a wall clock.
 const burnIters = 200_000
 // Iterations returns how many inner-loop steps Burn(difficulty) will run.
 // It is the seam the test asserts on: cost is linear in difficulty, no timing.
 func Iterations(difficulty int) int {
 	if difficulty < 1 {
 		difficulty = 1
 	}
 	return difficulty * burnIters
 }
 // Burn occupies a CPU for an amount of work proportional to difficulty and
 // returns a checksum. The caller must use the return value (write it to the
 // response) so the compiler cannot elide the loop.
 //
 // The mix is a cheap integer hash with a data dependency across iterations, so
 // each step depends on the last and the optimizer can't collapse the loop or
 // vectorize it away. There is deliberately no runtime.Gosched inside: Burn is
 // meant to genuinely hold a P, which is the whole point of the /compute probe
 // (how many CPU-bound requests before GOMAXPROCS saturates).
 func Burn(difficulty int) uint64 {
 	var x uint64 = 1469598103934665603 // FNV-1a offset basis, just a seed
 	n := Iterations(difficulty)
 	for i := 0; i < n; i++ {
 		x ^= uint64(i)
 		x *= 1099511628211 // FNV-1a prime
 		x = x<<13 | x>>51  // rotate so high bits feed back in
 	}
 	return x
 }
 // Percentile returns the p-th percentile (p in [0,1]) of an already-sorted
 // slice. Empty → 0, p<=0 → first, p>=1 → last. Nearest-rank, no interpolation —
 // good enough for "orders of magnitude" latency reporting.
 func Percentile(sorted []time.Duration, p float64) time.Duration {
 	if len(sorted) == 0 {
 		return 0
 	}
 	if p < 0 {
 		p = 0
 	}
 	if p > 1 {
 		p = 1
 	}
 	i := int(float64(len(sorted)-1) * p)
 	return sorted[i]
 }
 // Latencies accumulates request durations and reports percentiles, keeping the
 // client's main loop thin. Not safe for concurrent use — each worker should own
 // one and they get merged, or a single collector goroutine owns it.
 type Latencies struct {
 	d []time.Duration
 }
 func (l *Latencies) Add(d time.Duration) { l.d = append(l.d, d) }
 func (l *Latencies) Len() int            { return len(l.d) }
 // Merge folds another accumulator's samples in (for per-worker collection).
 func (l *Latencies) Merge(o *Latencies) { l.d = append(l.d, o.d...) }
 // Snapshot sorts and returns p50, p99, and max. Sorts in place; call once at end.
 func (l *Latencies) Snapshot() (p50, p99, max time.Duration) {
 	sort.Slice(l.d, func(i, j int) bool { return l.d[i] < l.d[j] })
 	return Percentile(l.d, 0.50), Percentile(l.d, 0.99), Percentile(l.d, 1)
 }
 // ErrClass names which side of the connection gave out. It turns the body-drain
 // "which error string means what" table into something the client can count.
 type ErrClass int
 const (
 	ErrNone             ErrClass = iota // no error
 	ErrClientFdLimit                    // this process ran out of fds (dial: too many open files)
 	ErrClientPortLimit                  // this process ran out of ephemeral ports (can't assign requested address)
 	ErrServerCantAccept                 // server can't accept (out of fds / backlog full) — we time out
 	ErrTimeout                          // request/response timed out (ambiguous, but usually the server)
 	ErrConnReset                        // peer reset/closed the connection
 	ErrRefused                          // nothing listening / server gone
 	ErrOther
 )
 func (c ErrClass) String() string {
 	switch c {
 	case ErrNone:
 		return "none"
 	case ErrClientFdLimit:
 		return "client-fd-limit"
 	case ErrClientPortLimit:
 		return "client-port-limit"
 	case ErrServerCantAccept:
 		return "server-cant-accept"
 	case ErrTimeout:
 		return "timeout"
 	case ErrConnReset:
 		return "conn-reset"
 	case ErrRefused:
 		return "conn-refused"
 	default:
 		return "other"
 	}
 }
 // Classify maps a net/http client error to the side that most likely caused it.
 // It leans on error strings because the kernel/runtime messages are the only
 // distinguishing signal for several of these; the checks are ordered most- to
 // least-specific so, e.g., "too many open files" wins over the generic timeout.
 func Classify(err error) ErrClass {
 	if err == nil {
 		return ErrNone
 	}
 	msg := strings.ToLower(err.Error())
 	switch {
 	case strings.Contains(msg, "too many open files"):
 		// A dial failing this way is THIS process out of fds. (A server out of
 		// fds fails on accept and never produces this string on the client.)
 		return ErrClientFdLimit
 	case strings.Contains(msg, "assign requested address"):
 		// EADDRNOTAVAIL: no ephemeral port left for a fresh 4-tuple to this
 		// server. A client-side wall, often hit before the fd limit.
 		return ErrClientPortLimit
 	case strings.Contains(msg, "connection refused"):
 		return ErrRefused
 	case strings.Contains(msg, "connection reset") || strings.Contains(msg, "broken pipe") ||
 		strings.Contains(msg, "eof"):
 		return ErrConnReset
 	}
 	// "awaiting response headers" / a dial timeout usually means the server
 	// accepted us into its backlog but can't service (or can't accept at all),
 	// so we sit waiting. Treat a timeout as a server-can't-keep-up signal, but
 	// keep a generic ErrTimeout for the plain deadline case.
 	var ne net.Error
 	if errors.As(err, &ne) && ne.Timeout() {
 		if strings.Contains(msg, "awaiting response headers") {
 			return ErrServerCantAccept
 		}
 		return ErrTimeout
 	}
 	if strings.Contains(msg, "timeout") || strings.Contains(msg, "deadline exceeded") {
 		return ErrTimeout
 	}
 	return ErrOther
 }
@@ -0,0 +1,147 @@
 package loadlib
 import (
 	"errors"
 	"fmt"
 	"net"
 	"testing"
 	"time"
 )
 func TestBurnDeterministic(t *testing.T) {
 	// Same difficulty must give the same checksum: the /compute endpoint returns
 	// it, and a flaky checksum would mean a non-deterministic loop.
 	for _, d := range []int{1, 3, 10} {
 		if a, b := Burn(d), Burn(d); a != b {
 			t.Errorf("Burn(%d) not deterministic: %d != %d", d, a, b)
 		}
 	}
 }
 func TestBurnDifficultyChangesResult(t *testing.T) {
 	// Different difficulties should not collide (they run different iteration
 	// counts); a collision here would suggest the loop body ignores i.
 	if Burn(1) == Burn(2) {
 		t.Error("Burn(1) == Burn(2): difficulty has no effect on the checksum")
 	}
 }
 func TestIterationsLinearAndFloored(t *testing.T) {
 	if got := Iterations(2); got != 2*burnIters {
 		t.Errorf("Iterations(2) = %d, want %d", got, 2*burnIters)
 	}
 	// difficulty < 1 is clamped to 1 so /compute?difficulty=0 still does work.
 	for _, d := range []int{0, -5} {
 		if got := Iterations(d); got != burnIters {
 			t.Errorf("Iterations(%d) = %d, want %d (floor of 1)", d, got, burnIters)
 		}
 	}
 	// Cost is monotonic in difficulty.
 	if Iterations(3) <= Iterations(2) {
 		t.Error("Iterations not increasing with difficulty")
 	}
 }
 func TestPercentileEdges(t *testing.T) {
 	if got := Percentile(nil, 0.5); got != 0 {
 		t.Errorf("Percentile(empty) = %v, want 0", got)
 	}
 	s := []time.Duration{10, 20, 30, 40, 50}
 	cases := []struct {
 		p    float64
 		want time.Duration
 	}{
 		{-1, 10},  // clamped to first
 		{0, 10},   // first
 		{0.5, 30}, // middle
 		{1, 50},   // last
 		{2, 50},   // clamped to last
 	}
 	for _, c := range cases {
 		if got := Percentile(s, c.p); got != c.want {
 			t.Errorf("Percentile(%v, %v) = %v, want %v", s, c.p, got, c.want)
 		}
 	}
 	if got := Percentile([]time.Duration{99}, 0.9); got != 99 {
 		t.Errorf("Percentile(single) = %v, want 99", got)
 	}
 }
 func TestLatenciesSnapshot(t *testing.T) {
 	var l Latencies
 	// Add out of order; Snapshot must sort before reporting.
 	for _, d := range []time.Duration{50, 10, 30, 40, 20} {
 		l.Add(d * time.Millisecond)
 	}
 	if l.Len() != 5 {
 		t.Fatalf("Len = %d, want 5", l.Len())
 	}
 	p50, p99, max := l.Snapshot()
 	if p50 != 30*time.Millisecond {
 		t.Errorf("p50 = %v, want 30ms", p50)
 	}
 	if max != 50*time.Millisecond {
 		t.Errorf("max = %v, want 50ms", max)
 	}
 	// Nearest-rank: index int(4*0.99)=3 → 40ms for this 5-sample set.
 	if p99 != 40*time.Millisecond {
 		t.Errorf("p99 = %v, want 40ms", p99)
 	}
 }
 func TestLatenciesMerge(t *testing.T) {
 	var a, b Latencies
 	a.Add(10 * time.Millisecond)
 	b.Add(20 * time.Millisecond)
 	a.Merge(&b)
 	if a.Len() != 2 {
 		t.Fatalf("after merge Len = %d, want 2", a.Len())
 	}
 }
 func TestClassify(t *testing.T) {
 	timeoutErr := &net.OpError{
 		Op:  "dial",
 		Err: timeoutError{},
 	}
 	headerTimeout := fmt.Errorf("net/http: timeout awaiting response headers")
 	cases := []struct {
 		name string
 		err  error
 		want ErrClass
 	}{
 		{"nil", nil, ErrNone},
 		{"client fds", errors.New("dial tcp 127.0.0.1:8098: socket: too many open files"), ErrClientFdLimit},
 		{"client ports", errors.New("dial tcp 127.0.0.1:8098: connect: can't assign requested address"), ErrClientPortLimit},
 		{"refused", errors.New("dial tcp 127.0.0.1:8098: connect: connection refused"), ErrRefused},
 		{"reset", errors.New("read tcp: connection reset by peer"), ErrConnReset},
 		{"eof", errors.New("unexpected EOF"), ErrConnReset},
 		{"plain net timeout", timeoutErr, ErrTimeout},
 		{"server cant accept", headerTimeoutNetErr{}, ErrServerCantAccept},
 		{"header timeout string", headerTimeout, ErrTimeout}, // not a net.Error, but has "timeout"
 		{"deadline", errors.New("context deadline exceeded"), ErrTimeout},
 		{"other", errors.New("something weird"), ErrOther},
 	}
 	for _, c := range cases {
 		if got := Classify(c.err); got != c.want {
 			t.Errorf("%s: Classify(%v) = %v, want %v", c.name, c.err, got, c.want)
 		}
 	}
 }
 // timeoutError is a minimal net.Error whose Timeout() reports true, used to build
 // a synthetic *net.OpError the way the runtime would on a real dial timeout.
 type timeoutError struct{}
 func (timeoutError) Error() string   { return "i/o timeout" }
 func (timeoutError) Timeout() bool   { return true }
 func (timeoutError) Temporary() bool { return true }
 // headerTimeoutNetErr is a net.Error whose message is the exact one net/http
 // produces when ResponseHeaderTimeout fires — the "server can't accept" signal.
 type headerTimeoutNetErr struct{}
 func (headerTimeoutNetErr) Error() string   { return "net/http: timeout awaiting response headers" }
 func (headerTimeoutNetErr) Timeout() bool   { return true }
 func (headerTimeoutNetErr) Temporary() bool { return true }
@@ -0,0 +1,219 @@
 // Experiment subject — a deliberately simple HTTP server whose only job is to
 // be measured. How many connections can it hold, and what breaks first?
 //
 // Endpoints:
 //
 //	GET /static              fixed tiny body, no per-request work   (the baseline)
 //	GET /compute?difficulty=N  burns CPU ∝ N, returns the checksum   (CPU-bound probe)
 //	GET /sleep?d=500ms       sleeps d, honoring client disconnect   (in-flight holder)
 //	GET /healthz             liveness
 //	GET /stats               JSON: accepted_total, active_conns, in_flight, goroutines, heap_bytes
 //
 // Observability. We separate three numbers that are easy to conflate:
 //
 //	accepted_total  every TCP conn ever accepted (only grows)
 //	active_conns    conns open right now (idle keep-alive OR busy)   — via ConnState
 //	in_flight       handlers actually running right now              — via middleware
 //
 // An idle keep-alive conn counts in active_conns but not in_flight; a running
 // /sleep counts in both. That gap is the whole point of the idle-vs-slow contrast.
 //
 // Timeout flags are the naive-vs-hardened knobs. They default to OFF so the
 // slowloris experiment works out of the box; turn them on to watch the server
 // reclaim connections:
 //
 //	-read-header-timeout 3s   drop a conn that dawdles sending headers (kills slowloris)
 //	-idle-timeout 30s         drop a keep-alive conn that goes quiet
 //	-read-timeout / -write-timeout   whole-request deadlines
 //
 // Run (terminal 1), then drive it with ./client (terminal 2):
 //
 //	go run ./98-http-server-experiments/server
 //	go run ./98-http-server-experiments/server -read-header-timeout 3s -idle-timeout 30s   # hardened
 //
 // Give the server its own fd budget so YOU decide when it runs out (not the OS
 // default of a few hundred / a few thousand). bash sets both soft and hard:
 //
 //	ulimit -Hn 4096; ulimit -Sn 4096; go run ./98-http-server-experiments/server
 //
 // (Since Go 1.19 the runtime raises the soft RLIMIT_NOFILE to the hard limit at
 // startup, so only the hard limit really caps you. zsh's `ulimit -n` sets only
 // the soft one — use `-Hn`.)
 package main
 import (
 	"context"
 	"encoding/json"
 	"flag"
 	"fmt"
 	"net"
 	"net/http"
 	"os"
 	"os/signal"
 	"runtime"
 	"strconv"
 	"sync/atomic"
 	"time"
 	"gosre-exercises/98-http-server-experiments/loadlib"
 )
 // server is the subject under test: an http.Server plus the three live counters.
 type server struct {
 	srv         *http.Server
 	ln          net.Listener
 	acceptedTot atomic.Int64 // every conn ever accepted
 	activeConns atomic.Int64 // conns open right now
 	inFlight    atomic.Int64 // handlers running right now
 }
 func (s *server) handleStatic(w http.ResponseWriter, r *http.Request) {
 	w.Header().Set("Content-Type", "text/plain")
 	fmt.Fprint(w, "ok\n")
 }
 // handleCompute burns CPU proportional to ?difficulty= and writes the checksum
 // back. Writing it matters: it forces the burn to actually happen (the compiler
 // can't prove the result is unused) and lets the client confirm the work ran.
 func (s *server) handleCompute(w http.ResponseWriter, r *http.Request) {
 	difficulty := 1
 	if v := r.URL.Query().Get("difficulty"); v != "" {
 		if n, err := strconv.Atoi(v); err == nil {
 			difficulty = n
 		}
 	}
 	sum := loadlib.Burn(difficulty)
 	w.Header().Set("Content-Type", "text/plain")
 	fmt.Fprintf(w, "difficulty=%d checksum=%d\n", difficulty, sum)
 }
 // handleSleep keeps the request in flight for ?d= but returns early if the
 // client disconnects or a server timeout fires — otherwise a client that hangs
 // up would leave the goroutine sleeping to full term, hiding the reclaim.
 func (s *server) handleSleep(w http.ResponseWriter, r *http.Request) {
 	d := 500 * time.Millisecond
 	if v := r.URL.Query().Get("d"); v != "" {
 		if parsed, err := time.ParseDuration(v); err == nil {
 			d = parsed
 		}
 	}
 	select {
 	case <-time.After(d):
 		w.Header().Set("Content-Type", "text/plain")
 		fmt.Fprintf(w, "slept %v\n", d)
 	case <-r.Context().Done():
 		// Client went away or a timeout fired; nothing to write.
 	}
 }
 type statsSnapshot struct {
 	AcceptedTotal int64  `json:"accepted_total"`
 	ActiveConns   int64  `json:"active_conns"`
 	InFlight      int64  `json:"in_flight"`
 	Goroutines    int    `json:"goroutines"`
 	HeapBytes     uint64 `json:"heap_bytes"`
 }
 func (s *server) snapshot() statsSnapshot {
 	var m runtime.MemStats
 	runtime.ReadMemStats(&m)
 	return statsSnapshot{
 		AcceptedTotal: s.acceptedTot.Load(),
 		ActiveConns:   s.activeConns.Load(),
 		InFlight:      s.inFlight.Load(),
 		Goroutines:    runtime.NumGoroutine(),
 		HeapBytes:     m.HeapInuse,
 	}
 }
 func (s *server) handleStats(w http.ResponseWriter, r *http.Request) {
 	w.Header().Set("Content-Type", "application/json")
 	json.NewEncoder(w).Encode(s.snapshot())
 }
 // countInFlight wraps a handler so in_flight tracks handlers actually running,
 // as distinct from conns merely open (which ConnState already counts).
 func (s *server) countInFlight(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		s.inFlight.Add(1)
 		defer s.inFlight.Add(-1)
 		next.ServeHTTP(w, r)
 	})
 }
 func main() {
 	addr := flag.String("addr", "127.0.0.1:8098", "address to listen on")
 	readHeaderTimeout := flag.Duration("read-header-timeout", 0, "Server.ReadHeaderTimeout (0 = off; set 3s to defeat slowloris)")
 	readTimeout := flag.Duration("read-timeout", 0, "Server.ReadTimeout (0 = off; whole-request read deadline)")
 	writeTimeout := flag.Duration("write-timeout", 0, "Server.WriteTimeout (0 = off)")
 	idleTimeout := flag.Duration("idle-timeout", 0, "Server.IdleTimeout (0 = off; set 30s to reclaim quiet keep-alives)")
 	maxHeaderBytes := flag.Int("max-header-bytes", http.DefaultMaxHeaderBytes, "Server.MaxHeaderBytes (1<<20 default)")
 	flag.Parse()
 	// NOTE: no -max-conns. Capping concurrent connections in the stdlib means
 	// wrapping the Listener (x/net/netutil.LimitListener) or a semaphore in the
 	// handler; we keep this zero-dependency and instead observe the ceiling the
 	// OS imposes (fds/backlog). See the README for the stdlib-only pattern.
 	ln, err := net.Listen("tcp", *addr)
 	if err != nil {
 		fmt.Fprintln(os.Stderr, "listen:", err)
 		os.Exit(1)
 	}
 	s := &server{ln: ln}
 	mux := http.NewServeMux()
 	mux.HandleFunc("/static", s.handleStatic)
 	mux.HandleFunc("/compute", s.handleCompute)
 	mux.HandleFunc("/sleep", s.handleSleep)
 	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "ok") })
 	mux.HandleFunc("/stats", s.handleStats)
 	s.srv = &http.Server{
 		Handler:           s.countInFlight(mux),
 		ReadHeaderTimeout: *readHeaderTimeout,
 		ReadTimeout:       *readTimeout,
 		WriteTimeout:      *writeTimeout,
 		IdleTimeout:       *idleTimeout,
 		MaxHeaderBytes:    *maxHeaderBytes,
 		ConnState: func(c net.Conn, st http.ConnState) {
 			switch st {
 			case http.StateNew:
 				s.acceptedTot.Add(1)
 				s.activeConns.Add(1)
 			case http.StateClosed, http.StateHijacked:
 				s.activeConns.Add(-1)
 			}
 		},
 	}
 	go func() {
 		if err := s.srv.Serve(ln); err != nil && err != http.ErrServerClosed {
 			fmt.Fprintln(os.Stderr, "serve:", err)
 		}
 	}()
 	fmt.Printf("serving on http://%s  (timeouts: readHeader=%v read=%v write=%v idle=%v)\n",
 		ln.Addr(), *readHeaderTimeout, *readTimeout, *writeTimeout, *idleTimeout)
 	fmt.Println("point the client at it:  go run ./98-http-server-experiments/client -mode idle -n 2000")
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
 	defer stop()
 	tick := time.NewTicker(time.Second)
 	defer tick.Stop()
 	for {
 		select {
 		case <-ctx.Done():
 			fmt.Println("\nshutting down")
 			shutCtx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
 			defer cancel()
 			s.srv.Shutdown(shutCtx)
 			return
 		case <-tick.C:
 			snap := s.snapshot()
 			fmt.Printf("%s  accepted_total=%d active=%d in_flight=%d goroutines=%d heap=%dMiB\n",
 				time.Now().Format("15:04:05"), snap.AcceptedTotal, snap.ActiveConns,
 				snap.InFlight, snap.Goroutines, snap.HeapBytes>>20)
 		}
 	}
 }