Add URL-level concurrency (-c flag, Step 7)

Probe multiple URLs in parallel with a bounded worker pool; the N samples
of each URL remain sequential to preserve accurate min/avg/max statistics.
Default auto-concurrency is min(numURLs, 8); -c 1 restores serial mode.
Output is always buffered and printed in original input order. Verified
clean with go test -race.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-07-01 01:33:03 +02:00
parent 45583ac2be
commit 9a69ba946f
9 changed files with 394 additions and 22 deletions

View File

@@ -0,0 +1,140 @@
# Plan: Concurrency (Go, Step 7)
## Context
Large probe runs are slow. The example matrices in `docs/custom-usage-examples.md`
are 15 URLs × 10 samples = 150 sequential HTTP requests, dominated by summed
network round-trips, so a single run takes minutes. Execution is currently
strictly sequential: `run()` loops over URLs and, for each, `runSamples()` loops
`-n` times calling `probe.Measure()`.
We want to cut wall-clock time by probing multiple **URLs** in parallel, while
keeping measurement honest. Concurrency and latency accuracy are in tension:
firing many requests at once makes them contend for the local NIC/CPU/resolver
and inflates timings. The decision (confirmed with the user) is to parallelize
**only across URLs** — the samples of a single URL stay sequential so each URL's
min/avg/max remains a clean, self-consistent measurement.
Confirmed decisions:
- **Granularity:** across URLs only; samples sequential per URL. Unit of work =
one URL with its full `-n` sample set.
- **Default concurrency:** auto when `-c` is not given → `min(numURLs, 8)`.
`-c 1` forces fully sequential (most accurate); `-c N` sets an explicit cap.
- **Output:** buffered and printed in original input order (deterministic;
identical layout to today, for both text and JSON).
## Files to modify
### `go/main.go` — core change
1. **New flag** (mirror the `-n`/`--count` pattern at lines 7879):
```go
conc := fs.Int("concurrency", 0, "max URLs probed in parallel (0 = auto)")
fs.IntVar(conc, "c", 0, "max URLs probed in parallel (shorthand)")
```
Add `-c, --concurrency` to `usageText` (around lines 2227).
2. **Resolve effective concurrency** after the empty-URL check (line 96):
```go
const defaultMaxConc = 8
workers := *conc
if workers <= 0 {
workers = min(len(urls), defaultMaxConc) // auto
}
workers = min(workers, len(urls)) // never exceed work units
if workers < 1 {
workers = 1
}
```
(Go 1.26 has builtin `min`.)
3. **Split `run()` into measure phase (concurrent) + render phase (sequential).**
Introduce a small result holder:
```go
type urlResult struct {
url string
succeeded, failed []probe.Result
}
```
Measure phase — bounded worker pool via a semaphore channel + `sync.WaitGroup`,
each goroutine writing only its own preallocated slot (no shared mutable state,
so no mutex / data race):
```go
results := make([]urlResult, len(urls))
sem := make(chan struct{}, workers)
var wg sync.WaitGroup
for i, rawURL := range urls {
wg.Add(1)
go func(i int, rawURL string) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
s, f := runSamples(rawURL, *count, opts)
results[i] = urlResult{rawURL, s, f}
}(i, rawURL)
}
wg.Wait()
```
Render phase — the existing per-URL loop body (lines 102127), unchanged in
behaviour, now reading from `results[i]` instead of calling `runSamples`
inline. `worstCode` accumulation, JSON entry building, and ordered text
printing all stay identical. JSON encode block (129136) and `return worstCode`
unchanged.
`runSamples`, `printURL`, `buildJSONEntry`, and all output helpers stay as-is.
Concurrency safety: `probe.Measure` uses `http.DefaultClient`, which is safe for
concurrent use; distinct URLs hit distinct hosts so the shared transport pool is
not a correctness concern. Output order is deterministic because rendering reads
the indexed `results` slice after `wg.Wait()`.
### `go/run_test.go` — add coverage
- `TestRunConcurrentOrder`: spin up several `httptest` servers (reuse existing
`statusSrv` helper), pass them with `-c` larger than 1, and assert the printed
blocks appear in **input order** and the exit code matches the sequential run.
- Add a mixed success/failure case under high `-c` to confirm `worstCode`
aggregation is unaffected by parallelism.
### `Makefile` + `docs/usage/makefile.md`
- Add `go-test-race`: `go -C $(GO_DIR) test -race ./...` (with `##` help comment,
added to `.PHONY`). Document it in the Go targets table. Leave `check` as-is
(race run kept as an explicit opt-in target).
### Docs & conventions
- `docs/usage/step-7-concurrency.md` (new): explain the `-c`/`--concurrency`
flag, the auto default (`min(numURLs, 8)`), the across-URLs-only model, the
accuracy trade-off (use `-c 1` for the most precise numbers), and a worked
example with before/after timing.
- `README.md`: add `-c, --concurrency` to the Usage flags block; add **Step 7 —
Concurrency** row (✅) to the Go roadmap table.
- `CHANGELOG.md`: new top entry, `2026-07-01 01:23 — Concurrency (Go, Step 7)`.
- `docs/custom-usage-examples.md`: note near the top that runs now parallelize
across URLs by default and that `-c 1` restores serial timing; revise the
Example 2/3/9 runtime estimates to reflect the speedup.
- `docs/plans/2026-07-01-01-23-concurrency.md`: copy of this plan (per CLAUDE.md).
## Verification
```sh
make build
# Auto concurrency (defaults to min(numURLs,8)) — should be much faster than before:
time ./go/latprobe -n 5 https://www.google.com https://www.bbc.co.uk \
https://www.lemonde.fr https://www.spiegel.de https://www.abc.net.au
# Forced serial for comparison / accuracy:
time ./go/latprobe -c 1 -n 5 https://www.google.com https://www.bbc.co.uk \
https://www.lemonde.fr https://www.spiegel.de https://www.abc.net.au
# Output order is deterministic regardless of -c:
./go/latprobe -c 8 https://www.google.com https://www.bbc.co.uk https://www.abc.net.au
# JSON array order also matches input order:
./go/latprobe --json -c 8 https://www.google.com https://www.bbc.co.uk | jq '.[].url'
make test # full suite incl. new ordering tests
make go-test-race # data-race detector must report clean
make check # fmt + vet + test gate
```
Confirm: identical output (modulo timing numbers) between `-c 1` and default for
the same URL set; exit codes unchanged; `-race` clean.