Add URL-level concurrency (-c flag, Step 7)
Probe multiple URLs in parallel with a bounded worker pool; the N samples of each URL remain sequential to preserve accurate min/avg/max statistics. Default auto-concurrency is min(numURLs, 8); -c 1 restores serial mode. Output is always buffered and printed in original input order. Verified clean with go test -race. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
140
docs/plans/2026-07-01-01-23-concurrency.md
Normal file
140
docs/plans/2026-07-01-01-23-concurrency.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# Plan: Concurrency (Go, Step 7)
|
||||
|
||||
## Context
|
||||
|
||||
Large probe runs are slow. The example matrices in `docs/custom-usage-examples.md`
|
||||
are 15 URLs × 10 samples = 150 sequential HTTP requests, dominated by summed
|
||||
network round-trips, so a single run takes minutes. Execution is currently
|
||||
strictly sequential: `run()` loops over URLs and, for each, `runSamples()` loops
|
||||
`-n` times calling `probe.Measure()`.
|
||||
|
||||
We want to cut wall-clock time by probing multiple **URLs** in parallel, while
|
||||
keeping measurement honest. Concurrency and latency accuracy are in tension:
|
||||
firing many requests at once makes them contend for the local NIC/CPU/resolver
|
||||
and inflates timings. The decision (confirmed with the user) is to parallelize
|
||||
**only across URLs** — the samples of a single URL stay sequential so each URL's
|
||||
min/avg/max remains a clean, self-consistent measurement.
|
||||
|
||||
Confirmed decisions:
|
||||
- **Granularity:** across URLs only; samples sequential per URL. Unit of work =
|
||||
one URL with its full `-n` sample set.
|
||||
- **Default concurrency:** auto when `-c` is not given → `min(numURLs, 8)`.
|
||||
`-c 1` forces fully sequential (most accurate); `-c N` sets an explicit cap.
|
||||
- **Output:** buffered and printed in original input order (deterministic;
|
||||
identical layout to today, for both text and JSON).
|
||||
|
||||
## Files to modify
|
||||
|
||||
### `go/main.go` — core change
|
||||
|
||||
1. **New flag** (mirror the `-n`/`--count` pattern at lines 78–79):
|
||||
```go
|
||||
conc := fs.Int("concurrency", 0, "max URLs probed in parallel (0 = auto)")
|
||||
fs.IntVar(conc, "c", 0, "max URLs probed in parallel (shorthand)")
|
||||
```
|
||||
Add `-c, --concurrency` to `usageText` (around lines 22–27).
|
||||
|
||||
2. **Resolve effective concurrency** after the empty-URL check (line 96):
|
||||
```go
|
||||
const defaultMaxConc = 8
|
||||
workers := *conc
|
||||
if workers <= 0 {
|
||||
workers = min(len(urls), defaultMaxConc) // auto
|
||||
}
|
||||
workers = min(workers, len(urls)) // never exceed work units
|
||||
if workers < 1 {
|
||||
workers = 1
|
||||
}
|
||||
```
|
||||
(Go 1.26 has builtin `min`.)
|
||||
|
||||
3. **Split `run()` into measure phase (concurrent) + render phase (sequential).**
|
||||
Introduce a small result holder:
|
||||
```go
|
||||
type urlResult struct {
|
||||
url string
|
||||
succeeded, failed []probe.Result
|
||||
}
|
||||
```
|
||||
Measure phase — bounded worker pool via a semaphore channel + `sync.WaitGroup`,
|
||||
each goroutine writing only its own preallocated slot (no shared mutable state,
|
||||
so no mutex / data race):
|
||||
```go
|
||||
results := make([]urlResult, len(urls))
|
||||
sem := make(chan struct{}, workers)
|
||||
var wg sync.WaitGroup
|
||||
for i, rawURL := range urls {
|
||||
wg.Add(1)
|
||||
go func(i int, rawURL string) {
|
||||
defer wg.Done()
|
||||
sem <- struct{}{}
|
||||
defer func() { <-sem }()
|
||||
s, f := runSamples(rawURL, *count, opts)
|
||||
results[i] = urlResult{rawURL, s, f}
|
||||
}(i, rawURL)
|
||||
}
|
||||
wg.Wait()
|
||||
```
|
||||
Render phase — the existing per-URL loop body (lines 102–127), unchanged in
|
||||
behaviour, now reading from `results[i]` instead of calling `runSamples`
|
||||
inline. `worstCode` accumulation, JSON entry building, and ordered text
|
||||
printing all stay identical. JSON encode block (129–136) and `return worstCode`
|
||||
unchanged.
|
||||
|
||||
`runSamples`, `printURL`, `buildJSONEntry`, and all output helpers stay as-is.
|
||||
|
||||
Concurrency safety: `probe.Measure` uses `http.DefaultClient`, which is safe for
|
||||
concurrent use; distinct URLs hit distinct hosts so the shared transport pool is
|
||||
not a correctness concern. Output order is deterministic because rendering reads
|
||||
the indexed `results` slice after `wg.Wait()`.
|
||||
|
||||
### `go/run_test.go` — add coverage
|
||||
|
||||
- `TestRunConcurrentOrder`: spin up several `httptest` servers (reuse existing
|
||||
`statusSrv` helper), pass them with `-c` larger than 1, and assert the printed
|
||||
blocks appear in **input order** and the exit code matches the sequential run.
|
||||
- Add a mixed success/failure case under high `-c` to confirm `worstCode`
|
||||
aggregation is unaffected by parallelism.
|
||||
|
||||
### `Makefile` + `docs/usage/makefile.md`
|
||||
|
||||
- Add `go-test-race`: `go -C $(GO_DIR) test -race ./...` (with `##` help comment,
|
||||
added to `.PHONY`). Document it in the Go targets table. Leave `check` as-is
|
||||
(race run kept as an explicit opt-in target).
|
||||
|
||||
### Docs & conventions
|
||||
|
||||
- `docs/usage/step-7-concurrency.md` (new): explain the `-c`/`--concurrency`
|
||||
flag, the auto default (`min(numURLs, 8)`), the across-URLs-only model, the
|
||||
accuracy trade-off (use `-c 1` for the most precise numbers), and a worked
|
||||
example with before/after timing.
|
||||
- `README.md`: add `-c, --concurrency` to the Usage flags block; add **Step 7 —
|
||||
Concurrency** row (✅) to the Go roadmap table.
|
||||
- `CHANGELOG.md`: new top entry, `2026-07-01 01:23 — Concurrency (Go, Step 7)`.
|
||||
- `docs/custom-usage-examples.md`: note near the top that runs now parallelize
|
||||
across URLs by default and that `-c 1` restores serial timing; revise the
|
||||
Example 2/3/9 runtime estimates to reflect the speedup.
|
||||
- `docs/plans/2026-07-01-01-23-concurrency.md`: copy of this plan (per CLAUDE.md).
|
||||
|
||||
## Verification
|
||||
|
||||
```sh
|
||||
make build
|
||||
# Auto concurrency (defaults to min(numURLs,8)) — should be much faster than before:
|
||||
time ./go/latprobe -n 5 https://www.google.com https://www.bbc.co.uk \
|
||||
https://www.lemonde.fr https://www.spiegel.de https://www.abc.net.au
|
||||
# Forced serial for comparison / accuracy:
|
||||
time ./go/latprobe -c 1 -n 5 https://www.google.com https://www.bbc.co.uk \
|
||||
https://www.lemonde.fr https://www.spiegel.de https://www.abc.net.au
|
||||
# Output order is deterministic regardless of -c:
|
||||
./go/latprobe -c 8 https://www.google.com https://www.bbc.co.uk https://www.abc.net.au
|
||||
# JSON array order also matches input order:
|
||||
./go/latprobe --json -c 8 https://www.google.com https://www.bbc.co.uk | jq '.[].url'
|
||||
|
||||
make test # full suite incl. new ordering tests
|
||||
make go-test-race # data-race detector must report clean
|
||||
make check # fmt + vet + test gate
|
||||
```
|
||||
|
||||
Confirm: identical output (modulo timing numbers) between `-c 1` and default for
|
||||
the same URL set; exit codes unchanged; `-race` clean.
|
||||
Reference in New Issue
Block a user