kacerr/http-latency-prober
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: hxprobe plans, summaries, explanations, usage, and changelog

Browse Source 
Plans (docs/plans/):
- 2026-07-01-23-47-py-hxprobe-httpx.md — initial httpx probe design
- 2026-07-02-09-32 through 14-05 — standalone project, toolchain, usage
  doc + Makefile, file input (-f), simplification pass, run-summary footer

Summaries (docs/summaries/): one per completed feature, recording what
was actually built, deviations from the plan, and verification steps

Explanations (docs/explanations/): two deep-dives written during review —
hxprobe concurrency model and worst-exit-code + render-loop analysis

Usage (docs/usage/hxprobe.md): overview with pointer to hxprobe/USAGE.md
for the full runnable reference

Walkthrough (docs/py-latprobe-walkthrough.md): narrative tour of the
latprobe Python package for interview / code-review context

CHANGELOG.md: entries for all hxprobe features (toolchain, usage doc,
file input, simplification, run-summary footer)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Jan Novak
2026-07-02 13:22:59 +02:00
parent 13a562e966
commit f487a4b1bd
19 changed files with 2367 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

56
docs/explanations/2026-07-02-13-10-hxprobe-concurrency.md Normal file
View File

@@ -0,0 +1,56 @@
# How does concurrency work in hxprobe?
## Question
Selection in `hxprobe/hxprobe/cli.py:448-449`:
```python
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as ex:
all_results = list(ex.map(_probe, urls))
```
> explain how concurrency works in hxprobe
## Answer
There are two independent levels of repetition:
1. **Across URLs** — parallel, via a thread pool.
2. **Across samples of the same URL** (`-n`/`--count`) — sequential, within a single thread.
### Worker count
`cli.py:440-443`
```python
workers = ns.concurrency
if workers <= 0:
workers = min(len(urls), 8)
workers = max(1, min(workers, len(urls)))
```
`-c`/`--concurrency` picks the pool size; `0` (default) means "auto" → `min(len(urls), 8)`. The final clamp guarantees `1 ≤ workers ≤ len(urls)` — never more threads than there are URLs to probe, never zero.
### The pool itself
`cli.py:445-449`
```python
def _probe(url: str) -> tuple[list[Result], list[Result]]:
return _run_samples(url, count, opts)
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as ex:
all_results = list(ex.map(_probe, urls))
```
One task per URL is submitted to the pool. `_probe` delegates to `_run_samples`, `cli.py:64-69`:
```python
def _run_samples(url: str, count: int, opts: Options) -> tuple[list[Result], list[Result]]:
succeeded, failed = [], []
for _ in range(count):
r = measure(url, opts)
(failed if r.err else succeeded).append(r)
return succeeded, failed
```
So within one URL's thread, the `count` samples run **one at a time** — never concurrently. This is deliberate: if N samples fired at the same host in parallel, they'd contend for the same TCP/TLS handshake path and connection setup, and the resulting min/avg/max per phase would reflect that contention rather than the host's actual latency. Keeping samples sequential is what makes `-n`'s statistics meaningful.
### Why threads work here despite the GIL
`measure()` does blocking socket/SSL I/O — `socket.connect()`, `.recv()`, `ssl.wrap_socket()`, etc. (see the custom `_TimingBackend`/`_TimingStream` in `probe.py`). CPython releases the GIL around blocking syscalls, so N threads genuinely overlap in wall-clock time waiting on the network, even though only one thread executes Python bytecode at once. This is I/O-bound concurrency, not CPU parallelism — threads are the right tool, not `asyncio` or multiprocessing.
### Ordering guarantee
`ex.map(_probe, urls)` returns results in the *same order as the input `urls`*, regardless of which thread finishes first — that's a documented property of `Executor.map`. That's why `list(ex.map(...))` can be zipped directly against `urls` afterward (`cli.py:454`) to produce deterministic text/JSON output order, even though the underlying probes complete out of order.
### No shared mutable state / no locks needed
Each thread's `_probe` call returns its own `(succeeded, failed)` tuple; nothing is written to a shared structure until back in the main thread after the `with` block exits (which also blocks until every submitted task completes, since `ThreadPoolExecutor.__exit__` calls `shutdown(wait=True)`). The worst-exit-code accumulation and JSON building (`cli.py:451-472`) then run single-threaded over `all_results`.

144
docs/explanations/2026-07-02-13-25-hxprobe-worst-exit-code-and-render-loop.md Normal file
View File

@@ -0,0 +1,144 @@
# What does the per-URL worst-exit-code / render loop do, and does it make sense?
## Question
Selection in `hxprobe/hxprobe/cli.py:454-472`:
```python
for i, (url, (succeeded, failed)) in enumerate(zip(urls, all_results)):
for r in failed:
c = _phase_code(r.fail_phase)
if c > worst:
worst = c
if ns.fail:
for r in succeeded:
if r.status_code >= 400:
worst = max(worst, EXIT_HTTP)
last_detail = succeeded[-1].detail if succeeded else (failed[-1].detail if failed else None)
if ns.json_out:
json_items.append(_build_json_entry(url, succeeded, failed, last_detail))
continue
if i > 0:
stdout.write("\n")
_print_url(url, succeeded, failed, count, ns.fail, stdout)
```
> explain following block and if it makes sense, other related stuff, and save it in explanations
## What it does
This is the single pass over per-URL results that runs after the thread pool
(`cli.py:448-449`, see [2026-07-02-13-10-hxprobe-concurrency.md](2026-07-02-13-10-hxprobe-concurrency.md))
finishes. It does two jobs in one loop: compute the process's final exit code,
and render output (text or accumulate JSON) — one URL at a time, in input
order (guaranteed by `zip(urls, all_results)` since `ex.map` preserves order).
**1. Network-failure exit code (`cli.py:455-458`)**
```python
for r in failed:
c = _phase_code(r.fail_phase)
if c > worst:
worst = c
```
Every failed sample (across every URL, since `worst` is declared once before
the loop) is mapped to an exit code via `_phase_code` / `_PHASE_EXIT`
(`cli.py:24-32`):
```python
_PHASE_EXIT: dict[str, int] = {
"dns": EXIT_DNS, # 2
"timeout": EXIT_TIMEOUT, # 4
"tls": EXIT_TLS, # 5
}
def _phase_code(fail_phase: str) -> int:
return _PHASE_EXIT.get(fail_phase, EXIT_CONNECT) # 3, the fallback
```
Any `fail_phase` not in the table — `"connect"`, `"transfer"`, `"request"` —
falls back to `EXIT_CONNECT` (3). `worst` tracks the running max across all
URLs/samples, so the process exit code always reflects the single worst
failure class seen, per the exit-code table (0 ok … 6 http via `--fail`).
**2. `--fail` (HTTP status ≥ 400) exit code (`cli.py:459-462`)**
```python
if ns.fail:
for r in succeeded:
if r.status_code >= 400:
worst = max(worst, EXIT_HTTP)
```
Only runs when `--fail` is passed. Note this scans `succeeded` — a 404 is not
a network failure, so those `Result`s land in `succeeded` with a populated
`status_code`; `--fail` is what turns "successfully got a bad status" into a
non-zero exit, curl-style.
**3. Verbose-detail selection (`cli.py:464`)**
```python
last_detail = succeeded[-1].detail if succeeded else (failed[-1].detail if failed else None)
```
Prefers the last successful sample's `detail` (freshest full picture: IP,
protocol, TLS, headers); falls back to the last *failed* sample's `detail` if
nothing succeeded (e.g. resolved IP is still known even on a connection
refusal); `None` if there's nothing to show. Reasonable design — surfaces
partial diagnostic info even on total failure.
**4. JSON accumulation vs. text rendering (`cli.py:466-472`)**
```python
if ns.json_out:
json_items.append(_build_json_entry(url, succeeded, failed, last_detail))
continue
if i > 0:
stdout.write("\n")
_print_url(url, succeeded, failed, count, ns.fail, stdout)
```
JSON mode builds up `json_items` (dumped once after the loop) and skips
straight to the next URL via `continue`. Text mode writes a blank-line
separator before every URL block except the first (`i > 0`), then delegates
actual formatting to `_print_url`, which picks one of four branches
(single / aggregate / all-failed / mixed) based on `n_ok`/`n_fail`/`total_count`.
## Does it make sense?
**Yes, structurally.** Combining exit-code accumulation and rendering into one
O(n) pass is reasonable for a CLI at this scale — no need to split into two
loops. The `last_detail` fallback logic is a genuinely good touch. The one
stylistic wrinkle — the DNS/connect/timeout/tls loop uses
`if c > worst: worst = c` while the `--fail` branch uses `worst = max(worst,
EXIT_HTTP)` for the same "keep the running max" purpose — is harmless
inconsistency, not a bug.
**Update (2026-07-02):** the "does a single worst-code exit even make sense
across multiple URLs with different errors" question came back as a follow-up
and led to a real feature — see
[2026-07-02-14-05-hxprobe-run-summary-footer.md](2026-07-02-14-05-hxprobe-run-summary-footer.md).
Short version: the scalar exit code is kept (it's a documented cross-
implementation contract with `latprobe`/Go), but multi-URL runs now get an
end-of-run summary footer tallying every URL's outcome, so the "worst code"
is no longer the only visibility into what happened. That change also
unified the two idioms noted above into one `max(...)` call.
**One real gap, found while checking this: `-n`/`--count` is unvalidated.**
`cli.py:349-355` declares `--count` as `type=int, default=1` with no minimum.
`_run_samples` (`cli.py:64-69`) does `for _ in range(count): ...`, so
`--count 0` (or any negative value) makes the loop body never execute, and
both `succeeded` and `failed` come back empty for that URL. Confirmed live:
```
$ hxprobe --count 0 https://example.com
(0, 0 samples)
exit=0
$ hxprobe --count -2 https://example.com
(0, 0 samples)
exit=0
```
The URL itself is missing from the header, `status` reads `0`, and the exit
code is `0` (success) — because `_print_url` falls into the `elif n_fail == 0`
aggregate branch with `summarize([])`, which returns a bare
`Aggregate()` (all defaults, `url=""`) rather than anything referencing the
actual `url` variable. This is silent garbage output instead of a clear
usage error, and it's inconsistent with how the rest of `run()` already
validates arguments (e.g. the `parser.error(...)` calls for the
`urls`/`--file` mutual-exclusion checks at `cli.py:407-410`). Worth a
`parser.error("count must be >= 1")`-style guard if this is ever picked up —
not fixed here since it wasn't asked for, just flagged as a finding.

130
docs/plans/2026-07-01-23-47-py-hxprobe-httpx.md Normal file
View File

@@ -0,0 +1,130 @@
# Plan: `hxprobe` — httpx-based Python probe (Go-client parity)
## Context
The existing Python `latprobe` package measures per-phase HTTP latency with raw
sockets. That gives an excellent DNS/TCP/TLS/TTFB/Transfer breakdown, but the
cost is that it always speaks **HTTP/1.1** and **does not follow redirects** —
diverging from the Go implementation, whose `http.DefaultClient` negotiates
**HTTP/2** via ALPN, **follows redirects** (up to 10), pools connections, and
verifies TLS by default (confirmed: `go/internal/probe/probe.go` uses
`http.DefaultClient.Do` with no custom transport/`CheckRedirect`).
Goal: add a **second Python implementation, `hxprobe`**, built on the `httpx`
library so it matches the Go client's protocol capabilities (HTTP/2, redirects,
pooling, TLS verification) **while preserving the full 6-phase timing** that is
latprobe's whole point. Python has no equivalent of Go's `net/http/httptrace`,
so the phase breakdown is recovered by instrumenting httpx's network backend.
This is an **additive, Python-only experiment** — intentionally outside the
CLAUDE.md "Go first, then Python port" flow, since the user explicitly asked for
a library-based Python variant. No Go change is required.
## Approach
New sibling package `python/hxprobe/`, reusing everything reusable from
`latprobe` (dataclasses, aggregation, duration parsing, CLI rendering) so the
only genuinely new code is the httpx probe backend.
### Key design: instrumented httpx transport
`httpx` (sync `httpx.Client`) runs on `httpcore`. To recover per-phase timing we
subclass httpcore's sync network backend and time the phases at the socket
level, letting httpx own HTTP framing, HTTP/2, redirects, and keep-alive:
- `connect_tcp(...)` — reimplement DNS + TCP as separate steps (port the
`socket.getaddrinfo` → `socket.connect` split already in
`latprobe/probe.py:167-203`), timestamping **DNS** and **TCP connect**
independently, and capturing the resolved IP.
- `start_tls(...)` — timestamp the **TLS handshake**; pull negotiated version /
cipher / peer cert from the SSL object for verbose mode.
- **TTFB** = headers-received minus end-of-TLS (server processing), measured via
`client.stream("GET", ...)` (the `stream()` context yields once response
headers arrive).
- **Transfer** = iterating `resp.iter_raw()` to EOF, minus headers-received.
- **Total** = wraps the whole `measure()` call.
Each `measure()` call uses a **fresh `httpx.Client` (no cross-sample pooling)** so
every `-n` sample yields a full phase breakdown — matching the current
raw-socket `latprobe` behavior rather than Go's pool-reuse quirk.
A per-call trace object (held by the backend instance) records timings and the
`fail_phase` at the exact point a phase raises, giving precise error
classification (`dns`/`connect`/`timeout`/`tls`/`transfer`/`request`) without
guessing from httpx exception types.
**Redirects (followed by default, Go parity):** DNS/connect/TLS are reported
from the **first** connection (mirrors Go's `connectStart.IsZero()` guard);
TTFB/Transfer/Total span the full followed chain. `redirect_count` and the
negotiated `http_version` (`h2` vs `http/1.1`) are surfaced as new verbose
fields — a genuine capability the socket version lacks.
## Files
**New:**
- `python/pyproject.toml` — project metadata; dependency `httpx[http2]` (pulls
`h2`). Makes `latprobe` + `hxprobe` `pip install -e .`-able; tests still run
via `PYTHONPATH`.
- `python/hxprobe/__init__.py`
- `python/hxprobe/probe.py` — `measure(url, opts) -> latprobe.probe.Result`
(imports & returns the **same `Result`** so aggregation/rendering just work);
`_TimingBackend`, the per-call trace, error classification, verbose capture.
- `python/hxprobe/cli.py` — thin: delegates to `latprobe.cli.run(...)` passing
`measure_fn=hxprobe.probe.measure` (see reuse edit below).
- `python/hxprobe/__main__.py` — `sys.exit(cli.run(sys.argv[1:], ...))`.
- `python/tests/test_hx_probe.py` — hermetic, local `http.server`: phase
presence, **redirect following** (302 handler — validates the Go-parity
feature), connection-refused → `connect`, black-hole port → `timeout`.
- `python/tests/test_hx_cli.py` — hermetic CLI via `run()` with `io.StringIO`
(mirrors `tests/test_cli.py:73-76`).
- `python/tests/test_integration_hx.py` — live, **excluded from default gate**
(filename starts `test_i…`, so the `test_[!i]*.py` glob skips it): probe a
real HTTP/2 host and assert negotiated `http_version == "h2"`; guard with a
`@skipUnless(_online())` like `tests/test_integration.py:29-40`.
- `docs/usage/py-hxprobe.md` — usage doc (what it does, flags, example with
expected output, and an explicit socket-vs-httpx capability comparison table),
per CLAUDE.md.
**Edited (small, backward-compatible):**
- `python/latprobe/cli.py` — parameterize `run()` and `_run_samples()` with an
injectable `measure_fn` (default = current `latprobe.probe.measure`), so
`hxprobe` reuses all argparse, concurrency, exit-code, text/JSON rendering
logic. Extend `_print_verbose_block` and `_build_json_entry` to show
`http_version` / `redirect_count` **when present** (existing socket path never
sets them → output unchanged).
- `python/latprobe/probe.py` — add optional fields with safe defaults:
`Options.follow_redirects=True`, `Options.http2=True` (ignored by the socket
measure); `VerboseDetail.http_version=""`, `VerboseDetail.redirect_count=0`.
- `Makefile` — add `py-deps` (create `.venv`, `pip install -e python`),
`hx-run` (`cd python && python -m hxprobe $(ARGS)`), and fold `test_hx_*` into
the existing `py-test` gate; add `hx-test-integration` for the live h2 check.
- `CHANGELOG.md` — append a timestamped one-line entry on completion.
**Reused as-is:** `latprobe.aggregate.summarize`,
`latprobe.duration.parse_duration`, `latprobe.probe.{Result,Phase,CertInfo}`,
and the entire `latprobe.cli` renderer via the `measure_fn` injection.
## Flags / behavior
Same surface as `latprobe` (`-n/--count`, `-c/--concurrency`, `--timeout`,
`--fail`, `--json`, `-v/--verbose`) via the reused parser, plus two new opt-outs
for the Go-like defaults: `--no-http2` and `--no-follow-redirects`. Exit codes
0–6 stay identical to Go/`latprobe`.
## Verification
1. `make py-deps` — create venv, install `httpx[http2]`.
2. `make hx-run ARGS="https://example.com"` — full 6-phase text output renders.
3. `make hx-run ARGS="-v https://www.cloudflare.com"` — verbose block shows
`http_version: h2` and TLS/cert details.
4. **Go-parity spot checks:**
- HTTP/2: `python -m hxprobe -v <h2-host>` reports `h2` where
`python -m latprobe -v <h2-host>` reports HTTP/1.1.
- Redirects: `python -m hxprobe http://github.com` follows to https and shows
`redirect_count > 0` (socket `latprobe` shows a raw 301).
5. `make py-test` — hermetic suite (now including `test_hx_probe.py`,
`test_hx_cli.py`) is green; `latprobe`'s existing tests still pass (proves the
`measure_fn`/`Options`/`VerboseDetail` edits are backward-compatible).
6. `make hx-test-integration` — live test confirms real `h2` negotiation.
7. Confirm `--json` output for `hxprobe` matches the `latprobe` schema plus the
optional `verbose.http_version` / `verbose.redirect_count` keys.

151
docs/plans/2026-07-02-09-32-hxprobe-standalone-project.md Normal file
View File

@@ -0,0 +1,151 @@
# Plan: extract `hxprobe` into a fully standalone top-level project
## Context
`hxprobe` currently lives at `python/hxprobe/` and imports shared code from
`latprobe` (`latprobe.probe.{Options,Phase,Result,VerboseDetail,_parse_cert}`,
`latprobe.cli.run`). That coupling was a deliberate reuse choice at the time,
but the user now wants `hxprobe` to be a genuinely independent project — no
`from latprobe import ...` anywhere, duplication accepted — and, per their
follow-up answer, independent enough that it could be `cp -r`'d into its own
repo tomorrow: own `pyproject.toml`, own venv, own Makefile section, own
top-level directory (sibling to `go/` and `python/`), not nested under `python/`.
Confirmed via `git status`/`git log`: everything hxprobe-related
(`python/hxprobe/`, `python/pyproject.toml`, `python/tests/test_hx_*.py`,
`docs/usage/py-hxprobe.md`, the two `docs/plans`/`docs/summaries` entries) is
still uncommitted from this session, and `python/latprobe/cli.py` /
`python/latprobe/probe.py` only diverge from the last commit (`24ea9c9`) by
the reuse-oriented additions made to support hxprobe. So this is a clean,
low-risk restructuring: move already-debugged code, revert latprobe with
`git checkout --`, no git history surgery needed.
## Target layout
```
hxprobe/ (new, top-level, sibling of go/ and python/)
├── pyproject.toml (own manifest: httpx[http2] dependency)
├── hxprobe/
│ ├── __init__.py
│ ├── probe.py (own Options/Phase/Result/VerboseDetail/CertInfo
│ │ + own _parse_cert/_parse_cert_date + existing
│ │ _Trace/_TimingStream/_TimingBackend/
│ │ _TimingTransport/measure() — unchanged logic)
│ ├── aggregate.py (verbatim copy of latprobe/aggregate.py —
│ │ its `from .probe import Result` is already
│ │ package-relative, needs zero edits)
│ ├── duration.py (verbatim copy of latprobe/duration.py — no
│ │ imports at all)
│ ├── cli.py (full standalone CLI — see below)
│ └── __main__.py (unchanged: `from .cli import run`)
└── tests/
├── __init__.py (empty, matches python/tests/__init__.py)
├── test_probe.py (moved from python/tests/test_hx_probe.py)
├── test_cli.py (moved from python/tests/test_hx_cli.py)
└── test_integration.py (moved from python/tests/test_integration_hx.py)
```
`python/` reverts to containing only `latprobe` — zero third-party deps, no
`pyproject.toml`, no venv, exactly its pre-hxprobe state.
## Step-by-step
**1. Move already-debugged files (preserve the bug fixes already made):**
`git mv`/`mv` (untracked, so plain `mv` is fine) `python/hxprobe/{probe.py,__init__.py,__main__.py}`
to `hxprobe/hxprobe/`, and the three `python/tests/test_hx_*.py` /
`test_integration_hx.py` files to `hxprobe/tests/` with the `hx_`/`_hx` name
segments dropped (`test_probe.py`, `test_cli.py`, `test_integration.py`).
Do **not** rewrite these from scratch — they already have the mark_dns /
verbose-on-failure-path / Content-Length-on-keep-alive fixes found during
the original implementation.
**2. Inline the dataclasses into `hxprobe/hxprobe/probe.py`:**
Replace `from latprobe.probe import Options, Phase, Result, VerboseDetail, _parse_cert`
with local definitions copied verbatim from `python/latprobe/probe.py`:
`CertInfo`, `VerboseDetail` (its `http_version`/`redirect_count` fields are
now simply always-meaningful, no more "populated only by hxprobe" caveat
comment needed), `Options` (with `follow_redirects`/`http2` as normal fields,
no more "ignored by socket measure()" caveat), `Phase`, `Result`,
`_parse_cert`, `_parse_cert_date`. Everything else in the file (`_Trace`,
`_TimingStream`, `_TimingBackend`, `_TimingTransport`, `measure()`,
`_classify`, `_unwrap`, `_fill_phases`, `_fill_verbose`) is untouched.
**3. Create `hxprobe/hxprobe/aggregate.py` and `duration.py`:**
Verbatim copies of `python/latprobe/aggregate.py` and `duration.py`.
**4. Write `hxprobe/hxprobe/cli.py` as a full standalone CLI:**
Start from the *current* `python/latprobe/cli.py` (it already has 100% of
the needed logic, including the `--no-http2`/`--no-follow-redirects` flags,
the verbose "Protocol" row, and the JSON `http_version`/`redirect_count`
keys — all added earlier specifically for hxprobe). Strip the
generalization scaffolding that only existed to let `latprobe` share this
code:
- Remove the `MeasureFn` type alias and the `measure_fn` parameter from
`_run_samples`/`run` — call `measure` directly (module-level import from
`.probe`).
- Remove `run()`'s `prog`/`description`/`protocol_flags` parameters —
hardcode `prog="hxprobe"` and the httpx-specific description.
- Make the `--no-http2`/`--no-follow-redirects` `add_argument` calls
unconditional (drop the `if protocol_flags:` guard).
- Everything else (exit codes, `_ArgExit`/`_Parser`, phase-label/verbose
constants, all `_print_*`/`_build_json_entry` rendering) carries over
unchanged — it's already correct standalone logic.
**5. Revert `python/latprobe/` to its pre-hxprobe state:**
`git checkout -- python/latprobe/cli.py python/latprobe/probe.py` (safe:
confirmed these are the only diffs since the last commit, and both diffs
are exactly the reuse scaffolding being removed here).
**6. Clean up the old shared-package artifacts:**
Delete `python/hxprobe/`, `python/pyproject.toml`, `python/.venv/`, and the
three `python/tests/test_hx_*`/`test_integration_hx.py` files (now moved).
**7. Makefile:**
- Remove `PY_VENV`/`PY_VENV_PYTHON` vars and the `py-deps` target; revert
`py-test`/`py-test-integration`/`py-check` to invoke `$(PYTHON)` directly
with no `py-deps` prerequisite (their pre-hxprobe form). Remove `hx-run`/
`hx-test-integration` from the Python section (moving out).
- Add `HX_DIR := hxprobe`, `HX_VENV := $(HX_DIR)/.venv`,
`HX_VENV_PYTHON := $(HX_VENV)/bin/$(PYTHON)` and a new "── hxprobe
(standalone) ──" section: `hx-deps` (idempotent venv+pip install, mirrors
the removed `py-deps`), `hx-run`, `hx-test` (hermetic, `test_[!i]*.py`
glob), `hx-test-integration` (`test_integration.py` pattern), `hx-check`,
`hx-clean`.
- Fold into the umbrella targets: `test: go-test py-test hx-test`,
`check: go-check py-check hx-check`, `clean: go-clean py-clean hx-clean`.
**8. Docs:**
- Rename `docs/usage/py-hxprobe.md` → `docs/usage/hxprobe.md` (drop the
`py-` prefix — it's no longer part of the Python port). Update: remove the
"reuses `latprobe.cli.run()` in full" claim (now false — say it's a fully
standalone implementation sharing only the *design*, not the code, with
`latprobe`); update setup/Makefile-target sections to `make hx-deps`/
`hx-run`/`hx-test`/`hx-test-integration`; update file paths from
`python/hxprobe/` to `hxprobe/hxprobe/`. Keep the TCP_NODELAY/Nagle finding
and the example transcripts (still accurate — behavior is unchanged, only
location/packaging changed).
- `CHANGELOG.md`: new entry describing the extraction.
- `docs/summaries/`: new dated summary per the CLAUDE.md convention
(leave the original `2026-07-02-00-29-py-hxprobe-httpx.md` as-is — it's a
historical record of that implementation; this is a follow-up).
- `docs/plans/`: save this plan as
`docs/plans/<yyyy-mm-dd-hh-mm>-hxprobe-standalone-project.md` at
implementation start, per CLAUDE.md.
## Verification
1. `rm -rf hxprobe/.venv python/.venv` (fresh state) then `make hx-deps` —
creates `hxprobe/.venv`, installs `httpx[http2]` from `hxprobe/pyproject.toml`.
2. `grep -rn "latprobe" hxprobe/` — must return nothing (proves the
decoupling).
3. `make hx-run ARGS="-v http://github.com"` — same output as before (HTTP/2,
1 redirect followed, IP/TLS/cert shown).
4. `make hx-test` — all hermetic hxprobe tests pass standalone.
5. `make hx-test-integration` — live HTTP/2 + redirect tests still pass.
6. `make py-test` — confirms `latprobe`'s own suite is back to its original,
dependency-free form and still green (no `py-deps` needed to run it).
7. `make check` (top-level) — Go + latprobe + hxprobe all green in one gate.
8. `diff <(python -m latprobe --help) <(git show 24ea9c9:python/latprobe/cli.py | ...)` —
or simpler: confirm `python -m latprobe --help` output is byte-identical
to before this whole feature existed (no leftover `--no-http2` etc.).

115
docs/plans/2026-07-02-09-57-hxprobe-toolchain-modernization.md Normal file
View File

@@ -0,0 +1,115 @@
# Plan: modernize hxprobe's Python toolchain
## Context
`hxprobe` (`hxprobe/`) is a fully standalone Python project (own
`pyproject.toml`, own venv, own tests — see
`docs/summaries/2026-07-02-09-32-hxprobe-standalone-project.md`). Its
toolchain is currently bare-minimum: hand-rolled `python -m venv` + `pip
install -e .`, no linter/formatter, no type checker, no dependency lockfile,
and stdlib `unittest` with a filename-glob convention
(`test_[!i]*.py`/`test_integration.py`) to separate hermetic from live tests.
User confirmed direction (via AskUserQuestion): adopt **uv** for env/deps
(with a lockfile), add **ruff** for lint+format, **skip mypy** for now, and
**swap the test runner to pytest** (existing `unittest.TestCase` classes run
unchanged under pytest — no test-code rewrite) using a proper
`@pytest.mark.integration` marker instead of the filename-glob trick.
Confirmed via inspection: `uv` (v0.11.19) is already installed globally
(Homebrew); `ruff`/`mypy`/`pytest` only exist under an unrelated pyenv 3.12.3
shim, not on the 3.14 interpreter this project targets — `uv` sidesteps that
mismatch by installing everything into the project's own venv. Git remote is
a self-hosted Gitea instance, not GitHub, so no CI is being set up here.
## Changes
**`hxprobe/pyproject.toml`:**
- Add `[dependency-groups]` with `dev = ["pytest>=8.0", "ruff>=0.8"]` (PEP
735, the current uv-native way to declare dev-only deps — keeps the
install-as-a-library `dependencies` list clean).
- Add `[tool.pytest.ini_options]`: `testpaths = ["tests"]` and a registered
`integration` marker (avoids `PytestUnknownMarkWarning`).
- Add `[tool.ruff]`: `target-version = "py311"` (matches `requires-python`)
and `line-length = 100` (close to the codebase's existing longest lines,
~103 chars, to minimize reformatting churn).
- Leave `[build-system]`/`[tool.setuptools]` untouched — build backend
wasn't part of the discussion, setuptools works fine here.
**`hxprobe/.python-version`:** new file, `3.14`, so `uv sync`/`uv run` pin the
same interpreter the rest of the repo uses without relying on `$PATH` order.
**`hxprobe/uv.lock`:** generated by `uv lock` — first real lockfile pinning
`httpx`, `httpcore`, `h2`, `certifi`, and friends. Committed (not gitignored;
lockfiles belong in version control).
**Test changes (runner swap only, no rewrite):**
- `hxprobe/tests/test_integration.py`: add `pytestmark = pytest.mark.integration`
at module level. This replaces the "named `test_integration.py` so the
`test_[!i]*.py` glob skips it" convention — selection becomes `-m
"not integration"` / `-m integration`, independent of filename.
- `test_probe.py`/`test_cli.py`: no changes — they're already hermetic
(local `http.server` fixtures only) and need no marker.
- `unittest.TestCase` classes, `_NEEDS_NET = unittest.skipUnless(...)`, and
the `if __name__ == "__main__": unittest.main()` guards all stay exactly
as-is — pytest natively discovers and runs unittest-style tests and
respects `unittest.skip*` decorators with zero changes required.
**Lint fixes:** run `ruff check --fix` / `ruff format` over `hxprobe/` and
review the diff. One known pre-existing issue it will flag: `import sys` in
`cli.py` is unused (inherited from the original `latprobe/cli.py`) — remove
it. Otherwise expect mostly whitespace/quote-style normalization.
**`Makefile`:** replace the `hx-*` section to run through `uv` instead of a
hand-managed venv:
```makefile
HX_DIR := hxprobe # (drop HX_VENV / HX_VENV_PYTHON — uv owns this now)
hx-deps: cd $(HX_DIR) && uv sync
hx-run: (deps: hx-deps) cd $(HX_DIR) && uv run python -m hxprobe $(ARGS)
hx-lint: (deps: hx-deps) cd $(HX_DIR) && uv run ruff check .
hx-fmt: (deps: hx-deps) cd $(HX_DIR) && uv run ruff format .
hx-test: (deps: hx-deps) cd $(HX_DIR) && uv run pytest tests -m "not integration" -v $(ARGS)
hx-test-integration: (deps: hx-deps) cd $(HX_DIR) && uv run pytest tests -m integration -v $(ARGS)
hx-check: hx-lint hx-test (test gate now includes lint, mirroring go-check's fmt+vet+test bundling)
hx-clean: also removes .pytest_cache / .ruff_cache alongside __pycache__/egg-info
```
Umbrella `test`/`check`/`clean` targets keep delegating to `hx-test`/
`hx-check`/`hx-clean` unchanged.
**`hxprobe/README.md`:** new, minimal — since this project is meant to be
`cp -r`-able to its own repo, it should carry its own quick-start
(`uv sync`, `uv run python -m hxprobe <url>`, `uv run pytest`) rather than
relying on the monorepo's root docs.
**Docs:** update `docs/usage/hxprobe.md`'s "Setup" section (`uv sync`
instead of manual venv+pip) and "Makefile targets" section (add
`hx-lint`/`hx-fmt`, update test invocation description). New
`docs/plans/<timestamp>-hxprobe-toolchain-modernization.md` and
`docs/summaries/<timestamp>-hxprobe-toolchain-modernization.md` per
CLAUDE.md convention. New `CHANGELOG.md` entry.
**`.gitignore`:** already covers `.venv/`/`__pycache__/`/`*.egg-info/`
unanchored; add `.pytest_cache/` and `.ruff_cache/` (new caches these tools
create).
## Verification
1. `cd hxprobe && uv sync` — creates `.venv`, generates/uses `uv.lock`,
installs `httpx[http2]` + dev deps (`pytest`, `ruff`).
2. `make hx-run ARGS="-v http://github.com"` — same output as before
(HTTP/2, 1 redirect, IP/TLS/cert shown) — confirms the runtime behavior
is untouched by the toolchain swap.
3. `make hx-lint` — clean (after fixing whatever `ruff check` surfaces,
including the unused `import sys`).
4. `make hx-test` — all hermetic tests pass under pytest; confirm the
`integration`-marked tests are excluded (test count matches the current
28 hermetic tests).
5. `make hx-test-integration` — the 9 live tests run and pass under the
`integration` marker selection.
6. `make check` (top-level) — Go + latprobe + hxprobe (lint + test) all
green in one gate.
7. `grep -rn "latprobe" hxprobe/` — still zero matches (toolchain change
must not reintroduce coupling).
8. Confirm `python/latprobe/` is untouched (`git diff --stat python/` empty)
— this is a hxprobe-only change.

112
docs/plans/2026-07-02-10-22-hxprobe-usage-doc-and-makefile.md Normal file
View File

@@ -0,0 +1,112 @@
# Plan: hxprobe-specific usage doc (case-by-case) + hxprobe-specific Makefile
## Context
`hxprobe` is a fully standalone project (own `pyproject.toml`, `uv.lock`,
`README.md` — see `docs/summaries/2026-07-02-09-32-hxprobe-standalone-project.md`
and `...-toolchain-modernization.md`). Two gaps remain versus the other
Python implementations and versus hxprobe's own "could be `cp -r`'d to its
own repo" design goal:
1. **Docs**: `latprobe`/`phases.py`/`simple.py` each have two usage docs —
the CLAUDE.md-mandated `docs/usage/py-<name>.md` (what/flags/one example)
_and_ a much richer `python/configs/usage-<name>.md` "Runnable Usage
Reference" walking through a full set of concrete cases with real
captured output (confirmed by reading `python/configs/usage-latprobe.md`,
313 lines: basic, verbose × 4 variants, sampling, multi-URL, `--fail`,
JSON × 2, timeout, exit-codes table, Makefile shortcuts). `hxprobe` only
has the first kind. User confirmed: add the second kind, placed _inside_
`hxprobe/` itself (not under `python/configs/`, since hxprobe no longer
lives there) so the doc travels with the project if extracted.
2. **Makefile**: hxprobe currently has no `Makefile` of its own — the only
way to run/test/lint it is through the parent repo's root `Makefile`.
Extracted to its own repo, there'd be no `make` interface left. User
confirmed: add `hxprobe/Makefile`, fully independent from the root
Makefile's existing `hx-*` targets (no delegation either direction —
both keep their own complete logic, at the cost of some duplication).
Confirmed: neither `latprobe` nor `hxprobe` accept a config file (both take
URLs as positional CLI args — only `simple.py`/`phases.py` read the
`python/configs/*.txt` files), so no `.txt`-config-file equivalent is needed
for hxprobe; this is a docs+Makefile-only task.
## Changes
**`hxprobe/USAGE.md`** (new) — modeled directly on
`python/configs/usage-latprobe.md`'s structure and tone (concrete `sh`
command blocks immediately followed by real captured output, real IPs/certs/
timings, brief explanatory notes, `---` section separators). Cases, in order:
1. Basic — single URL
2. Verbose — HTTPS site (shows `Protocol: HTTP/2`, TLS, cert)
3. Verbose — plain HTTP (no TLS block)
4. Verbose — redirect followed by default (`http://github.com` → 200,
`Protocol: HTTP/2 (1 redirect)`) — **hxprobe-specific**, latprobe has no
equivalent
5. `--no-follow-redirects` — same URL, raw `301` instead — **hxprobe-specific**
6. `--no-http2` — forces `Protocol: HTTP/1.1` — **hxprobe-specific**
7. Verbose — TLS failure (expired cert, badssl.com)
8. Verbose — DNS failure (empty verbose block, suppressed)
9. Sampling (`-n`) — min/avg/max table, plus verbose+sampling
10. Multiple URLs (parallel probing)
11. `--fail` flag — exit 6 on HTTP 4xx
12. JSON output
13. JSON + verbose (includes `http_version`/`redirect_count` keys)
14. Timeout
15. Exit codes table (same 0–6 scheme as `latprobe`/Go)
16. Makefile shortcuts — both the new `hxprobe/Makefile` (`make run`,
`make test`, etc., run from inside `hxprobe/`) and the parent repo's
root shortcuts (`make hx-run`, run from the repo root)
Cases 1, 2, 3, 4, 5, 6, 8, 11 can reuse real output already captured earlier
in this session (still accurate — no code changed since). Cases 7, 9, 10,
12, 13, 14 need fresh live runs during implementation to get real numbers
(same standard the other `usage-*.md` docs hold themselves to — no
fabricated timings).
**`docs/usage/hxprobe.md`** (edit) — add a one-line pointer near the top:
"For a full case-by-case runnable reference, see `hxprobe/USAGE.md`." No
other changes; it stays the CLAUDE.md-mandated summary doc.
**`hxprobe/Makefile`** (new) — fully self-sufficient, same auto-generated
`## comment` help style as the root Makefile, short target names (no `hx-`
prefix needed since it's already scoped by being inside `hxprobe/`):
```makefile
PYTHON ?= python3.14
ARGS ?=
.DEFAULT_GOAL := help
help # auto-generated from ## comments, same style as root Makefile
deps # uv sync
run # uv run python -m hxprobe $(ARGS) (deps: deps)
lint # uv run ruff check . (deps: deps)
fmt # uv run ruff format . (deps: deps)
test # uv run pytest tests -m "not integration" -v $(ARGS) (deps: deps)
test-integration # uv run pytest tests -m integration -v $(ARGS) (deps: deps)
check # lint + test
clean # remove __pycache__/*.pyc/*.egg-info/.pytest_cache/.ruff_cache
```
No changes to the root `Makefile` — its existing `hx-*` targets are left
exactly as-is per the "keep both independent" decision.
**Docs housekeeping** (per CLAUDE.md convention): save this plan to
`docs/plans/<timestamp>-hxprobe-usage-doc-and-makefile.md`, write a summary
to `docs/summaries/<timestamp>-hxprobe-usage-doc-and-makefile.md` after
implementation, and append a `CHANGELOG.md` entry.
## Verification
1. `cd hxprobe && make help` — lists all targets with descriptions, works
with zero dependency on the parent repo's Makefile.
2. `cd hxprobe && make run ARGS="-v https://example.com"` — same output as
`make hx-run ARGS="-v https://example.com"` from the repo root (proves
the two Makefiles agree, without one calling the other).
3. `cd hxprobe && make check` — lint + hermetic tests pass (28 tests).
4. `cd hxprobe && make test-integration` — 9 live tests pass.
5. Re-run every command block in `hxprobe/USAGE.md` and confirm the
captured output matches what's printed in the doc (structure must be
stable even if exact millisecond timings drift).
6. Confirm the root `Makefile`'s `hx-*` targets are byte-for-byte unchanged
(`git diff Makefile` shows no `hx-*` section changes from this task).

102
docs/plans/2026-07-02-11-14-hxprobe-file-input.md Normal file
View File

@@ -0,0 +1,102 @@
# Plan: read target URLs from a file for hxprobe
## Context
`hxprobe` currently only accepts URLs as positional CLI arguments
(`hxprobe/hxprobe/cli.py:368`, `nargs="+"`). The user wants a file-based
input mode too, matching the existing convention `simple.py`/`phases.py`
already use (`python/simple.py:20-30`'s `load_sites()`: plain text, one URL
per line, `#`-comments and blank lines skipped, first whitespace-separated
token taken per line).
User-confirmed decisions:
- **Mutually exclusive** with positional URL args (either pass URLs on the
command line, or `-f FILE`, never both).
- **hxprobe only** — `latprobe` is intentionally left untouched.
- **Add example fixture files** (`hxprobe/configs/*.txt`, mirroring
`python/configs/*.txt`'s exact set: all-ok, dns-failure,
connection-refused, timeout, tls-errors, http-errors, mixed) plus one new
section in the existing `hxprobe/USAGE.md` demonstrating the flag.
## Implementation
**`hxprobe/hxprobe/cli.py`:**
- `urls` positional becomes `nargs="*"` (was `nargs="+"`) — no longer
required on its own, since `-f` is now a second valid source.
- New flag: `-f, --file PATH` — "read URLs from a file, one per line,
`#` comments allowed (mutually exclusive with positional url args)".
Placed right after the `urls` positional definition in the argparse
block, since the two are the two ways of specifying what to probe.
- New helper `_load_urls(path: str) -> list[str]`, duplicating (not
importing) `simple.py`'s `load_sites()` logic — consistent with hxprobe's
established "imports nothing outside its own directory" rule from the
standalone-extraction work.
- After `parser.parse_args()`, manual validation (mirrors the existing
`--timeout` invalid-value handling style — write to the injected
`stderr`, `return EXIT_USAGE`, rather than routing through
`argparse`'s mutually-exclusive-group machinery, which doesn't mix
cleanly with a variadic positional):
- both `ns.urls` and `ns.file` given → `parser.error(...)` (usage error,
consistent with how `_Parser.error()` already handles bad usage)
- neither given → `parser.error(...)`
- `ns.file` given but unreadable (`FileNotFoundError`/`OSError`) →
`stderr.write(...)`; `return EXIT_USAGE`
- `ns.file` given but yields zero URLs → same treatment
- otherwise `urls = _load_urls(ns.file)` or `urls = ns.urls`
**`hxprobe/tests/test_cli.py`:** new hermetic tests — successful multi-URL
run from a file, missing-file error, empty-file error, and the
both-sources-given usage error. Uses a temp file (`tempfile`), no network
needed for the parsing-error cases.
**`hxprobe/configs/*.txt`** (new directory) — same 7 fixtures as
`python/configs/`, adapted:
- `all-ok.txt`, `dns-failure.txt`, `connection-refused.txt`,
`tls-errors.txt` — same URLs, same behavior (DNS/TCP/TLS failures are
identical regardless of HTTP client sophistication); only the header
comments change (`hxprobe -f configs/<name>.txt` instead of
`python3.14 python/simple.py ...`).
- `timeout.txt` — same two targets (`10.255.255.1`, `192.0.2.1`, RFC 5737
TEST-NET-1) as the original; "expected exit code" documents normal-network
behavior (exit 4), same caveat the original file already carries about
network-dependent behavior.
- `http-errors.txt` — same 404 URLs; header comment updated to show the
demo command with `--fail` (hxprobe treats 4xx as success without
`--fail`, unlike `simple.py`, which always raises on HTTPError) —
"expected exit code" becomes 6, not `simple.py`'s blanket 1.
- `mixed.txt` — same mixed set; demo command includes `--fail`; expected
exit code recalculated as the worst code across the included classes
(dns=2, connect=3, tls=5, http=6 with `--fail`) → 6.
- Each header's "Expected exit code" will be verified by actually running
the fixture through `hxprobe -f ...` during implementation, not assumed
from the `simple.py` originals — hxprobe's worst-code-wins exit scheme
(`hxprobe/hxprobe/cli.py:455-481`) differs fundamentally from
`simple.py`'s blanket 0/1.
**`hxprobe/USAGE.md`:** new section "Reading URLs from a file (`-f`)",
placed after the "Multiple URLs" case (same family of "what to probe"
examples) — command + real captured output using `configs/all-ok.txt` or
`configs/mixed.txt`, plus a short list of the other fixture files available
and what each demonstrates.
**Docs housekeeping** (per CLAUDE.md convention): save this plan under
`docs/plans/`, write a summary under `docs/summaries/` after implementation,
append a `CHANGELOG.md` entry.
## Verification
1. `cd hxprobe && uv run python -m hxprobe -f configs/all-ok.txt` — probes
all 3 URLs, exit 0.
2. `cd hxprobe && uv run python -m hxprobe -f configs/dns-failure.txt` —
exit 2; `connection-refused.txt` → exit 3; `tls-errors.txt` → exit 5;
`--fail -f configs/http-errors.txt` → exit 6; `--fail -f configs/mixed.txt`
→ exit 6 (confirms the worst-code documented in each header is accurate).
3. `uv run python -m hxprobe https://example.com -f configs/all-ok.txt` —
usage error (both sources given).
4. `uv run python -m hxprobe -f /no/such/file` — usage error, clear message.
5. `cd hxprobe && make check` — new hermetic tests pass alongside the
existing 28.
6. Re-run the new `hxprobe/USAGE.md` section's command and confirm captured
output matches what's printed in the doc.
7. `grep -n "import" hxprobe/hxprobe/cli.py` — confirm no new import from
`python/simple.py` or anywhere outside `hxprobe/`.

174
docs/plans/2026-07-02-12-05-hxprobe-simplification.md Normal file
View File

@@ -0,0 +1,174 @@
# hxprobe simplification plan
## Context
The `hxprobe` Python package works and is well-tested, but it carries several
pieces of **speculative scaffolding** — generality added for things that "might
happen later" — plus some **redundant state** and **duplicated logic**. The
user's explicit directive: keep the code simple, with no unnecessary
abstractions or scaffolding for hypothetical future features.
This plan removes that overhead without changing any observable behavior. All
existing tests in `tests/test_cli.py` and `tests/test_probe.py` must continue
to pass unchanged (they are the behavioral contract). Net effect: ~40–50 fewer
lines, fewer moving parts, no new abstractions.
The trigger was a question about the `_Parser._print_message`/`print_help`/
`print_usage` overrides, whose `file=` parameter is accepted but never used —
the classic "kept for a future that never came" smell. Investigation showed the
whole `_Parser` subclass re-implements behavior the standard library already
provides.
---
## Changes
### 1. Delete the `_Parser` subclass — use stdlib stream redirection
**File:** `hxprobe/hxprobe/cli.py` (lines 60–94, and the `run()` wiring ~375–455)
The `_Parser` class + `_ArgExit` exception (~32 lines) exist only to (a) route
argparse's help/usage/error output to the injected `stdout`/`stderr` streams and
(b) raise instead of calling `sys.exit()`. The standard library already does
both:
- `argparse.ArgumentParser` writes help to `sys.stdout` and usage/errors to
`sys.stderr` by default, and already raises `SystemExit` (not a hard exit) —
so it is already testable.
- `contextlib.redirect_stdout(stdout)` / `redirect_stderr(stderr)` patch the
streams argparse writes to.
**Do:**
- Remove `class _ArgExit`, `class _Parser`, and all four overrides
(`_print_message`, `print_help`, `print_usage`, `error`, `exit`).
- In `run()`, build a plain `argparse.ArgumentParser(prog="hxprobe", ...)` (drop
the `out=`/`err=` kwargs).
- Wrap the parse + the two manual validations in redirection and catch
`SystemExit`:
```python
import contextlib
...
try:
with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
ns = parser.parse_args(args)
if ns.urls and ns.file:
parser.error("cannot combine positional url arguments with -f/--file")
if not ns.urls and not ns.file:
parser.error("no URLs given (pass as arguments or with -f/--file)")
except SystemExit as exc:
return EXIT_OK if not exc.code else EXIT_USAGE
```
**Behavior parity (verified against tests):**
- `-h` → argparse prints help to `stdout`, raises `SystemExit(0)` → returns
`EXIT_OK`. (`test_help_shows_hxprobe_prog_name`, `test_help_lists_protocol_flags`)
- `parser.error(...)` → prints `prog: error: msg` + usage to `stderr`, raises
`SystemExit(2)` → mapped to `EXIT_USAGE`. (`test_*_is_usage_error`)
- Note: stdlib `error()` exits with code 2; we normalise any non-zero parse exit
to `EXIT_USAGE` (1) at the single catch site, replacing the per-method code
baked into the old override.
Only the parse/validate block needs redirection; the rest of `run()` keeps
writing directly to the passed `stdout`/`stderr`.
### 2. Trim `_TimingStream.get_extra_info` to the branch that's actually used
**File:** `hxprobe/hxprobe/probe.py` (lines 262–275)
`server_addr` and `client_addr` are only ever queried by `httpx/_main.py` (the
`httpx` CLI command), never by the request path hxprobe drives — verified by
grepping the installed `httpcore`/`httpx`. The only branch httpcore's sync
connection path calls is `ssl_object` (and `is_readable`, which we intentionally
leave unhandled → `None`).
**Do:** reduce the method to:
```python
def get_extra_info(self, info: str):
if info == "ssl_object" and isinstance(self._sock, ssl.SSLSocket):
return self._sock
return None
```
### 3. Drop the `_load_urls` "future annotations" scaffolding
**File:** `hxprobe/hxprobe/cli.py` (lines 358–369)
The `line.split()[0]` + docstring ("forward-compatible with future
'url key=value' annotations") is scaffolding for a feature that doesn't exist.
Use the stripped line directly and simplify the docstring to describe what it
actually does (one URL per line, `#` comments and blank lines skipped).
`test_reads_urls_from_file` (comments + blanks) still passes.
### 4. Remove redundant `_set` flags in `_Trace`
**File:** `hxprobe/hxprobe/probe.py` (lines 126–191)
`_dns_set`, `_connect_set`, `_tls_set` duplicate information already carried by
`Phase.present` on the corresponding `self.dns` / `self.connect` / `self.tls`.
The initial `Phase()` has `present=False`, so the first-hop-wins guard is
identical.
**Do:** delete the three boolean fields; replace each guard, e.g.
`if not self._dns_set:` → `if not self.dns.present:` (same for connect/tls).
Keeps the redirect first-hop-wins semantics documented at probe.py:114–123.
### 5. De-duplicate the failure counting logic
**File:** `hxprobe/hxprobe/cli.py`
The identical "dedupe failures into ordered (phase, message, count)" loop appears
twice: `_print_failure_summary` (lines 242–258) and `_build_json_entry`
(lines 314–325). Extract one helper next to the other rendering helpers:
```python
def _summarize_failures(failed: list[Result]) -> list[tuple[str, str, int]]:
counts: dict[tuple[str, str], int] = {}
order: list[tuple[str, str]] = []
for r in failed:
key = (r.fail_phase, str(r.err))
if key not in counts:
order.append(key)
counts[key] = 0
counts[key] += 1
return [(ph, msg, counts[(ph, msg)]) for ph, msg in order]
```
Rewrite both call sites to consume it (text side keeps the `1 ×` vs `N ×`
formatting; JSON side maps to `{"phase", "count", "message"}`).
---
## Explicitly NOT changing (considered, kept)
- **The custom httpcore backend** (`_TimingBackend`/`_TimingStream`/
`_TimingTransport`) — this *is* the tool's reason to exist (splitting DNS/TCP,
timing TLS). Not scaffolding.
- **`ThreadPoolExecutor` concurrency** — backs the shipped `-c/--concurrency`
and multi-URL/`-f` features. Real, not speculative.
- **Explicit per-phase dataclass fields** in `probe.py`/`aggregate.py` — a loop
would be shorter but less readable; explicit is clearer and matches the text/
JSON renderers. Leave as-is.
- **`CertInfo.sans`** — not shown in the text block but *is* emitted in JSON
verbose output; it's a real feature, not dead.
---
## Verification
1. **Unit/CLI tests (primary contract):**
```
cd hxprobe && .venv/bin/python -m pytest tests/test_cli.py tests/test_probe.py -q
```
All must pass with no edits to the test files. These already cover: `-h`
help→stdout + exit 0, the four usage errors→stderr + exit 1, DNS/connect/
timeout failures, `--fail`, `--json`, redirects, `--no-http2`,
`--no-follow-redirects`, `-f` file input, and verbose/cert/TLS detail.
2. **Lint:** `cd hxprobe && .venv/bin/ruff check hxprobe/`
3. **Manual smoke (help + error routing, since #1 rewrites that path):**
```
.venv/bin/python -m hxprobe -h # help on stdout, exit 0
.venv/bin/python -m hxprobe # "no URLs given" on stderr, exit 1
.venv/bin/python -m hxprobe https://example.com -v # phases + verbose block
```
4. **Per-CLAUDE.md project conventions:** after implementing, add a
`docs/summaries/<yyyy-mm-dd-hh-mm>-hxprobe-simplification.md` summary and a
`CHANGELOG.md` entry. (No `docs/usage/` change — behavior is unchanged.)

136
docs/plans/2026-07-02-14-05-hxprobe-run-summary-footer.md Normal file
View File

@@ -0,0 +1,136 @@
# hxprobe: end-of-run summary footer for multi-URL runs
## Context
A single scalar exit code is inherently lossy when probing multiple URLs with
different failure classes: today `run()` returns the numeric max across all
URLs (e.g. DNS-fail on one URL + TLS-fail on another → exit `5`, and the DNS
failure is invisible in the code). The failures *are* all printed per-URL, but
there's no consolidated view — for many URLs you must scroll and eyeball each
block to know what happened overall.
The exit code's job is a coarse pass/fail + severity hint for scripts, and it's
a **deliberate cross-implementation contract** shared with `latprobe` and the
Go version (documented in `hxprobe/USAGE.md:475-488`, asserted by 13 tests in
`tests/test_cli.py`). So we keep the worst-code exit unchanged and instead give
humans the full picture the scalar can't: an **end-of-run summary footer** that
tallies every URL's outcome and shows how the exit code was derived.
Decisions (confirmed with the user):
- Exit code: **unchanged** (worst/highest severity across URLs).
- Summary: **text footer, multi-URL only** (`len(urls) > 1`). Single-URL text
output stays byte-identical; JSON output stays a bare array (unchanged).
## Design
### Per-URL classification (reuses existing helpers)
Each URL gets one "worst outcome code" using the *existing* mapping — no new
severity scheme:
- start at `EXIT_OK`;
- for each failed sample: `code = max(code, _phase_code(r.fail_phase))`
(`_phase_code` at `cli.py:31`);
- if `--fail`: for each succeeded sample with `status_code >= 400`:
`code = max(code, EXIT_HTTP)`.
A URL is "ok" iff its code is `EXIT_OK`, else "failed" and bucketed by its code.
A partially-failed URL (some samples ok, some failed) classifies by its worst
sample — consistent with how its own block and the global exit code already
behave.
### Footer format (text, only when `len(urls) > 1`)
Printed once after the last URL block, before `return worst`. Failure classes
use the same `✗` bullet style as `_print_failure_summary` (`cli.py:219-224`).
The final `→ exit N (label)` line explicitly ties the tally to the returned
code — directly answering "why is the exit code what it is". Example (3 URLs):
```
https://example.com (200)
... phase rows ...
http://no.such.host.invalid (FAILED)
✗ dns: [Errno 8] nodename nor servname provided
https://self-signed.badssl.com (FAILED)
✗ tls: certificate verify failed
─────────────────────────────────────────────────
Summary: 3 URLs — 1 ok, 2 failed
✗ dns : 1
✗ tls : 1
→ exit 5 (tls)
```
All-ok multi-URL run → `Summary: 3 URLs — 3 ok`, no `✗` lines, `→ exit 0 (ok)`.
## Changes
All in `hxprobe/hxprobe/cli.py` unless noted.
1. **Reverse label map** next to the exit-code constants (`cli.py:16-32`): a
small `_EXIT_LABELS: dict[int, str]` mapping `EXIT_DNS→"dns"`,
`EXIT_CONNECT→"connect"`, `EXIT_TIMEOUT→"timeout"`, `EXIT_TLS→"tls"`,
`EXIT_HTTP→"http"`, `EXIT_OK→"ok"`. (Inverse of the existing forward mapping;
kept explicit for readability, matching the codebase's style.)
2. **Refactor the accumulation loop** (`cli.py:454-462`) to compute a per-URL
code and fold it into `worst`, collecting `url_codes: list[int]` (one per
URL, index-aligned with `urls`). This also unifies the two "running max"
idioms flagged in
`docs/explanations/2026-07-02-13-25-hxprobe-worst-exit-code-and-render-loop.md`
(`if c > worst` vs `max(...)`) into one — a small simplification bonus.
3. **New `_print_run_summary(urls, url_codes, worst, out)` helper** (near the
other `_print_*` renderers): builds the ok/failed tally + per-class counts
from `url_codes` and writes the footer. No-op guard is the caller's
`len(urls) > 1` check.
4. **Call site** after the loop (`cli.py:470-475`): in text mode only
(`if not ns.json_out and len(urls) > 1:`), call `_print_run_summary(...)`
before `return worst`. JSON path untouched — still `json.dumps(json_items)`
as a bare array.
## Tests (`hxprobe/tests/test_cli.py`)
Add hermetic tests (reuse the existing `_OKHandler`/`_start_server`/`_invoke`
harness and `_free_port` for a refused connection):
- **multi-URL mixed** — one OK server URL + one `http://127.0.0.1:<free>`
(connection refused): assert `code == EXIT_CONNECT`, and the footer strings
are present (`"Summary: 2 URLs"`, `"1 ok"`, `"1 failed"`, `"connect : 1"`,
`"→ exit 3"`).
- **multi-URL all ok** — two OK URLs: assert `code == EXIT_OK` and
`"Summary: 2 URLs — 2 ok"` present; verify no `"✗"` in the footer region.
- **single URL has NO footer** — one OK URL: assert `"Summary:"` NOT in `out`
(locks the multi-URL-only rule).
Backward-compat guard: the footer contains no `"200"` substring, so the
existing `test_reads_urls_from_file` assertion `out.count("200") == 2` still
holds; all 13 existing exit-code tests are unaffected (worst-code unchanged).
## Docs / project conventions (per CLAUDE.md)
- **Copy this approved plan** into
`docs/plans/<yyyy-mm-dd-hh-mm>-hxprobe-run-summary-footer.md` as the first
implementation step (before code) — see the `feedback_plan_mode_docs_plans`
memory note.
- `hxprobe/USAGE.md`: add a short "Multi-URL summary footer" subsection with a
real captured example; add a sentence to the Exit-codes section noting the
footer shows the per-class breakdown behind the scalar. Exit-code table
itself is unchanged.
- `docs/usage/hxprobe.md`: one-line mention if it lists features.
- New `docs/explanations/<ts>-hxprobe-run-summary-footer.md` is optional; the
existing worst-exit-code explanation can get a short "Update:" pointer.
- `CHANGELOG.md`: new timestamped entry.
- `docs/summaries/<ts>-hxprobe-run-summary-footer.md`: implementation summary.
## Verification
1. `cd hxprobe && .venv/bin/python -m pytest tests/test_cli.py tests/test_probe.py -q`
— all existing + new tests pass.
2. `.venv/bin/ruff check hxprobe/` — clean.
3. Manual, capturing exit codes:
```
.venv/bin/python -m hxprobe https://example.com https://example.org # footer, exit 0
.venv/bin/python -m hxprobe https://example.com http://no.such.host.invalid; echo $? # footer w/ dns:1, exit 2
.venv/bin/python -m hxprobe https://example.com # single URL: NO footer, unchanged
.venv/bin/python -m hxprobe --json https://example.com https://example.org # bare JSON array, NO footer
```

139
docs/py-latprobe-walkthrough.md Normal file
View File

@@ -0,0 +1,139 @@
# latprobe (Python) — code walkthrough
A guided tour of `python/latprobe/` for anyone about to edit the code by hand.
For CLI usage/flags, see [docs/usage/py-latprobe.md](usage/py-latprobe.md).
## Layout
```
python/
├── simple.py step 1 (standalone) — reachability + total elapsed time
├── phases.py step 2 (standalone) — per-phase timing, single file
├── sites.txt sample config for simple.py / phases.py
├── configs/ fixture configs (all-ok, dns-failure, timeout, tls-errors, ...)
├── latprobe/ step 3 — the real package
│ ├── __main__.py
│ ├── probe.py
│ ├── aggregate.py
│ ├── duration.py
│ └── cli.py
└── tests/
├── test_probe.py
├── test_cli.py
└── test_integration.py
```
`simple.py` and `phases.py` are earlier milestones, kept as reference (mirrors
the Go step-by-step convention: simple → phases → full package). They are not
imported by the package.
## The package: `python/latprobe/`
### `probe.py` — the engine
No argparse, no printing. `measure(url, opts) -> Result` does one HTTP(S)
request over raw sockets (not `requests`/`urllib`) so it can time each phase
itself:
- DNS (`socket.getaddrinfo`)
- TCP connect
- TLS handshake (`ssl.wrap_socket`, HTTPS only)
- TTFB (write request → first byte, reading until `\r\n\r\n`)
- Transfer (drain the rest of the body)
Each phase is timed with `time.perf_counter()` and wrapped by the `_p()`
helper into a `Phase(ms, present)`.
Every failure path (DNS error, connect timeout, TLS error, etc.) returns
early with a **partially-filled** `Result` — `fail_phase` says where it died,
`err` holds the exception, and whatever phases completed before the failure
are preserved. This is why the CLI can still print e.g. "DNS: 12ms" even if
TCP connect then failed.
`Options(timeout, verbose)` goes in, `Result` comes out. When `verbose=True`,
`Result.detail` (`VerboseDetail`) is also filled in: resolved IP, TLS
version/cipher, parsed certificate (`CertInfo`), and all response headers.
This is the only file with actual measurement logic — change *how* something
is measured here (e.g. add a redirect-follow phase, redefine what counts as
TTFB).
### `aggregate.py` — pure math, no I/O
`summarize(results: list[Result]) -> Aggregate` takes a list of successful
`Result`s (from repeated `-n` sampling of the same URL) and computes
min/avg/max per phase into `PhaseStats`. `Aggregate.status_code` uses the
*last* result's status. Small and self-contained — safe to extend (e.g. add
p50/p95) without touching anything else.
### `duration.py`
One function: `parse_duration("500ms" | "10s" | "2m") -> float` seconds, used
for `--timeout`.
### `cli.py` — everything else
Argument parsing, orchestration, and both text/JSON rendering. The file
you'll touch most for UX changes.
- `_Parser` subclasses `argparse.ArgumentParser` to redirect all output
through injected `stdout`/`stderr` streams and raise `_ArgExit` instead of
calling `sys.exit` — this is what makes `run()` fully testable without
subprocess (tests just pass in `io.StringIO()`).
- Exit codes (`EXIT_DNS=2`, `EXIT_CONNECT=3`, etc.) are commented as
mirroring the Go version. `_phase_code()` maps a `fail_phase` string to the
matching code, and `run()` tracks the *worst* code across all URLs/samples.
- `run(args, stdout, stderr) -> int` is the entry point:
1. parse args → build `Options`
2. run `_run_samples()` per URL concurrently via `ThreadPoolExecutor`
(`--concurrency`, defaulting to `min(len(urls), 8)`)
3. for each URL, pick one of four print paths in `_print_url`
(`_print_single` / `_print_aggregate` / `_print_all_failed`), based on
success/failure counts and whether `-n` > 1
4. or, if `--json`, build dict entries via `_build_json_entry` and dump
them all at the end.
- Verbose rendering (`_print_verbose_block`) is shared between the single
and aggregate text paths; JSON verbose data is built separately in
`_build_json_entry`.
### `__main__.py`
Trivial shim: `sys.exit(run(sys.argv[1:], sys.stdout, sys.stderr))`, letting
you run `python -m latprobe <url>`.
## Tests (`python/tests/`)
- `test_probe.py` — unit tests against `measure()` directly (against the
`configs/*.txt` failure-mode fixtures: DNS failure, connection refused,
TLS errors, timeouts).
- `test_cli.py` — drives `run()` with injected `io.StringIO` streams,
checking text/JSON output and exit codes.
- `test_integration.py` — end-to-end, against the `configs/*.txt` files
(`all-ok.txt`, `mixed.txt`, `http-errors.txt`, etc.).
## How it connects, end to end
```
__main__.py
→ cli.run()
→ parses flags
→ for each URL: probe.measure() (× count, across a thread pool)
→ collects Result objects
→ if count > 1: aggregate.summarize() → Aggregate
→ cli.py's _print_* / _build_json_entry render text or JSON
→ returns the worst exit code seen
```
## Where to make changes
| Change you want | File |
|-------------------------------------------|------------------|
| Timing/measurement behavior | `probe.py` |
| min/avg/max or new stats | `aggregate.py` |
| Flags, output formatting, exit-code logic | `cli.py` |
| Duration string parsing | `duration.py` |
The dataclasses (`Result`, `Phase`, `Options`, `VerboseDetail`, `Aggregate`,
`PhaseStats`) are the contracts between these files — adding a field to
`Result` in `probe.py` typically means threading it through `aggregate.py`
(if it should be averaged) and `cli.py` (if it should be printed/JSON-encoded).

100
docs/summaries/2026-07-02-00-29-py-hxprobe-httpx.md Normal file
View File

@@ -0,0 +1,100 @@
# Summary: `hxprobe` — httpx-based Python probe
Plan: [docs/plans/2026-07-01-23-47-py-hxprobe-httpx.md](../plans/2026-07-01-23-47-py-hxprobe-httpx.md)
## What was built
A new sibling package, `python/hxprobe/`, alongside the existing raw-socket
`latprobe`. It uses `httpx` so the client matches Go's `http.DefaultClient`:
HTTP/2 negotiated via ALPN, redirects followed by default, connection
pooling, and default TLS verification — while still reporting the full
six-phase breakdown (DNS, TCP connect, TLS, TTFB, Transfer, Total).
Files:
- `python/pyproject.toml` — first third-party dependency in this repo
(`httpx[http2]`)
- `python/hxprobe/probe.py` — the core: `_Trace`, `_TimingStream`,
`_TimingBackend`, `_TimingTransport`, `measure()`
- `python/hxprobe/cli.py`, `__main__.py`, `__init__.py` — thin wrappers
- `python/latprobe/probe.py` (edited) — added `Options.follow_redirects`/
`Options.http2` (ignored by the socket `measure()`) and
`VerboseDetail.http_version`/`redirect_count` (always `""`/`0` there)
- `python/latprobe/cli.py` (edited) — `run()`/`_run_samples()` take an
injectable `measure_fn`, plus `prog`/`description`/`protocol_flags`
overrides, so `hxprobe.cli.run()` reuses the entire argparse/concurrency/
exit-code/rendering pipeline unchanged
- `python/tests/test_hx_probe.py` (17 tests), `test_hx_cli.py` (11 tests),
`test_integration_hx.py` (9 live tests, excluded from the default gate via
the existing `test_i*` naming convention)
- `Makefile` — `py-deps` (creates `python/.venv`, installs `httpx[http2]`),
`hx-run`, `hx-test-integration`; `py-test`/`py-check` now run through the
venv and include the new hermetic hxprobe tests
- `docs/usage/py-hxprobe.md` — usage doc with real captured output
- `.gitignore` — added `*.egg-info/` (editable-install artifact)
## Key design decisions
- **Instrument the transport, don't hand-roll HTTP.** Subclassed
`httpcore.NetworkBackend`/`NetworkStream` to time DNS/TCP connect/TLS at
the socket level, letting httpx own HTTP/1.1 vs HTTP/2 framing, redirects,
and keep-alive. This was the reason to use httpx at all — get the protocol
behavior of a real client while keeping latprobe's phase granularity.
- **First-hop-wins for dns/connect/tls; last-hop-wins for ttfb/transfer.**
When redirects are followed, connection-identity fields (dns/connect/tls
timing, resolved IP, TLS/cert info) reflect the *first* connection.
`wrote_request`/`first_byte` are simply overwritten on every write/read, so
they naturally end up reflecting the *last* hop — which mirrors how Go's
own unguarded `httptrace.ClientTrace` hooks behave for a followed redirect.
- **Fresh `httpx.Client` per `measure()` call, no cross-sample pooling** —
matches latprobe's per-call socket creation so every `-n` sample gets a
full phase breakdown.
- **`measure_fn` injection over subclassing/duplication** in `latprobe.cli`,
so hxprobe reuses argparse, concurrency, exit codes, and text/JSON
rendering with zero duplicated logic — the socket and httpx probes only
differ in `probe.py`.
- **New CLI flags gated behind `protocol_flags=True`** so `latprobe`'s own
`--help` output stays byte-for-byte unchanged (verified) — `--no-http2`/
`--no-follow-redirects` only appear for `hxprobe`.
## Notable finding (not part of the original plan)
While comparing `hxprobe` and `latprobe` timings against the same live host,
`hxprobe`'s TTFB was consistently ~40-50ms *lower*. Verified experimentally
(not just assumed) that this is a real effect, not noise: `latprobe`'s raw
socket never sets `TCP_NODELAY`, so its request write is subject to Nagle's
algorithm interacting with the server's delayed-ACK timer — a well-known
artifact. Forcing `TCP_NODELAY` onto `latprobe`'s socket collapsed its TTFB
to match `hxprobe`'s. `hxprobe` sets `TCP_NODELAY` (matching httpcore's own
default backend and Go's `net.Dialer`), so its TTFB numbers are the more
accurate of the two — not just different. Did not change `latprobe` itself
(out of scope for this task); documented the divergence in
`docs/usage/py-hxprobe.md`, in the CHANGELOG, and inline in
`hxprobe/probe.py`.
## Deviations from the plan
- Plan sketched `TTFB = headers-received minus end-of-TLS`; implemented as
`TTFB = first-byte minus wrote-request` instead (matches both Go and the
existing `latprobe` definition — the plan's phrasing was an approximation).
- Plan said verbose TLS/cert metadata could follow "last hop"; implemented
as first-hop-wins uniformly across dns/connect/tls/ip/tls-info for
simplicity and consistency (only `ttfb`/`transfer` are last-hop).
- Everything else (library choice, transport-instrumentation approach,
pyproject.toml, sibling-package placement, flag surface) matches the
approved plan as written.
## Verification
- `make check` (Go + Python full gate): exit 0, 83 hermetic Python tests
(72 pre-existing + 11 new hermetic CLI + hxprobe's share of the 17 probe
tests already counted), zero regressions to latprobe's original 39.
- `make hx-test-integration`: 9/9 live tests pass, including real HTTP/2
negotiation against example.com (Cloudflare) and a real http→https redirect
follow against github.com.
- Manual spot checks: `--json` schema, `--fail` exit code 6, DNS failure
(exit 2), connection-refused (exit 3), `--no-http2`/`--no-follow-redirects`
flag behavior, `latprobe --help` output diffed byte-for-byte against
pre-change output to confirm no regression.
- Caught and reverted an incidental `gofmt` whitespace diff in
`go/internal/probe/probe.go` that `make check`'s `go-fmt` step produced —
unrelated to this change, out of scope, not committed.

117
docs/summaries/2026-07-02-09-32-hxprobe-standalone-project.md Normal file
View File

@@ -0,0 +1,117 @@
# Summary: extract `hxprobe` into a standalone top-level project
Plan: [docs/plans/2026-07-02-09-32-hxprobe-standalone-project.md](../plans/2026-07-02-09-32-hxprobe-standalone-project.md)
## What was built
`hxprobe` moved from `python/hxprobe/` to a new top-level `hxprobe/`
directory (sibling of `go/` and `python/`), structured as a genuinely
independent project: own `pyproject.toml`, own venv (`hxprobe/.venv`), own
`tests/` directory. It imports nothing from `python/latprobe` — verified with
`grep -rn "latprobe" hxprobe/` returning zero matches in code (a few comments
mention Go's `http.DefaultClient` for context, which is fine; no comment
references `latprobe` as a concrete path/module anymore either).
Files:
- `hxprobe/pyproject.toml` — own manifest, `httpx[http2]` dependency,
`packages = ["hxprobe"]`
- `hxprobe/hxprobe/probe.py` — moved from `python/hxprobe/probe.py`, with
`Options`, `Phase`, `Result`, `VerboseDetail`, `CertInfo` dataclasses and
`_parse_cert`/`_parse_cert_date` helpers now defined locally (copied from
`latprobe/probe.py`) instead of imported; all the httpx-instrumentation
logic (`_Trace`, `_TimingStream`, `_TimingBackend`, `_TimingTransport`,
`measure()`) is unchanged
- `hxprobe/hxprobe/aggregate.py`, `duration.py` — verbatim copies of
`latprobe`'s; their imports were already package-relative (`from .probe
import Result`), so copying required zero edits
- `hxprobe/hxprobe/cli.py` — rewritten as a full standalone CLI. Previously a
27-line wrapper delegating to `latprobe.cli.run()` via an injected
`measure_fn`; now has its own exit codes, argparse, and text/JSON
rendering (copied from the shared `latprobe/cli.py`, then stripped of the
`measure_fn`/`prog`/`description`/`protocol_flags` parameterization that
only existed to let two packages share one `run()`)
- `hxprobe/tests/{test_probe,test_cli,test_integration}.py` — moved from
`python/tests/test_hx_*.py` / `test_integration_hx.py`, `hx`-prefix
dropped, imports repointed from `latprobe.*` to `hxprobe.*`
- `python/latprobe/{cli.py,probe.py}` — reverted via `git checkout --` to
their exact pre-`hxprobe` committed state (confirmed zero diff afterward)
- `Makefile` — `py-deps`/`PY_VENV*`/shared `hx-run`/`hx-test-integration`
removed from the Python section; new standalone `hx-deps`/`hx-run`/
`hx-test`/`hx-test-integration`/`hx-check`/`hx-clean` targets under
`HX_DIR`/`HX_VENV*`; umbrella `test`/`check`/`clean` now run all three
(`go-*`/`py-*`/`hx-*`)
- `docs/usage/py-hxprobe.md` → `docs/usage/hxprobe.md` (renamed + rewritten
for the new structure)
- Old `python/hxprobe/`, `python/pyproject.toml`, `python/.venv`,
`python/latprobe_python.egg-info/` deleted
## Key design decisions
- **Move, don't rewrite, the already-debugged files.** `probe.py` and the
three test files carry real bug fixes found during the original
implementation (the `mark_dns` present-on-failure bug, verbose detail not
populated on the failure path, `Content-Length` needed once the test
fixtures switched to HTTP/1.1 keep-alive). Rewriting from scratch would
have risked reintroducing them.
- **`aggregate.py`/`duration.py` needed zero import changes** — their
existing relative imports (`from .probe import ...`) already resolve
correctly once copied into a new package with its own `probe.py`. Not
every file needed the same treatment as `probe.py`/`cli.py`.
- **`cli.py`'s starting point was the *current* shared `latprobe/cli.py`**,
not a from-scratch rewrite — it already contained 100% of the needed
logic (including the `--no-http2`/`--no-follow-redirects` flags and the
verbose Protocol row/JSON keys, both added earlier specifically for
hxprobe). The only work was deleting the generalization scaffolding
(`measure_fn`, `MeasureFn`, `prog`/`description`/`protocol_flags` params)
that existed solely to let two packages share one `run()`.
- **Reverting `latprobe` via `git checkout --` rather than hand-editing** —
confirmed first via `git log`/`git diff --stat` that `cli.py`/`probe.py`
had no changes besides the hxprobe-sharing scaffolding since the last
commit, making this a safe, exact, zero-risk revert.
- **Comments referencing `latprobe` by file path were reworded**, not just
the imports. A comment like "already used by latprobe/probe.py" becomes a
dangling reference once this directory is genuinely portable to another
repo. Reworded ~5 comments/docstrings to describe the technique generically
(e.g., "ports the getaddrinfo → connect split" → "splits DNS and TCP
connect into two timed steps") instead of naming the other project.
## Deviations from the plan
None of substance. One judgment call not spelled out in the plan: the old
`test_cli.py`'s docstring said its tests "focus on what's different... since
run() delegates almost entirely to latprobe.cli.run()" — no longer true now
that `cli.py` is a full standalone reimplementation. Rewrote that docstring
for accuracy rather than leaving a stale claim, without expanding the test
suite itself (that would be a larger, separate scope-creep beyond what was
asked — see the coverage note below).
## Notable follow-up worth flagging
`hxprobe/tests/test_cli.py` has ~11 tests versus `latprobe/tests/test_cli.py`'s
~23. That gap was fine when hxprobe's `cli.py` was a thin wrapper around
already-tested shared code; it's a real gap now that `cli.py` is an
independent ~440-line reimplementation with its own copy of every rendering
branch. Existing tests do exercise the core paths (single URL, `--fail`,
JSON, DNS/connect failures) so this isn't uncovered, but reaching
`latprobe`-level depth (multi-URL separator, sampling aggregate output, JSON
error grouping, all-failed multi-header) would be a reasonable next step if
full independent confidence in the standalone project matters. Not done here
— out of scope for a decoupling/move task.
## Verification
- `grep -rn "latprobe" hxprobe/` — zero matches in code; comment mentions
reworded to be self-contained
- `git diff --stat python/latprobe/` — empty (exact revert to last commit)
- `make hx-deps` — creates `hxprobe/.venv`, installs `httpx[http2]` from
`hxprobe/pyproject.toml`
- `make hx-run ARGS="-v http://github.com"` — same output as before the
move (HTTP/2, 1 redirect followed, IP/TLS/cert shown)
- `make hx-test` — all hermetic hxprobe tests pass standalone (own venv,
own `PYTHONPATH=hxprobe`)
- `make hx-test-integration` — live HTTP/2 + redirect tests still pass
- `make py-test` — `latprobe`'s own suite passes with zero setup (no
`py-deps`/venv needed), confirming the revert didn't leave residue
- `make check` (top-level) — Go + latprobe + hxprobe all green in one gate
- `python -m latprobe --help` output confirmed unchanged from before the
whole hxprobe feature existed (no leftover `--no-http2` etc.)

73
docs/summaries/2026-07-02-09-57-hxprobe-toolchain-modernization.md Normal file
View File

@@ -0,0 +1,73 @@
# Summary: hxprobe toolchain modernization (uv, ruff, pytest)
Plan: [docs/plans/2026-07-02-09-57-hxprobe-toolchain-modernization.md](../plans/2026-07-02-09-57-hxprobe-toolchain-modernization.md)
## What was built
Modernized `hxprobe`'s Python toolchain per the user's confirmed choices
(uv, ruff, pytest; mypy skipped). No behavior change to the probe/CLI
itself — this is purely tooling.
- `hxprobe/pyproject.toml`: added `[dependency-groups] dev = ["pytest>=8.0",
"ruff>=0.8"]` (PEP 735), `[tool.pytest.ini_options]` (testpaths, a
registered `integration` marker), `[tool.ruff]` (`target-version =
"py311"`, `line-length = 100`)
- `hxprobe/.python-version`: new, pins `3.14`
- `hxprobe/uv.lock`: new, 18 packages resolved and pinned (`httpx`,
`httpcore`, `h2`, `certifi`, `pytest`, `ruff`, and their transitive deps)
- `hxprobe/tests/test_integration.py`: added `pytestmark =
pytest.mark.integration`, replacing the `test_[!i]*.py` filename-glob
convention with a real pytest marker
- Ran `ruff check --fix` (one fix: removed unused `import sys` in `cli.py`,
inherited from the original `latprobe/cli.py`) and `ruff format .`
(6 files reformatted — collapsed the hand-aligned `=`/dict-key columns to
single-space; no semantic changes)
- `hxprobe/README.md`: new, standalone quick-start
- `Makefile`: `hx-deps`/`hx-run`/`hx-test`/`hx-test-integration` now shell
out to `uv sync`/`uv run` instead of manual venv+pip; added `hx-lint`/
`hx-fmt`; `hx-check` now bundles lint + hermetic tests (mirrors
`go-check`'s fmt+vet+test pattern)
- `docs/usage/hxprobe.md`: Setup and Makefile-targets sections updated
- `.gitignore`: added `.pytest_cache/`, `.ruff_cache/`
## Key design decisions
- **`[dependency-groups]` (PEP 735) over `[tool.uv.dev-dependencies]`** —
the standardized, current uv-recommended way to declare dev-only deps,
keeps them out of the installable package's `dependencies` list.
- **Runner swap only, no test rewrite.** `unittest.TestCase` classes,
`unittest.skipUnless` decorators, and `if __name__ == "__main__":
unittest.main()` guards are untouched — pytest is a superset runner for
unittest-style tests. Only the marker-based selection mechanism changed.
- **`hx-check` now includes lint**, not just tests — matches how
`go-check: go-fmt go-vet go-test` already bundles static checks with
tests in this repo, rather than treating lint as a separate, easy-to-skip
step.
- **Verified the `ruff format` diff was purely cosmetic** before accepting
it: reviewed the actual diff (whitespace/alignment only, no reordering or
logic changes), then confirmed with `ast.parse` on every file post-format
and a full pytest run afterward (37 tests: 28 hermetic + 9 integration,
all passing) — did not run the suite pre-format, so this confirms the
post-format state is correct rather than a strict before/after diff.
## Deviations from the plan
None. Implemented exactly as planned; the `import sys` removal and the
~6-file reformatting were both explicitly anticipated in the plan text
("Lint fixes" section) rather than being surprises.
## Verification
- `cd hxprobe && uv sync` — creates `.venv`, installs from `uv.lock`
(`httpx[http2]`, `pytest`, `ruff` + transitive deps)
- `make hx-run ARGS="-v http://github.com"` — unchanged output (HTTP/2,
1 redirect, IP/TLS/cert)
- `make hx-lint` — `ruff check .` clean (`All checks passed!`)
- `make hx-test` — `pytest tests -m "not integration"`: 28 passed, 9
deselected (matches the pre-toolchain-change hermetic test count exactly)
- `make hx-test-integration` — `pytest tests -m integration`: 9 passed, 28
deselected
- `grep -rn "latprobe" hxprobe/` — zero matches (toolchain change didn't
reintroduce coupling)
- `git diff --stat python/` — empty (hxprobe-only change, `latprobe`
untouched)

74
docs/summaries/2026-07-02-10-22-hxprobe-usage-doc-and-makefile.md Normal file
View File

@@ -0,0 +1,74 @@
# Summary: hxprobe usage reference doc + standalone Makefile
Plan: [docs/plans/2026-07-02-10-22-hxprobe-usage-doc-and-makefile.md](../plans/2026-07-02-10-22-hxprobe-usage-doc-and-makefile.md)
## What was built
- **`hxprobe/USAGE.md`** (new): a "Runnable Usage Reference" matching the
depth/format of `python/configs/usage-latprobe.md` (concrete `sh` command
→ real captured output, brief explanatory notes, `---` section
separators). 16 cases: basic, verbose (HTTPS, plain HTTP, redirect
followed, `--no-follow-redirects`, `--no-http2`, TLS failure, DNS
failure), sampling, multi-URL, `--fail`, JSON, JSON+verbose, timeout,
exit-codes table, Makefile shortcuts (both `hxprobe/Makefile`'s own and
the parent repo's `hx-*` ones). Placed inside `hxprobe/` per the user's
explicit choice, so the doc travels with the project if it's ever
extracted to its own repo.
- **`docs/usage/hxprobe.md`** (edit): one-line pointer added at the top to
`hxprobe/USAGE.md`. No other changes.
- **`hxprobe/Makefile`** (new): standalone, `help`/`deps`/`run`/`lint`/
`fmt`/`test`/`test-integration`/`check`/`clean`, same auto-generated
`## comment` help style as the root Makefile. Deliberately independent
from the root Makefile's `hx-*` targets (neither calls into the other) —
explicit user decision over the "delegate" alternative.
## Key design decisions
- **Every output in `USAGE.md` is real, freshly captured this session** —
10 of 16 cases were run live during implementation specifically for this
doc (TLS failure, sampling ×2, multi-URL, `--fail`, JSON ×2, timeout,
basic, plain-HTTP), the rest reused real captures from earlier in the
same session (redirect-follow, `--no-follow-redirects`, `--no-http2`, DNS
failure) since the code hadn't changed since those were taken. No numbers
were fabricated or extrapolated.
- **Timeout example uses `192.0.2.1`, not `10.255.255.1`.** Tested both:
`10.255.255.1` resolves to an immediate `connect: Connection refused` in
this dev sandbox (the sandbox's network layer actively rejects the
packet rather than dropping it silently), which would misrepresent the
timeout path. `192.0.2.1` (RFC 5737 TEST-NET-1, reserved/unreachable)
reliably produces a genuine ~500ms timeout here, so that's what the doc
uses and explains.
- **No delegation between the two Makefiles**, per explicit user
instruction — both have complete, independent implementations of the
same `uv sync`/`uv run pytest`/`ruff` commands. This is deliberate
duplication: a change to one Makefile's command flags won't silently
break the other, at the cost of needing to update both if the underlying
`uv run ...` invocations ever change.
- **`hxprobe/Makefile` target names have no `hx-` prefix** (`run`, `test`,
not `hx-run`, `hx-test`) since the prefix's whole purpose — disambiguating
from `go-*`/`py-*` targets in the same file — doesn't apply once you're
already inside `hxprobe/`'s own Makefile.
## Deviations from the plan
One, driven by what the live network in this sandbox actually does: the
plan didn't anticipate the timeout target needing to change from
`10.255.255.1` (used in `latprobe`'s own timeout example) to `192.0.2.1`.
Discovered and resolved by testing both live before writing the doc, rather
than assuming `latprobe`'s example target would work identically for
hxprobe.
## Verification
- `cd hxprobe && make help` — lists all 9 targets, no dependency on the
root Makefile
- `cd hxprobe && make run ARGS="-v https://example.com"` — matches
`make hx-run ARGS="-v https://example.com"` output shape from the repo
root (both Makefiles agree independently)
- `cd hxprobe && make check` — 28 hermetic tests pass (lint + test)
- `cd hxprobe && make test-integration` — 9 live tests pass
- Every command block in `USAGE.md` was actually executed during
authoring; output pasted directly from the terminal, not hand-edited
beyond JSON float-precision rounding (documented as such in the doc)
- `git diff Makefile` — confirms the root Makefile's `hx-*` section is
unchanged by this task

77
docs/summaries/2026-07-02-11-14-hxprobe-file-input.md Normal file
View File

@@ -0,0 +1,77 @@
# Summary: hxprobe reads target URLs from a file
Plan: [docs/plans/2026-07-02-11-14-hxprobe-file-input.md](../plans/2026-07-02-11-14-hxprobe-file-input.md)
## What was built
- **`hxprobe/hxprobe/cli.py`**: new `-f`/`--file PATH` flag, mutually
exclusive with positional `url` args. `urls` positional changed
`nargs="+"` → `nargs="*"`. New `_load_urls()` helper (same format as
`simple.py`'s `load_sites()`: one URL per line, `#` comments, blank
lines skipped, first token per line) — reimplemented locally rather than
imported, keeping hxprobe's "no imports outside its own directory" rule
intact. Validation added right after `parser.parse_args()`, inside the
same `try/except _ArgExit` block: both-given and neither-given are usage
errors via `parser.error()`; missing/unreadable/empty file are usage
errors via direct `stderr.write()` + `return EXIT_USAGE` (mirroring the
existing `--timeout` invalid-value handling style already in the file).
- **`hxprobe/tests/test_cli.py`**: new `TestCLIFileInput` class, 5 tests —
reads URLs from a temp file successfully, missing file, empty file,
both-sources error, neither-given error.
- **`hxprobe/configs/*.txt`**: 7 new fixtures mirroring
`python/configs/`'s exact set. Each header comment states an "Expected
exit code" that was verified by actually running the fixture through
`hxprobe -f ...` during implementation (not assumed from the
`simple.py` originals, whose blanket 0/1 exit scheme is fundamentally
different from hxprobe's per-failure-class 0–6 worst-code-wins scheme).
- **`hxprobe/USAGE.md`**: new "Reading URLs from a file (`-f`)" section,
placed after "Multiple URLs", with real captured output for the
successful case, the mutually-exclusive error case, and one failure
fixture (`dns-failure.txt`), plus a table listing all 7 fixtures and
their expected exit codes.
## Key design decisions
- **Manual post-parse validation instead of `argparse`'s
`add_mutually_exclusive_group`** — a variadic positional (`nargs="*"`)
doesn't mix cleanly with argparse's built-in mutually-exclusive-group
machinery. Manual checks after `parser.parse_args()` (still inside the
same `try/except _ArgExit`, using `parser.error()`) give the same
usage-error behavior with full control over the message text.
- **File I/O errors don't go through `parser.error()`** — they use the
same direct `stderr.write()` + `return EXIT_USAGE` pattern already
established for `--timeout` parsing failures, since they're discovered
after parsing succeeds, not during it.
- **Every fixture's exit code was verified live, not assumed.** Two
required real judgment calls the `simple.py` originals didn't need:
`http-errors.txt` and `mixed.txt` both needed `--fail` added to their
demo command (hxprobe treats 4xx as success without it, unlike
`simple.py` which always raises on `HTTPError`) to actually demonstrate
a failure — without `--fail` both would silently show exit 0.
## Deviations from the plan
None of substance. The plan anticipated needing to verify exit codes
live rather than assume them; that anticipation paid off exactly as
expected for `http-errors.txt`/`mixed.txt` (needed `--fail` added) and
`timeout.txt` (needed `--timeout 2s` added to keep the demo fast, and a
note added about this sandbox occasionally short-circuiting one of the two
timeout targets to an immediate "connection refused" — observed directly:
in this session's test run, `10.255.255.1` genuinely timed out; in an
earlier, unrelated test earlier in the session it instead got refused
immediately. The fixture keeps both targets so at least one demonstrates
the real timeout path regardless).
## Verification
- `all-ok.txt` → exit 0, `dns-failure.txt` → 2, `connection-refused.txt`
→ 3, `tls-errors.txt` → 5, `timeout.txt` (with `--timeout 2s`) → 4,
`http-errors.txt` (with `--fail`) → 6, `mixed.txt` (with `--fail`) → 6 —
all confirmed via real `$?` checks (not through a `grep` pipe, which
masks the real exit code — caught and corrected this during testing)
- `uv run pytest tests/test_cli.py -m "not integration"` — 16/16 pass
(11 pre-existing + 5 new)
- `uv run ruff check .` / `uv run ruff format --check .` — clean
- Both-sources-given and missing-file error messages verified against the
doc's pasted output, including the `usage:` block that's actually
printed (an early draft of the doc omitted it — caught on review)

83
docs/summaries/2026-07-02-12-05-hxprobe-simplification.md Normal file
View File

@@ -0,0 +1,83 @@
# hxprobe simplification — summary
Plan: `docs/plans/2026-07-02-12-05-hxprobe-simplification.md` (five targeted
removals of speculative scaffolding / redundant state / duplicated logic, no
behavior change). Trigger: a question about a dead `file=` parameter on
`_Parser.print_help`/`print_usage` in `hxprobe/hxprobe/cli.py`, which led to a
broader pass over the whole package for the same pattern.
## What changed
1. **`hxprobe/hxprobe/cli.py` — deleted the `_Parser` subclass and `_ArgExit`
exception** (~32 lines). It re-implemented, by hand, behavior the standard
library already provides: `argparse.ArgumentParser` already writes help to
`sys.stdout` and usage/errors to `sys.stderr`, and already raises
`SystemExit` rather than hard-exiting. `run()` now builds a plain
`argparse.ArgumentParser`, wraps the parse + validation calls in
`contextlib.redirect_stdout(stdout)` / `redirect_stderr(stderr)`, and
catches `SystemExit` at a single site:
```python
except SystemExit as exc:
return EXIT_OK if not exc.code else EXIT_USAGE
```
Deviation from the original plan sketch: the plan's snippet used
`EXIT_OK if not exc.code else EXIT_USAGE`, matching what was actually
implemented (equivalent to, but slightly more defensive than, the
`exc.code == 0` check first drafted, since argparse can in principle pass
`None`).
2. **`hxprobe/hxprobe/probe.py` — trimmed `_TimingStream.get_extra_info`** to
the single branch actually consumed on the request path (`ssl_object`).
Verified by grepping the installed `httpcore`/`httpx` packages: the
`server_addr`/`client_addr` branches were only ever queried by `httpx`'s own
`_main.py` (the `httpx` CLI command), never by anything hxprobe's request
path touches.
3. **`hxprobe/hxprobe/cli.py` — simplified `_load_urls`**: dropped the
`line.split()[0]` "forward-compatible with future `url key=value`
annotations" scaffolding; a stripped line is used directly. Docstring
updated to describe only what the function does today.
4. **`hxprobe/hxprobe/probe.py` — removed `_dns_set`/`_connect_set`/
`_tls_set`** from `_Trace`. These booleans duplicated `Phase.present` on
`self.dns`/`self.connect`/`self.tls` (each starts as `Phase()`, i.e.
`present=False`). Guards rewritten from `if not self._dns_set:` to
`if not self.dns.present:` (and analogously for connect/tls) — identical
first-hop-wins semantics for redirects, one less piece of parallel state.
5. **`hxprobe/hxprobe/cli.py` — extracted `_summarize_failures`**, a shared
helper that dedupes `failed: list[Result]` into `[(phase, message, count),
...]` in first-seen order. `_print_failure_summary` (text rendering) and
`_build_json_entry` (JSON rendering) previously each carried an identical
~12-line dedup loop; both now call the helper and only differ in how they
format the tuple.
## Not changed (considered, kept — per the plan)
- The custom httpcore backend (`_TimingBackend`/`_TimingStream`/
`_TimingTransport`) — this is the tool's actual reason to exist.
- `ThreadPoolExecutor` concurrency — backs the shipped `-c/--concurrency` and
multi-URL/`-f` features.
- Explicit per-phase dataclass fields in `probe.py`/`aggregate.py`.
- `CertInfo.sans` — unused in text output but real (emitted in JSON verbose
output).
## Verification
- `hxprobe/.venv/bin/python -m pytest tests/test_cli.py tests/test_probe.py -q`
→ **33 passed**, no test file edits.
- `hxprobe/.venv/bin/ruff check hxprobe/` → **all checks passed**.
- Manual smoke tests (`python -m hxprobe -h`, no-args, and a live
`https://example.com -v` request):
- `-h` → help text on stdout, exit `0`.
- No args → usage + `no URLs given` error on **stderr only** (confirmed via
separate stdout/stderr redirection to files — stdout was empty), exit `1`.
- Live verbose request → correct 6-phase timing, resolved IP, HTTP/2
protocol, TLS version/cipher, and certificate detail — confirms the
`get_extra_info` trim didn't break `ssl_object` retrieval and the
`_Trace` refactor didn't break phase/first-hop-wins tracking.
Net: `cli.py` and `probe.py` are shorter and carry less parallel/duplicated
state; no observable behavior changed.

105
docs/summaries/2026-07-02-14-05-hxprobe-run-summary-footer.md Normal file
View File

@@ -0,0 +1,105 @@
# hxprobe: end-of-run summary footer — summary
Plan: `docs/plans/2026-07-02-14-05-hxprobe-run-summary-footer.md`.
Trigger: a follow-up to
`docs/explanations/2026-07-02-13-25-hxprobe-worst-exit-code-and-render-loop.md`
— does a single worst-code exit even make sense across multiple URLs with
different possible errors? The answer landed on: keep the scalar exit code
(it's a documented cross-implementation contract with `latprobe`/Go — see
`hxprobe/USAGE.md`'s Exit codes table, asserted by 13 existing tests), and
instead add the missing visibility as an end-of-run summary footer.
## What changed
All in `hxprobe/hxprobe/cli.py` unless noted, matching the plan exactly (no
deviations):
1. **`_EXIT_LABELS`** (new, next to `_PHASE_EXIT`): inverse mapping from exit
code → short label (`ok`, `dns`, `connect`, `timeout`, `tls`, `http`), used
to render the footer's `→ exit N (label)` line.
2. **Accumulation loop refactor**: previously computed a single running
`worst` directly inside the per-URL loop via two different idioms
(`if c > worst: worst = c` for network failures, `max(worst, EXIT_HTTP)`
for `--fail`). Now each URL first gets its own `code` (`EXIT_OK` folded up
via `max()` across its failed samples and, if `--fail`, its ≥400 successes),
appended to a new `url_codes: list[int]` (index-aligned with `urls`), and
*then* folded into `worst = max(worst, code)`. Unifies both idioms into one.
3. **`_print_run_summary(urls, url_codes, worst, out)`** (new helper, next to
`_print_failure_summary`): writes a separator line, an
`N URLs — X ok[, Y failed]` header, one `✗ {label:<8}: {count}` line per
non-OK class present (first-seen order, same dedup style as
`_summarize_failures`), and the closing `→ exit N (label)` line.
4. **Call site**: `elif len(urls) > 1: _print_run_summary(urls, url_codes,
worst, stdout)` — added after the per-URL loop, in the `else` branch of the
existing `if ns.json_out:` check (so it's text-mode-only), right before
`return worst`. JSON path (`json.dumps(json_items, ...)`) is completely
untouched — still a bare array, no top-level summary object, preserving the
documented "same shape as `latprobe`'s JSON" contract.
## Design decisions (confirmed with user before implementing)
- **Exit code stays a scalar** (worst/highest severity across URLs) — not
count-of-failed-URLs, not binary 0/1. Both alternatives were presented and
rejected because they'd break the documented 0–6 table, Go/`latprobe`
parity, and the 13 existing exit-code tests.
- **Summary is a text footer, multi-URL only** (`len(urls) > 1`) — not always
shown, and not also duplicated into JSON as a top-level object (which would
turn the JSON array into an object and break the documented array-shape
parity). Single-URL text output is untouched; JSON output is untouched.
## Tests
`hxprobe/tests/test_cli.py`: new `TestCLIRunSummary` class, 3 tests, reusing
the existing `_OKHandler`/`_start_server`/`_free_port`/`_invoke` harness:
- `test_multi_url_mixed_shows_summary` — one OK URL + one connection-refused
URL: asserts `EXIT_CONNECT`, and the footer strings (`"Summary: 2 URLs"`,
`"1 ok"`, `"1 failed"`, `"connect : 1"`, `"→ exit 3"`).
- `test_multi_url_all_ok_summary` — two OK URLs: asserts `EXIT_OK`,
`"Summary: 2 URLs — 2 ok"`, and no `"✗"` anywhere in output.
- `test_single_url_has_no_summary` — one OK URL: asserts `"Summary:"` is
absent (locks the multi-URL-only rule).
All 33 pre-existing tests pass unedited (33 + 3 new = 36 total).
## Verification
- `hxprobe/.venv/bin/python -m pytest tests/test_cli.py tests/test_probe.py -q`
→ **36 passed**.
- `hxprobe/.venv/bin/ruff check hxprobe/ tests/` → **all checks passed**.
- Manual smoke tests, all matching the plan's expected behavior exactly:
- `hxprobe https://example.com https://example.org` → footer
`Summary: 2 URLs — 2 ok` / `→ exit 0 (ok)`.
- `hxprobe https://example.com http://no.such.host.invalid` → footer
`Summary: 2 URLs — 1 ok, 1 failed` / `✗ dns : 1` / `→ exit 2 (dns)`;
process exit code confirmed `2` via `echo $?`.
- `hxprobe https://example.com` (single URL) → **no** footer, output
byte-for-byte the same shape as before this change.
- `hxprobe --json https://example.com https://example.org` → still a bare
JSON array, no summary object.
- Also captured a 3-URL mixed run (`example.com` ok, DNS failure, TLS
failure against `self-signed.badssl.com`) to confirm severity ordering in
the footer: DNS (2) and TLS (5) both counted, exit reported as `5 (tls)`
— the higher-severity class correctly wins the scalar while the footer
still shows the DNS failure that the scalar alone would hide.
## Docs updated (per CLAUDE.md conventions)
- `hxprobe/USAGE.md`: new "Multi-URL summary footer" section with a real
3-URL mixed-outcome capture; refreshed the pre-existing "Multiple URLs" and
`-f configs/all-ok.txt` / `-f configs/dns-failure.txt` examples, which were
captured before this feature existed and were now stale (missing the
footer) — replaced with fresh live captures; added a sentence to the Exit
codes section pointing at the new section.
- `docs/usage/hxprobe.md`: one-line addition to the exit-codes bullet
pointing at `hxprobe/USAGE.md`'s new section.
- `docs/explanations/2026-07-02-13-25-hxprobe-worst-exit-code-and-render-loop.md`:
appended an "Update (2026-07-02)" paragraph pointing forward to this work,
per the plan's "optional — pointer instead of a new file" option.
- `CHANGELOG.md`: new entry at the top.
- This file.
No deviations from the approved plan.

232
docs/usage/hxprobe.md Normal file
View File

@@ -0,0 +1,232 @@
# `hxprobe/` — httpx-based HTTP latency probe (Go-client parity)
> For a full case-by-case runnable reference (every flag, every failure
> mode, real captured output), see [`hxprobe/USAGE.md`](../../hxprobe/USAGE.md).
## What it does
A standalone Python CLI, independent of anything else in this repo (own
`pyproject.toml`, own venv — see [Setup](#setup)). It measures per-phase HTTP
latency like [`python/latprobe`](py-latprobe.md), but is built on the
[httpx](https://www.python-httpx.org/) library instead of raw sockets, so the
client matches Go's `http.DefaultClient`: **HTTP/2 negotiated via ALPN**,
**redirects followed by default**, connection pooling, and default TLS
verification — while still reporting the same six-phase breakdown (DNS, TCP
connect, TLS, TTFB, Transfer, Total).
Python has no equivalent of Go's `net/http/httptrace`, so the phase timing is
recovered by instrumenting httpx's network backend directly
(`hxprobe/probe.py`): a custom `NetworkBackend`/`NetworkStream` pair times
DNS, TCP connect, and TLS at the socket level, while HTTP framing (HTTP/1.1
or HTTP/2), redirect-following, and keep-alive stay entirely owned by httpx.
Run it from the `hxprobe/` directory with `uv run python -m hxprobe`, or via
`make hx-run ARGS="…"` from the repo root.
## Setup
`hxprobe` is a fully self-contained project — it could be copied out of this
repo into its own tomorrow and still work, with its own `pyproject.toml`,
lockfile (`uv.lock`), and [uv](https://docs.astral.sh/uv/)-managed venv
(`hxprobe/.venv`), separate from anything under `python/`. Install once:
```sh
make hx-deps
```
This runs `uv sync` inside `hxprobe/`, creating `.venv` and installing
`httpx` (plus `h2` for HTTP/2), `pytest`, and `ruff` from `uv.lock` — pinned,
reproducible versions, not whatever the resolver happens to pick at install
time. It's a prerequisite of `make hx-run`/`make hx-test`/etc., so those
targets set it up automatically on first run — `make hx-deps` is only needed
if you want to call `uv run python -m hxprobe` directly from inside
`hxprobe/`. See [hxprobe/README.md](../../hxprobe/README.md) for the
from-inside-the-directory quick start.
## Flags / arguments
Same surface as `latprobe`, plus two opt-outs for the Go-like defaults:
```
python -m hxprobe [flags] <url> [url ...]
```
| Flag | Default | Description |
|------|---------|-------------|
| `url …` (positional) | required | One or more URLs to probe |
| `-n N`, `--count N` | `1` | Number of requests per URL |
| `-c N`, `--concurrency N` | `0` (auto) | Max parallel URLs; `0` = `min(len(urls), 8)` |
| `--timeout DURATION` | `10s` | Per-request timeout; supports `ms`, `s`, `m`, or bare seconds |
| `--fail` | off | Exit non-zero when any HTTP status ≥ 400 |
| `--json` | off | Output as JSON array instead of text |
| `-v`, `--verbose` | off | Show resolved IP, negotiated protocol/redirects, TLS info, certificate, response headers |
| `--no-http2` | off (HTTP/2 on) | Disable HTTP/2 negotiation, force HTTP/1.1 |
| `--no-follow-redirects` | off (follow on) | Report the raw redirect response instead of following it |
| `-h`, `--help` | — | Show help and exit 0 |
**Exit codes:** identical to `latprobe` (0 ok, 1 usage, 2 dns, 3 connect,
4 timeout, 5 tls, 6 http≥400 with `--fail`); the highest code across all URLs
is returned. When more than one URL is probed, a summary footer is appended
after the last URL block tallying every URL's outcome (ok / dns / connect /
timeout / tls / http) and the resulting exit code — see "Multi-URL summary
footer" in [`hxprobe/USAGE.md`](../../hxprobe/USAGE.md) for a real example.
## Examples
### Single URL — negotiates HTTP/2 by default
```sh
make hx-run ARGS="https://example.com"
```
```
https://example.com (200)
DNS lookup : 2.81 ms
TCP connect : 8.28 ms
TLS handshake : 16.13 ms
Server (TTFB) : 1.04 ms
Transfer : 0.62 ms
─────────────────────────────
Total : 42.53 ms
```
### Verbose — shows the negotiated protocol
```sh
make hx-run ARGS="-v https://example.com"
```
```
https://example.com (200)
DNS lookup : 2.93 ms
TCP connect : 8.58 ms
TLS handshake : 17.71 ms
Server (TTFB) : 4.35 ms
Transfer : 0.84 ms
─────────────────────────────
Total : 44.06 ms
IP : 104.20.23.154
Protocol : HTTP/2
TLS : TLSv1.3 TLS_AES_256_GCM_SHA384 256 bit
Cert : CN=example.com valid until 2026-08-29 SSL Corporation
```
### Redirects followed by default
```sh
make hx-run ARGS="-v http://github.com"
```
```
http://github.com (200)
DNS lookup : 14.01 ms
TCP connect : 19.42 ms
TLS handshake : 22.72 ms
Server (TTFB) : 0.01 ms
Transfer : 65.66 ms
─────────────────────────────
Total : 196.72 ms
IP : 140.82.121.4
Protocol : HTTP/2 (1 redirect)
TLS : TLSv1.3 TLS_AES_128_GCM_SHA256 128 bit
Cert : CN=github.com valid until 2026-08-02 Sectigo Limited
```
Pass `--no-follow-redirects` to get the raw redirect response instead:
```sh
make hx-run ARGS="--no-follow-redirects http://github.com"
```
```
http://github.com (301)
DNS lookup : 2.85 ms
TCP connect : 24.16 ms
Server (TTFB) : 24.50 ms
Transfer : 0.59 ms
─────────────────────────────
Total : 52.58 ms
```
### Forcing HTTP/1.1
```sh
make hx-run ARGS="-v --no-http2 https://example.com"
```
```
https://example.com (200)
DNS lookup : 2.31 ms
TCP connect : 8.75 ms
TLS handshake : 13.61 ms
Server (TTFB) : 13.51 ms
Transfer : 0.44 ms
─────────────────────────────
Total : 39.07 ms
IP : 104.20.23.154
Protocol : HTTP/1.1
TLS : TLSv1.3 TLS_AES_256_GCM_SHA384 256 bit
Cert : CN=example.com valid until 2026-08-29 SSL Corporation
```
### Sampling, JSON, `--fail`, DNS failure
Same shape as `latprobe`'s own examples — see
[py-latprobe.md](py-latprobe.md#examples) for `-n`, `--json`, multi-URL,
`--fail`, and timeout output. hxprobe's JSON verbose object adds two keys:
```json
"verbose": {
"ip": "104.20.23.154",
"http_version": "HTTP/2",
"redirect_count": 1,
"tls_version": "TLSv1.3",
...
}
```
`http_version`/`redirect_count` are omitted from the text view when
`http_version` is unknown (only happens on failures before headers arrived).
## HTTP/2, redirects, and a TTFB accuracy note
**HTTP/2 and redirects** are the headline reasons this project exists
alongside the raw-socket `latprobe` — see [py-latprobe.md](py-latprobe.md) for
what that implementation does instead (always HTTP/1.1, never follows
redirects).
**A TTFB finding worth knowing, if you've compared numbers against another
raw-socket HTTP client:** during development, this implementation's TTFB
came in consistently ~40-50ms *lower* than a comparable raw-socket
implementation that didn't set `TCP_NODELAY` on its connect socket. That gap
wasn't noise — a socket that never sets `TCP_NODELAY` is subject to Nagle's
algorithm interacting with the server's delayed-ACK timer, a well-known
~40ms artifact. This implementation sets `TCP_NODELAY` on every connection
(matching both httpcore's own default backend and Go's `net.Dialer`),
avoiding that penalty. Forcing `TCP_NODELAY` onto the other socket
experimentally collapsed its TTFB to match this one's — confirming the
cause. If you're comparing hxprobe's numbers against some other HTTP/1.1
client that doesn't set `TCP_NODELAY`, expect this implementation's TTFB to
read lower, and correctly so.
**Redirect semantics for `dns`/`connect`/`tls` vs `ttfb`/`transfer`:** when
redirects are followed, `dns`/`connect`/`tls` (and the verbose IP/TLS/cert
fields) reflect the **first** connection only — "cost of reaching the origin
server." `ttfb`/`transfer` reflect the **last** hop, because each write/read
call overwrites them — which mirrors how Go's own `httptrace.ClientTrace`
hooks behave for a followed redirect (they aren't guarded either, so the last
hop wins there too).
## Limitations
- Always `GET`, no custom headers/body/auth — matches Go's `http.DefaultClient`.
- `Options.timeout` applies uniformly to connect/read/write/pool phases (a
single value); it is not split into separate per-phase budgets.
- No UNIX socket support.
- HTTP/2 requires TLS (`https://`) in practice — cleartext `h2c` is not
attempted for `http://` URLs (matches nearly every real HTTP/2 deployment).
## Makefile targets
```sh
make hx-deps # one-time: uv sync (venv + lockfile install)
make hx-run ARGS="-v -n 3 https://example.com" # run it
make hx-test # hermetic unit tests (pytest -m "not integration")
make hx-test-integration # live tests: real HTTP/2 negotiation, redirects
make hx-lint # ruff check
make hx-fmt # ruff format
make hx-check # lint + hermetic tests (the pre-commit-style gate)
```
`hx-check` (lint + hermetic tests) also runs as part of the repo-root
`make check`; `hx-test` runs as part of `make test`.