docs: hxprobe plans, summaries, explanations, usage, and changelog
Plans (docs/plans/): - 2026-07-01-23-47-py-hxprobe-httpx.md — initial httpx probe design - 2026-07-02-09-32 through 14-05 — standalone project, toolchain, usage doc + Makefile, file input (-f), simplification pass, run-summary footer Summaries (docs/summaries/): one per completed feature, recording what was actually built, deviations from the plan, and verification steps Explanations (docs/explanations/): two deep-dives written during review — hxprobe concurrency model and worst-exit-code + render-loop analysis Usage (docs/usage/hxprobe.md): overview with pointer to hxprobe/USAGE.md for the full runnable reference Walkthrough (docs/py-latprobe-walkthrough.md): narrative tour of the latprobe Python package for interview / code-review context CHANGELOG.md: entries for all hxprobe features (toolchain, usage doc, file input, simplification, run-summary footer) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
130
docs/plans/2026-07-01-23-47-py-hxprobe-httpx.md
Normal file
130
docs/plans/2026-07-01-23-47-py-hxprobe-httpx.md
Normal file
@@ -0,0 +1,130 @@
|
||||
# Plan: `hxprobe` — httpx-based Python probe (Go-client parity)
|
||||
|
||||
## Context
|
||||
|
||||
The existing Python `latprobe` package measures per-phase HTTP latency with raw
|
||||
sockets. That gives an excellent DNS/TCP/TLS/TTFB/Transfer breakdown, but the
|
||||
cost is that it always speaks **HTTP/1.1** and **does not follow redirects** —
|
||||
diverging from the Go implementation, whose `http.DefaultClient` negotiates
|
||||
**HTTP/2** via ALPN, **follows redirects** (up to 10), pools connections, and
|
||||
verifies TLS by default (confirmed: `go/internal/probe/probe.go` uses
|
||||
`http.DefaultClient.Do` with no custom transport/`CheckRedirect`).
|
||||
|
||||
Goal: add a **second Python implementation, `hxprobe`**, built on the `httpx`
|
||||
library so it matches the Go client's protocol capabilities (HTTP/2, redirects,
|
||||
pooling, TLS verification) **while preserving the full 6-phase timing** that is
|
||||
latprobe's whole point. Python has no equivalent of Go's `net/http/httptrace`,
|
||||
so the phase breakdown is recovered by instrumenting httpx's network backend.
|
||||
|
||||
This is an **additive, Python-only experiment** — intentionally outside the
|
||||
CLAUDE.md "Go first, then Python port" flow, since the user explicitly asked for
|
||||
a library-based Python variant. No Go change is required.
|
||||
|
||||
## Approach
|
||||
|
||||
New sibling package `python/hxprobe/`, reusing everything reusable from
|
||||
`latprobe` (dataclasses, aggregation, duration parsing, CLI rendering) so the
|
||||
only genuinely new code is the httpx probe backend.
|
||||
|
||||
### Key design: instrumented httpx transport
|
||||
|
||||
`httpx` (sync `httpx.Client`) runs on `httpcore`. To recover per-phase timing we
|
||||
subclass httpcore's sync network backend and time the phases at the socket
|
||||
level, letting httpx own HTTP framing, HTTP/2, redirects, and keep-alive:
|
||||
|
||||
- `connect_tcp(...)` — reimplement DNS + TCP as separate steps (port the
|
||||
`socket.getaddrinfo` → `socket.connect` split already in
|
||||
`latprobe/probe.py:167-203`), timestamping **DNS** and **TCP connect**
|
||||
independently, and capturing the resolved IP.
|
||||
- `start_tls(...)` — timestamp the **TLS handshake**; pull negotiated version /
|
||||
cipher / peer cert from the SSL object for verbose mode.
|
||||
- **TTFB** = headers-received minus end-of-TLS (server processing), measured via
|
||||
`client.stream("GET", ...)` (the `stream()` context yields once response
|
||||
headers arrive).
|
||||
- **Transfer** = iterating `resp.iter_raw()` to EOF, minus headers-received.
|
||||
- **Total** = wraps the whole `measure()` call.
|
||||
|
||||
Each `measure()` call uses a **fresh `httpx.Client` (no cross-sample pooling)** so
|
||||
every `-n` sample yields a full phase breakdown — matching the current
|
||||
raw-socket `latprobe` behavior rather than Go's pool-reuse quirk.
|
||||
|
||||
A per-call trace object (held by the backend instance) records timings and the
|
||||
`fail_phase` at the exact point a phase raises, giving precise error
|
||||
classification (`dns`/`connect`/`timeout`/`tls`/`transfer`/`request`) without
|
||||
guessing from httpx exception types.
|
||||
|
||||
**Redirects (followed by default, Go parity):** DNS/connect/TLS are reported
|
||||
from the **first** connection (mirrors Go's `connectStart.IsZero()` guard);
|
||||
TTFB/Transfer/Total span the full followed chain. `redirect_count` and the
|
||||
negotiated `http_version` (`h2` vs `http/1.1`) are surfaced as new verbose
|
||||
fields — a genuine capability the socket version lacks.
|
||||
|
||||
## Files
|
||||
|
||||
**New:**
|
||||
- `python/pyproject.toml` — project metadata; dependency `httpx[http2]` (pulls
|
||||
`h2`). Makes `latprobe` + `hxprobe` `pip install -e .`-able; tests still run
|
||||
via `PYTHONPATH`.
|
||||
- `python/hxprobe/__init__.py`
|
||||
- `python/hxprobe/probe.py` — `measure(url, opts) -> latprobe.probe.Result`
|
||||
(imports & returns the **same `Result`** so aggregation/rendering just work);
|
||||
`_TimingBackend`, the per-call trace, error classification, verbose capture.
|
||||
- `python/hxprobe/cli.py` — thin: delegates to `latprobe.cli.run(...)` passing
|
||||
`measure_fn=hxprobe.probe.measure` (see reuse edit below).
|
||||
- `python/hxprobe/__main__.py` — `sys.exit(cli.run(sys.argv[1:], ...))`.
|
||||
- `python/tests/test_hx_probe.py` — hermetic, local `http.server`: phase
|
||||
presence, **redirect following** (302 handler — validates the Go-parity
|
||||
feature), connection-refused → `connect`, black-hole port → `timeout`.
|
||||
- `python/tests/test_hx_cli.py` — hermetic CLI via `run()` with `io.StringIO`
|
||||
(mirrors `tests/test_cli.py:73-76`).
|
||||
- `python/tests/test_integration_hx.py` — live, **excluded from default gate**
|
||||
(filename starts `test_i…`, so the `test_[!i]*.py` glob skips it): probe a
|
||||
real HTTP/2 host and assert negotiated `http_version == "h2"`; guard with a
|
||||
`@skipUnless(_online())` like `tests/test_integration.py:29-40`.
|
||||
- `docs/usage/py-hxprobe.md` — usage doc (what it does, flags, example with
|
||||
expected output, and an explicit socket-vs-httpx capability comparison table),
|
||||
per CLAUDE.md.
|
||||
|
||||
**Edited (small, backward-compatible):**
|
||||
- `python/latprobe/cli.py` — parameterize `run()` and `_run_samples()` with an
|
||||
injectable `measure_fn` (default = current `latprobe.probe.measure`), so
|
||||
`hxprobe` reuses all argparse, concurrency, exit-code, text/JSON rendering
|
||||
logic. Extend `_print_verbose_block` and `_build_json_entry` to show
|
||||
`http_version` / `redirect_count` **when present** (existing socket path never
|
||||
sets them → output unchanged).
|
||||
- `python/latprobe/probe.py` — add optional fields with safe defaults:
|
||||
`Options.follow_redirects=True`, `Options.http2=True` (ignored by the socket
|
||||
measure); `VerboseDetail.http_version=""`, `VerboseDetail.redirect_count=0`.
|
||||
- `Makefile` — add `py-deps` (create `.venv`, `pip install -e python`),
|
||||
`hx-run` (`cd python && python -m hxprobe $(ARGS)`), and fold `test_hx_*` into
|
||||
the existing `py-test` gate; add `hx-test-integration` for the live h2 check.
|
||||
- `CHANGELOG.md` — append a timestamped one-line entry on completion.
|
||||
|
||||
**Reused as-is:** `latprobe.aggregate.summarize`,
|
||||
`latprobe.duration.parse_duration`, `latprobe.probe.{Result,Phase,CertInfo}`,
|
||||
and the entire `latprobe.cli` renderer via the `measure_fn` injection.
|
||||
|
||||
## Flags / behavior
|
||||
|
||||
Same surface as `latprobe` (`-n/--count`, `-c/--concurrency`, `--timeout`,
|
||||
`--fail`, `--json`, `-v/--verbose`) via the reused parser, plus two new opt-outs
|
||||
for the Go-like defaults: `--no-http2` and `--no-follow-redirects`. Exit codes
|
||||
0–6 stay identical to Go/`latprobe`.
|
||||
|
||||
## Verification
|
||||
|
||||
1. `make py-deps` — create venv, install `httpx[http2]`.
|
||||
2. `make hx-run ARGS="https://example.com"` — full 6-phase text output renders.
|
||||
3. `make hx-run ARGS="-v https://www.cloudflare.com"` — verbose block shows
|
||||
`http_version: h2` and TLS/cert details.
|
||||
4. **Go-parity spot checks:**
|
||||
- HTTP/2: `python -m hxprobe -v <h2-host>` reports `h2` where
|
||||
`python -m latprobe -v <h2-host>` reports HTTP/1.1.
|
||||
- Redirects: `python -m hxprobe http://github.com` follows to https and shows
|
||||
`redirect_count > 0` (socket `latprobe` shows a raw 301).
|
||||
5. `make py-test` — hermetic suite (now including `test_hx_probe.py`,
|
||||
`test_hx_cli.py`) is green; `latprobe`'s existing tests still pass (proves the
|
||||
`measure_fn`/`Options`/`VerboseDetail` edits are backward-compatible).
|
||||
6. `make hx-test-integration` — live test confirms real `h2` negotiation.
|
||||
7. Confirm `--json` output for `hxprobe` matches the `latprobe` schema plus the
|
||||
optional `verbose.http_version` / `verbose.redirect_count` keys.
|
||||
Reference in New Issue
Block a user