http-latency-prober/docs/summaries/2026-07-02-00-29-py-hxprobe-httpx.md

Summary: hxprobe — httpx-based Python probe

Plan: docs/plans/2026-07-01-23-47-py-hxprobe-httpx.md

What was built

A new sibling package, python/hxprobe/, alongside the existing raw-socket latprobe. It uses httpx so the client matches Go's http.DefaultClient: HTTP/2 negotiated via ALPN, redirects followed by default, connection pooling, and default TLS verification — while still reporting the full six-phase breakdown (DNS, TCP connect, TLS, TTFB, Transfer, Total).

Files:

• python/pyproject.toml — first third-party dependency in this repo (httpx[http2])
• python/hxprobe/probe.py — the core: _Trace, _TimingStream, _TimingBackend, _TimingTransport, measure()
• python/hxprobe/cli.py, __main__.py, __init__.py — thin wrappers
• python/latprobe/probe.py (edited) — added Options.follow_redirects/ Options.http2 (ignored by the socket measure()) and VerboseDetail.http_version/redirect_count (always ""/0 there)
• python/latprobe/cli.py (edited) — run()/_run_samples() take an injectable measure_fn, plus prog/description/protocol_flags overrides, so hxprobe.cli.run() reuses the entire argparse/concurrency/ exit-code/rendering pipeline unchanged
• python/tests/test_hx_probe.py (17 tests), test_hx_cli.py (11 tests), test_integration_hx.py (9 live tests, excluded from the default gate via the existing test_i* naming convention)
• Makefile — py-deps (creates python/.venv, installs httpx[http2]), hx-run, hx-test-integration; py-test/py-check now run through the venv and include the new hermetic hxprobe tests
• docs/usage/py-hxprobe.md — usage doc with real captured output
• .gitignore — added *.egg-info/ (editable-install artifact)

Key design decisions

• Instrument the transport, don't hand-roll HTTP. Subclassed httpcore.NetworkBackend/NetworkStream to time DNS/TCP connect/TLS at the socket level, letting httpx own HTTP/1.1 vs HTTP/2 framing, redirects, and keep-alive. This was the reason to use httpx at all — get the protocol behavior of a real client while keeping latprobe's phase granularity.
• First-hop-wins for dns/connect/tls; last-hop-wins for ttfb/transfer. When redirects are followed, connection-identity fields (dns/connect/tls timing, resolved IP, TLS/cert info) reflect the first connection. wrote_request/first_byte are simply overwritten on every write/read, so they naturally end up reflecting the last hop — which mirrors how Go's own unguarded httptrace.ClientTrace hooks behave for a followed redirect.
• Fresh httpx.Client per measure() call, no cross-sample pooling — matches latprobe's per-call socket creation so every -n sample gets a full phase breakdown.
• measure_fn injection over subclassing/duplication in latprobe.cli, so hxprobe reuses argparse, concurrency, exit codes, and text/JSON rendering with zero duplicated logic — the socket and httpx probes only differ in probe.py.
• New CLI flags gated behind protocol_flags=True so latprobe's own --help output stays byte-for-byte unchanged (verified) — --no-http2/ --no-follow-redirects only appear for hxprobe.

Notable finding (not part of the original plan)

While comparing hxprobe and latprobe timings against the same live host, hxprobe's TTFB was consistently ~40-50ms lower. Verified experimentally (not just assumed) that this is a real effect, not noise: latprobe's raw socket never sets TCP_NODELAY, so its request write is subject to Nagle's algorithm interacting with the server's delayed-ACK timer — a well-known artifact. Forcing TCP_NODELAY onto latprobe's socket collapsed its TTFB to match hxprobe's. hxprobe sets TCP_NODELAY (matching httpcore's own default backend and Go's net.Dialer), so its TTFB numbers are the more accurate of the two — not just different. Did not change latprobe itself (out of scope for this task); documented the divergence in docs/usage/py-hxprobe.md, in the CHANGELOG, and inline in hxprobe/probe.py.

Deviations from the plan

• Plan sketched TTFB = headers-received minus end-of-TLS; implemented as TTFB = first-byte minus wrote-request instead (matches both Go and the existing latprobe definition — the plan's phrasing was an approximation).
• Plan said verbose TLS/cert metadata could follow "last hop"; implemented as first-hop-wins uniformly across dns/connect/tls/ip/tls-info for simplicity and consistency (only ttfb/transfer are last-hop).
• Everything else (library choice, transport-instrumentation approach, pyproject.toml, sibling-package placement, flag surface) matches the approved plan as written.

Verification

• make check (Go + Python full gate): exit 0, 83 hermetic Python tests (72 pre-existing + 11 new hermetic CLI + hxprobe's share of the 17 probe tests already counted), zero regressions to latprobe's original 39.
• make hx-test-integration: 9/9 live tests pass, including real HTTP/2 negotiation against example.com (Cloudflare) and a real http→https redirect follow against github.com.
• Manual spot checks: --json schema, --fail exit code 6, DNS failure (exit 2), connection-refused (exit 3), --no-http2/--no-follow-redirects flag behavior, latprobe --help output diffed byte-for-byte against pre-change output to confirm no regression.
• Caught and reverted an incidental gofmt whitespace diff in go/internal/probe/probe.go that make check's go-fmt step produced — unrelated to this change, out of scope, not committed.