http-latency-prober/docs/plans/2026-07-01-23-47-py-hxprobe-httpx.md

# Plan: `hxprobe` — httpx-based Python probe (Go-client parity)

## Context

The existing Python `latprobe` package measures per-phase HTTP latency with raw
sockets. That gives an excellent DNS/TCP/TLS/TTFB/Transfer breakdown, but the
cost is that it always speaks **HTTP/1.1** and **does not follow redirects** —
diverging from the Go implementation, whose `http.DefaultClient` negotiates
**HTTP/2** via ALPN, **follows redirects** (up to 10), pools connections, and
verifies TLS by default (confirmed: `go/internal/probe/probe.go` uses
`http.DefaultClient.Do` with no custom transport/`CheckRedirect`).

Goal: add a **second Python implementation, `hxprobe`**, built on the `httpx`
library so it matches the Go client's protocol capabilities (HTTP/2, redirects,
pooling, TLS verification) **while preserving the full 6-phase timing** that is
latprobe's whole point. Python has no equivalent of Go's `net/http/httptrace`,
so the phase breakdown is recovered by instrumenting httpx's network backend.

This is an **additive, Python-only experiment** — intentionally outside the
CLAUDE.md "Go first, then Python port" flow, since the user explicitly asked for
a library-based Python variant. No Go change is required.

## Approach

New sibling package `python/hxprobe/`, reusing everything reusable from
`latprobe` (dataclasses, aggregation, duration parsing, CLI rendering) so the
only genuinely new code is the httpx probe backend.

### Key design: instrumented httpx transport

`httpx` (sync `httpx.Client`) runs on `httpcore`. To recover per-phase timing we
subclass httpcore's sync network backend and time the phases at the socket
level, letting httpx own HTTP framing, HTTP/2, redirects, and keep-alive:

- `connect_tcp(...)` — reimplement DNS + TCP as separate steps (port the
  `socket.getaddrinfo` → `socket.connect` split already in
  `latprobe/probe.py:167-203`), timestamping **DNS** and **TCP connect**
  independently, and capturing the resolved IP.
- `start_tls(...)` — timestamp the **TLS handshake**; pull negotiated version /
  cipher / peer cert from the SSL object for verbose mode.
- **TTFB** = headers-received minus end-of-TLS (server processing), measured via
  `client.stream("GET", ...)` (the `stream()` context yields once response
  headers arrive).
- **Transfer** = iterating `resp.iter_raw()` to EOF, minus headers-received.
- **Total** = wraps the whole `measure()` call.

Each `measure()` call uses a **fresh `httpx.Client` (no cross-sample pooling)** so
every `-n` sample yields a full phase breakdown — matching the current
raw-socket `latprobe` behavior rather than Go's pool-reuse quirk.

A per-call trace object (held by the backend instance) records timings and the
`fail_phase` at the exact point a phase raises, giving precise error
classification (`dns`/`connect`/`timeout`/`tls`/`transfer`/`request`) without
guessing from httpx exception types.

**Redirects (followed by default, Go parity):** DNS/connect/TLS are reported
from the **first** connection (mirrors Go's `connectStart.IsZero()` guard);
TTFB/Transfer/Total span the full followed chain. `redirect_count` and the
negotiated `http_version` (`h2` vs `http/1.1`) are surfaced as new verbose
fields — a genuine capability the socket version lacks.

## Files

**New:**
- `python/pyproject.toml` — project metadata; dependency `httpx[http2]` (pulls
  `h2`). Makes `latprobe` + `hxprobe` `pip install -e .`-able; tests still run
  via `PYTHONPATH`.
- `python/hxprobe/__init__.py`
- `python/hxprobe/probe.py` — `measure(url, opts) -> latprobe.probe.Result`
  (imports & returns the **same `Result`** so aggregation/rendering just work);
  `_TimingBackend`, the per-call trace, error classification, verbose capture.
- `python/hxprobe/cli.py` — thin: delegates to `latprobe.cli.run(...)` passing
  `measure_fn=hxprobe.probe.measure` (see reuse edit below).
- `python/hxprobe/__main__.py` — `sys.exit(cli.run(sys.argv[1:], ...))`.
- `python/tests/test_hx_probe.py` — hermetic, local `http.server`: phase
  presence, **redirect following** (302 handler — validates the Go-parity
  feature), connection-refused → `connect`, black-hole port → `timeout`.
- `python/tests/test_hx_cli.py` — hermetic CLI via `run()` with `io.StringIO`
  (mirrors `tests/test_cli.py:73-76`).
- `python/tests/test_integration_hx.py` — live, **excluded from default gate**
  (filename starts `test_i…`, so the `test_[!i]*.py` glob skips it): probe a
  real HTTP/2 host and assert negotiated `http_version == "h2"`; guard with a
  `@skipUnless(_online())` like `tests/test_integration.py:29-40`.
- `docs/usage/py-hxprobe.md` — usage doc (what it does, flags, example with
  expected output, and an explicit socket-vs-httpx capability comparison table),
  per CLAUDE.md.

**Edited (small, backward-compatible):**
- `python/latprobe/cli.py` — parameterize `run()` and `_run_samples()` with an
  injectable `measure_fn` (default = current `latprobe.probe.measure`), so
  `hxprobe` reuses all argparse, concurrency, exit-code, text/JSON rendering
  logic. Extend `_print_verbose_block` and `_build_json_entry` to show
  `http_version` / `redirect_count` **when present** (existing socket path never
  sets them → output unchanged).
- `python/latprobe/probe.py` — add optional fields with safe defaults:
  `Options.follow_redirects=True`, `Options.http2=True` (ignored by the socket
  measure); `VerboseDetail.http_version=""`, `VerboseDetail.redirect_count=0`.
- `Makefile` — add `py-deps` (create `.venv`, `pip install -e python`),
  `hx-run` (`cd python && python -m hxprobe $(ARGS)`), and fold `test_hx_*` into
  the existing `py-test` gate; add `hx-test-integration` for the live h2 check.
- `CHANGELOG.md` — append a timestamped one-line entry on completion.

**Reused as-is:** `latprobe.aggregate.summarize`,
`latprobe.duration.parse_duration`, `latprobe.probe.{Result,Phase,CertInfo}`,
and the entire `latprobe.cli` renderer via the `measure_fn` injection.

## Flags / behavior

Same surface as `latprobe` (`-n/--count`, `-c/--concurrency`, `--timeout`,
`--fail`, `--json`, `-v/--verbose`) via the reused parser, plus two new opt-outs
for the Go-like defaults: `--no-http2` and `--no-follow-redirects`. Exit codes
0–6 stay identical to Go/`latprobe`.

## Verification

1. `make py-deps` — create venv, install `httpx[http2]`.
2. `make hx-run ARGS="https://example.com"` — full 6-phase text output renders.
3. `make hx-run ARGS="-v https://www.cloudflare.com"` — verbose block shows
   `http_version: h2` and TLS/cert details.
4. **Go-parity spot checks:**
   - HTTP/2: `python -m hxprobe -v <h2-host>` reports `h2` where
     `python -m latprobe -v <h2-host>` reports HTTP/1.1.
   - Redirects: `python -m hxprobe http://github.com` follows to https and shows
     `redirect_count > 0` (socket `latprobe` shows a raw 301).
5. `make py-test` — hermetic suite (now including `test_hx_probe.py`,
   `test_hx_cli.py`) is green; `latprobe`'s existing tests still pass (proves the
   `measure_fn`/`Options`/`VerboseDetail` edits are backward-compatible).
6. `make hx-test-integration` — live test confirms real `h2` negotiation.
7. Confirm `--json` output for `hxprobe` matches the `latprobe` schema plus the
   optional `verbose.http_version` / `verbose.redirect_count` keys.