http-latency-prober/docs/usage/hxprobe.md

# `hxprobe/` — httpx-based HTTP latency probe (Go-client parity)

> For a full case-by-case runnable reference (every flag, every failure
> mode, real captured output), see [`hxprobe/USAGE.md`](../../hxprobe/USAGE.md).

## What it does

A standalone Python CLI, independent of anything else in this repo (own
`pyproject.toml`, own venv — see [Setup](#setup)). It measures per-phase HTTP
latency like [`python/latprobe`](py-latprobe.md), but is built on the
[httpx](https://www.python-httpx.org/) library instead of raw sockets, so the
client matches Go's `http.DefaultClient`: **HTTP/2 negotiated via ALPN**,
**redirects followed by default**, connection pooling, and default TLS
verification — while still reporting the same six-phase breakdown (DNS, TCP
connect, TLS, TTFB, Transfer, Total).

Python has no equivalent of Go's `net/http/httptrace`, so the phase timing is
recovered by instrumenting httpx's network backend directly
(`hxprobe/probe.py`): a custom `NetworkBackend`/`NetworkStream` pair times
DNS, TCP connect, and TLS at the socket level, while HTTP framing (HTTP/1.1
or HTTP/2), redirect-following, and keep-alive stay entirely owned by httpx.

Run it from the `hxprobe/` directory with `uv run python -m hxprobe`, or via
`make hx-run ARGS="…"` from the repo root.

## Setup

`hxprobe` is a fully self-contained project — it could be copied out of this
repo into its own tomorrow and still work, with its own `pyproject.toml`,
lockfile (`uv.lock`), and [uv](https://docs.astral.sh/uv/)-managed venv
(`hxprobe/.venv`), separate from anything under `python/`. Install once:

```sh
make hx-deps
```

This runs `uv sync` inside `hxprobe/`, creating `.venv` and installing
`httpx` (plus `h2` for HTTP/2), `pytest`, and `ruff` from `uv.lock` — pinned,
reproducible versions, not whatever the resolver happens to pick at install
time. It's a prerequisite of `make hx-run`/`make hx-test`/etc., so those
targets set it up automatically on first run — `make hx-deps` is only needed
if you want to call `uv run python -m hxprobe` directly from inside
`hxprobe/`. See [hxprobe/README.md](../../hxprobe/README.md) for the
from-inside-the-directory quick start.

## Flags / arguments

Same surface as `latprobe`, plus two opt-outs for the Go-like defaults:

```
python -m hxprobe [flags] <url> [url ...]
```

| Flag | Default | Description |
|------|---------|-------------|
| `url …` (positional) | required | One or more URLs to probe |
| `-n N`, `--count N` | `1` | Number of requests per URL |
| `-c N`, `--concurrency N` | `0` (auto) | Max parallel URLs; `0` = `min(len(urls), 8)` |
| `--timeout DURATION` | `10s` | Per-request timeout; supports `ms`, `s`, `m`, or bare seconds |
| `--fail` | off | Exit non-zero when any HTTP status ≥ 400 |
| `--json` | off | Output as JSON array instead of text |
| `-v`, `--verbose` | off | Show resolved IP, negotiated protocol/redirects, TLS info, certificate, response headers |
| `--no-http2` | off (HTTP/2 on) | Disable HTTP/2 negotiation, force HTTP/1.1 |
| `--no-follow-redirects` | off (follow on) | Report the raw redirect response instead of following it |
| `-h`, `--help` | — | Show help and exit 0 |

**Exit codes:** identical to `latprobe` (0 ok, 1 usage, 2 dns, 3 connect,
4 timeout, 5 tls, 6 http≥400 with `--fail`); the highest code across all URLs
is returned. When more than one URL is probed, a summary footer is appended
after the last URL block tallying every URL's outcome (ok / dns / connect /
timeout / tls / http) and the resulting exit code — see "Multi-URL summary
footer" in [`hxprobe/USAGE.md`](../../hxprobe/USAGE.md) for a real example.

## Examples

### Single URL — negotiates HTTP/2 by default
```sh
make hx-run ARGS="https://example.com"
```
```
https://example.com  (200)
  DNS lookup     :      2.81 ms
  TCP connect    :      8.28 ms
  TLS handshake  :     16.13 ms
  Server (TTFB)  :      1.04 ms
  Transfer       :      0.62 ms
  ─────────────────────────────
  Total          :     42.53 ms
```

### Verbose — shows the negotiated protocol
```sh
make hx-run ARGS="-v https://example.com"
```
```
https://example.com  (200)
  DNS lookup     :      2.93 ms
  TCP connect    :      8.58 ms
  TLS handshake  :     17.71 ms
  Server (TTFB)  :      4.35 ms
  Transfer       :      0.84 ms
  ─────────────────────────────
  Total          :     44.06 ms
  IP             :  104.20.23.154
  Protocol       :  HTTP/2
  TLS            :  TLSv1.3  TLS_AES_256_GCM_SHA384  256 bit
  Cert           :  CN=example.com  valid until 2026-08-29  SSL Corporation
```

### Redirects followed by default
```sh
make hx-run ARGS="-v http://github.com"
```
```
http://github.com  (200)
  DNS lookup     :     14.01 ms
  TCP connect    :     19.42 ms
  TLS handshake  :     22.72 ms
  Server (TTFB)  :      0.01 ms
  Transfer       :     65.66 ms
  ─────────────────────────────
  Total          :    196.72 ms
  IP             :  140.82.121.4
  Protocol       :  HTTP/2  (1 redirect)
  TLS            :  TLSv1.3  TLS_AES_128_GCM_SHA256  128 bit
  Cert           :  CN=github.com  valid until 2026-08-02  Sectigo Limited
```

Pass `--no-follow-redirects` to get the raw redirect response instead:
```sh
make hx-run ARGS="--no-follow-redirects http://github.com"
```
```
http://github.com  (301)
  DNS lookup     :      2.85 ms
  TCP connect    :     24.16 ms
  Server (TTFB)  :     24.50 ms
  Transfer       :      0.59 ms
  ─────────────────────────────
  Total          :     52.58 ms
```

### Forcing HTTP/1.1
```sh
make hx-run ARGS="-v --no-http2 https://example.com"
```
```
https://example.com  (200)
  DNS lookup     :      2.31 ms
  TCP connect    :      8.75 ms
  TLS handshake  :     13.61 ms
  Server (TTFB)  :     13.51 ms
  Transfer       :      0.44 ms
  ─────────────────────────────
  Total          :     39.07 ms
  IP             :  104.20.23.154
  Protocol       :  HTTP/1.1
  TLS            :  TLSv1.3  TLS_AES_256_GCM_SHA384  256 bit
  Cert           :  CN=example.com  valid until 2026-08-29  SSL Corporation
```

### Sampling, JSON, `--fail`, DNS failure

Same shape as `latprobe`'s own examples — see
[py-latprobe.md](py-latprobe.md#examples) for `-n`, `--json`, multi-URL,
`--fail`, and timeout output. hxprobe's JSON verbose object adds two keys:

```json
"verbose": {
  "ip": "104.20.23.154",
  "http_version": "HTTP/2",
  "redirect_count": 1,
  "tls_version": "TLSv1.3",
  ...
}
```

`http_version`/`redirect_count` are omitted from the text view when
`http_version` is unknown (only happens on failures before headers arrived).

## HTTP/2, redirects, and a TTFB accuracy note

**HTTP/2 and redirects** are the headline reasons this project exists
alongside the raw-socket `latprobe` — see [py-latprobe.md](py-latprobe.md) for
what that implementation does instead (always HTTP/1.1, never follows
redirects).

**A TTFB finding worth knowing, if you've compared numbers against another
raw-socket HTTP client:** during development, this implementation's TTFB
came in consistently ~40-50ms *lower* than a comparable raw-socket
implementation that didn't set `TCP_NODELAY` on its connect socket. That gap
wasn't noise — a socket that never sets `TCP_NODELAY` is subject to Nagle's
algorithm interacting with the server's delayed-ACK timer, a well-known
~40ms artifact. This implementation sets `TCP_NODELAY` on every connection
(matching both httpcore's own default backend and Go's `net.Dialer`),
avoiding that penalty. Forcing `TCP_NODELAY` onto the other socket
experimentally collapsed its TTFB to match this one's — confirming the
cause. If you're comparing hxprobe's numbers against some other HTTP/1.1
client that doesn't set `TCP_NODELAY`, expect this implementation's TTFB to
read lower, and correctly so.

**Redirect semantics for `dns`/`connect`/`tls` vs `ttfb`/`transfer`:** when
redirects are followed, `dns`/`connect`/`tls` (and the verbose IP/TLS/cert
fields) reflect the **first** connection only — "cost of reaching the origin
server." `ttfb`/`transfer` reflect the **last** hop, because each write/read
call overwrites them — which mirrors how Go's own `httptrace.ClientTrace`
hooks behave for a followed redirect (they aren't guarded either, so the last
hop wins there too).

## Limitations

- Always `GET`, no custom headers/body/auth — matches Go's `http.DefaultClient`.
- `Options.timeout` applies uniformly to connect/read/write/pool phases (a
  single value); it is not split into separate per-phase budgets.
- No UNIX socket support.
- HTTP/2 requires TLS (`https://`) in practice — cleartext `h2c` is not
  attempted for `http://` URLs (matches nearly every real HTTP/2 deployment).

## Makefile targets

```sh
make hx-deps                                   # one-time: uv sync (venv + lockfile install)
make hx-run ARGS="-v -n 3 https://example.com" # run it
make hx-test                                   # hermetic unit tests (pytest -m "not integration")
make hx-test-integration                       # live tests: real HTTP/2 negotiation, redirects
make hx-lint                                   # ruff check
make hx-fmt                                    # ruff format
make hx-check                                  # lint + hermetic tests (the pre-commit-style gate)
```

`hx-check` (lint + hermetic tests) also runs as part of the repo-root
`make check`; `hx-test` runs as part of `make test`.