kacerr/http-latency-prober
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(hxprobe): httpx-based HTTP probe — full standalone package

Browse Source 
Third Python implementation in the latency-tool progression.
Mirrors latprobe's feature set but replaces raw sockets with httpx,
gaining HTTP/2 support and redirect following.

Package layout (hxprobe/hxprobe/):
- probe.py: asyncio + httpx.AsyncClient with a _TimingStream transport
  wrapper that captures DNS/connect/TLS/TTFB/transfer phase timings via
  httpx event hooks (get_connection_stats, request_started, etc.).
  VerboseDetail captures resolved IP, TLS version/cipher/bits, cert CN/
  expiry/issuer (from httpx's SSLObject), and response headers.
  Options: timeout, verbose, follow_redirects, http2
- aggregate.py: summarize() → per-phase min/avg/max (same schema as latprobe)
- cli.py: run(args,stdout,stderr)->int; argparse with redirect_stdout/
  redirect_stderr + SystemExit catch; -n/--count, -c/--concurrency,
  --timeout, --fail, --json, -v/--verbose, --no-follow-redirects,
  --no-http2, -f/--file (URL list from file, mutually exclusive with args);
  multi-URL summary footer (tally + exit label) when len(urls) > 1;
  worst-exit-code logic mirrors Go/latprobe; JSON output is bare array
- duration.py: same parse_duration() as latprobe
- Exit codes: 0 ok, 1 usage, 2 dns, 3 connect, 4 timeout, 5 tls, 6 http≥400

Tests (hxprobe/tests/):
- test_probe.py: 18 hermetic tests using anyio + in-process ASGI servers
- test_cli.py: 36 hermetic tests (success, failures, JSON, verbose, -f flag,
  run-summary footer, worst-code accumulation)
- test_integration.py: pytest-marked @integration (excluded from hx-test)

Toolchain: uv + ruff + pytest; pyproject.toml with [dependency-groups];
hxprobe/Makefile standalone (help, deps, run, lint, fmt, test,
test-integration, check, clean); .python-version pins 3.14
Configs: 8 fixture files mirroring python/configs/ (all-ok through mixed)
USAGE.md: 16 runnable examples with real captured output

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Jan Novak
2026-07-02 13:22:31 +02:00
parent 24ea9c9e71
commit 404b372070
24 changed files with 2924 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

703
hxprobe/USAGE.md Normal file
View File

@@ -0,0 +1,703 @@
# `hxprobe` — Runnable Usage Reference
All commands below assume you're inside this directory (`cd hxprobe`) with
dependencies installed (`make deps` or `uv sync`). Timings will differ on
your machine and network; the output structure is stable. Every example
below was run against the live internet.
---
## Basic — single URL
```sh
make run ARGS="https://example.com"
# or directly:
uv run python -m hxprobe https://example.com
```
```
https://example.com (200)
DNS lookup : 17.98 ms
TCP connect : 8.18 ms
TLS handshake : 15.41 ms
Server (TTFB) : 3.38 ms
Transfer : 7.99 ms
─────────────────────────────
Total : 61.93 ms
```
---
## Verbose mode — IP, protocol, TLS, certificate
```sh
uv run python -m hxprobe --verbose https://example.com
```
```
https://example.com (200)
DNS lookup : 2.33 ms
TCP connect : 14.73 ms
TLS handshake : 15.22 ms
Server (TTFB) : 5.63 ms
Transfer : 0.57 ms
─────────────────────────────
Total : 49.16 ms
IP : 104.20.23.154
Protocol : HTTP/2
TLS : TLSv1.3 TLS_AES_256_GCM_SHA384 256 bit
Cert : CN=example.com valid until 2026-08-29 SSL Corporation
```
`Protocol` is the one row `latprobe` (the raw-socket sibling implementation)
never prints — it's the ALPN-negotiated HTTP version, only meaningful for a
client that can actually speak more than one.
---
## Verbose — plain HTTP (no TLS block)
```sh
uv run python -m hxprobe --verbose http://example.com
```
```
http://example.com (200)
DNS lookup : 2.15 ms
TCP connect : 8.52 ms
Server (TTFB) : 16.33 ms
Transfer : 0.42 ms
─────────────────────────────
Total : 27.80 ms
IP : 104.20.23.154
Protocol : HTTP/1.1
```
No `TLS` or `Cert` rows for `http://` URLs. `Protocol` still shows —
HTTP/2 is not attempted over cleartext (see Limitations in
[`docs/usage/hxprobe.md`](../docs/usage/hxprobe.md)), so this is always
`HTTP/1.1`.
---
## Verbose — redirect followed by default
```sh
uv run python -m hxprobe --verbose http://github.com
```
```
http://github.com (200)
DNS lookup : 14.34 ms
TCP connect : 20.88 ms
TLS handshake : 25.91 ms
Server (TTFB) : 3.44 ms
Transfer : 63.33 ms
─────────────────────────────
Total : 197.21 ms
IP : 140.82.121.3
Protocol : HTTP/2 (1 redirect)
TLS : TLSv1.3 TLS_AES_128_GCM_SHA256 128 bit
Cert : CN=github.com valid until 2026-08-02 Sectigo Limited
```
`http://github.com` 301-redirects to `https://github.com`; hxprobe follows
it by default (matching Go's `http.DefaultClient`) and reports the final
response. `latprobe` has no equivalent — it would print the bare `301` and
stop. `DNS`/`TCP connect`/`TLS` are timed from the *first* connection only;
`Server (TTFB)`/`Transfer` reflect the final hop (see "Divergence from
`latprobe`" in [`docs/usage/hxprobe.md`](../docs/usage/hxprobe.md) for why).
---
## `--no-follow-redirects` — report the raw redirect instead
```sh
uv run python -m hxprobe --no-follow-redirects http://github.com
```
```
http://github.com (301)
DNS lookup : 2.85 ms
TCP connect : 24.16 ms
Server (TTFB) : 24.50 ms
Transfer : 0.59 ms
─────────────────────────────
Total : 52.58 ms
```
Same shape `latprobe` would show for any redirect — hxprobe just makes it
opt-in rather than the default.
---
## `--no-http2` — force HTTP/1.1
```sh
uv run python -m hxprobe --verbose --no-http2 https://example.com
```
```
https://example.com (200)
DNS lookup : 2.31 ms
TCP connect : 8.75 ms
TLS handshake : 13.61 ms
Server (TTFB) : 13.51 ms
Transfer : 0.44 ms
─────────────────────────────
Total : 39.07 ms
IP : 104.20.23.154
Protocol : HTTP/1.1
TLS : TLSv1.3 TLS_AES_256_GCM_SHA384 256 bit
Cert : CN=example.com valid until 2026-08-29 SSL Corporation
```
Useful for isolating whether a latency difference is due to protocol version
rather than network conditions.
---
## Verbose — TLS failure (certificate expired)
```sh
uv run python -m hxprobe --verbose https://expired.badssl.com/
echo "exit: $?"
```
```
https://expired.badssl.com/ (FAILED)
DNS lookup : 33.34 ms
TCP connect : 1129.53 ms
TLS handshake : 318.54 ms
─────────────────────────────
Total : 1484.12 ms
✗ tls: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1081)
IP : 104.154.89.105
exit: 5
```
The IP is shown even on TLS failure (DNS and TCP both succeeded). badssl.com
is a shared, often-loaded demo host — `TCP connect` here is unusually slow;
that's the host, not hxprobe.
---
## Verbose — DNS failure (no IP to show)
```sh
uv run python -m hxprobe --verbose http://no.such.host.invalid
echo "exit: $?"
```
```
http://no.such.host.invalid (FAILED)
Total : 13.22 ms
✗ dns: [Errno 8] nodename nor servname provided, or not known
exit: 2
```
The verbose block is empty (IP never resolved), so it's suppressed entirely
— same behavior as `latprobe`.
---
## Sampling (`-n`) — min / avg / max table
```sh
uv run python -m hxprobe -n 3 https://example.com
```
```
https://example.com (200, 3 samples)
min avg max
DNS lookup : 1.23 ms 2.29 ms 3.02 ms
TCP connect : 8.05 ms 8.73 ms 9.81 ms
TLS handshake : 12.99 ms 15.65 ms 17.48 ms
Server (TTFB) : 0.01 ms 7.23 ms 11.95 ms
Transfer : 0.63 ms 2.18 ms 5.15 ms
─────────────────────────────────────────────────
Total : 38.56 ms 49.16 ms 54.70 ms
```
With `--verbose`, the block is appended below the table using the last
successful sample's detail:
```sh
uv run python -m hxprobe --verbose -n 3 https://example.com
```
```
https://example.com (200, 3 samples)
min avg max
...
Total : 39.95 ms 42.55 ms 44.66 ms
IP : 104.20.23.154
Protocol : HTTP/2
TLS : TLSv1.3 TLS_AES_256_GCM_SHA384 256 bit
Cert : CN=example.com valid until 2026-08-29 SSL Corporation
```
---
## Multiple URLs (probed in parallel)
```sh
uv run python -m hxprobe https://example.com https://www.iana.org
```
```
https://example.com (200)
DNS lookup : 3.96 ms
TCP connect : 13.13 ms
TLS handshake : 15.31 ms
Server (TTFB) : 0.01 ms
Transfer : 7.20 ms
─────────────────────────────
Total : 50.16 ms
https://www.iana.org (200)
DNS lookup : 26.67 ms
TCP connect : 9.05 ms
TLS handshake : 14.73 ms
Server (TTFB) : 3.37 ms
Transfer : 0.63 ms
─────────────────────────────
Total : 70.42 ms
─────────────────────────────────────────────────
Summary: 2 URLs — 2 ok
→ exit 0 (ok)
```
Output for each URL is separated by a blank line. Exit code is the worst
across all — see "Multi-URL summary footer" below for how that scalar breaks
down when URLs have different outcomes.
---
## Multi-URL summary footer
Whenever more than one URL is probed (positional args or `-f`), a footer is
appended after the last URL block: a per-outcome tally plus the exit code it
produced. Single-URL runs never show it — text output is otherwise unchanged.
```sh
uv run python -m hxprobe https://example.com http://no.such.host.invalid https://self-signed.badssl.com
echo "exit: $?"
```
```
https://example.com (200)
DNS lookup : 4.65 ms
TCP connect : 9.56 ms
TLS handshake : 14.90 ms
Server (TTFB) : 0.01 ms
Transfer : 8.84 ms
─────────────────────────────
Total : 47.28 ms
http://no.such.host.invalid (FAILED)
Total : 4.54 ms
✗ dns: [Errno 8] nodename nor servname provided, or not known
https://self-signed.badssl.com (FAILED)
DNS lookup : 24.75 ms
TCP connect : 1126.05 ms
TLS handshake : 327.05 ms
─────────────────────────────
Total : 1478.35 ms
✗ tls: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1081)
─────────────────────────────────────────────────
Summary: 3 URLs — 1 ok, 2 failed
✗ dns : 1
✗ tls : 1
→ exit 5 (tls)
exit: 5
```
Each URL is classified by its own worst sample (same `_phase_code` mapping the
overall exit code uses), then tallied. The `→ exit N (label)` line ties the
tally directly to the process exit code, since the code alone can't show
*which* URLs failed *how* — here a DNS failure (would be exit `2` alone) is
outranked by the TLS failure (`5`), and the footer is what makes that visible.
JSON output (`--json`) is unaffected — it stays a bare array; per-URL failures
are already in each entry's `errors[]`.
---
## Reading URLs from a file (`-f`)
```sh
uv run python -m hxprobe -f configs/all-ok.txt
```
```
https://example.com (200)
DNS lookup : 3.60 ms
TCP connect : 17.61 ms
TLS handshake : 16.39 ms
Server (TTFB) : 4.06 ms
Transfer : 0.60 ms
─────────────────────────────
Total : 56.24 ms
https://www.google.com (200)
DNS lookup : 4.18 ms
TCP connect : 17.23 ms
TLS handshake : 30.62 ms
Server (TTFB) : 2.68 ms
Transfer : 69.83 ms
─────────────────────────────
Total : 142.95 ms
https://www.iana.org (200)
DNS lookup : 4.25 ms
TCP connect : 17.18 ms
TLS handshake : 18.87 ms
Server (TTFB) : 8.13 ms
Transfer : 0.50 ms
─────────────────────────────
Total : 60.76 ms
─────────────────────────────────────────────────
Summary: 3 URLs — 3 ok
→ exit 0 (ok)
```
`-f`/`--file` reads a plain-text URL list — one per line, blank lines and
`#`-prefixed comment lines skipped — the same format `simple.py`/`phases.py`
use elsewhere in this repo. It's **mutually exclusive** with positional URL
arguments: pass one or the other, not both.
```sh
uv run python -m hxprobe -f configs/all-ok.txt https://extra.example.com
```
```
usage: hxprobe [-h] [-f PATH] [-n N] [-c N] [--timeout DURATION] [--fail]
[--json] [-v] [--no-http2] [--no-follow-redirects]
[url ...]
hxprobe: error: cannot combine positional url arguments with -f/--file
```
`configs/` ships one fixture per failure class, each self-documenting its
expected exit code in a header comment (verified by actually running it,
not just asserted):
| File | Demonstrates | Exit code |
|------|---------------|-----------|
| `configs/all-ok.txt` | Everything succeeds | 0 |
| `configs/dns-failure.txt` | Unresolvable hostnames | 2 |
| `configs/connection-refused.txt` | Loopback ports with no listener | 3 |
| `configs/timeout.txt` | Non-routable IPs (RFC 5737 TEST-NET-1) | 4 |
| `configs/tls-errors.txt` | badssl.com cert failures | 5 |
| `configs/http-errors.txt` | 404s, run with `--fail` | 6 |
| `configs/mixed.txt` | One of each class, run with `--fail` | 6 |
| `configs/large-mixed.txt` | 30 URLs, ≥10 failing across all 5 classes — good demo of the multi-URL summary footer | 6 |
```sh
uv run python -m hxprobe -f configs/dns-failure.txt
echo "exit: $?"
```
```
https://this-host-does-not-exist.invalid (FAILED)
Total : 16.91 ms
✗ dns: [Errno 8] nodename nor servname provided, or not known
http://no.such.host.invalid (FAILED)
Total : 4.56 ms
✗ dns: [Errno 8] nodename nor servname provided, or not known
─────────────────────────────────────────────────
Summary: 2 URLs — 0 ok, 2 failed
✗ dns : 2
→ exit 2 (dns)
exit: 2
```
`configs/large-mixed.txt` scales this up to 30 URLs so the summary footer has
something substantial to tally — 20 real sites expected to succeed plus 10
deliberately broken across all five failure classes at once:
```sh
uv run python -m hxprobe --fail --timeout 2s -f configs/large-mixed.txt
echo "exit: $?"
```
```
https://example.com (200)
DNS lookup : 3.96 ms
TCP connect : 10.24 ms
TLS handshake : 14.76 ms
Server (TTFB) : 4.14 ms
Transfer : 3.71 ms
─────────────────────────────
Total : 49.54 ms
... 18 more successful URLs ...
https://stackoverflow.com (200)
DNS lookup : 12.99 ms
TCP connect : 9.00 ms
TLS handshake : 16.58 ms
Server (TTFB) : 197.72 ms
Transfer : 289.92 ms
─────────────────────────────
Total : 684.24 ms
https://this-host-does-not-exist.invalid (FAILED)
Total : 1.25 ms
✗ dns: [Errno 8] nodename nor servname provided, or not known
http://no.such.host.invalid (FAILED)
Total : 0.92 ms
✗ dns: [Errno 8] nodename nor servname provided, or not known
http://127.0.0.1:9999 (FAILED)
DNS lookup : 0.01 ms
TCP connect : 0.09 ms
─────────────────────────────
Total : 0.26 ms
✗ connect: [Errno 61] Connection refused
http://127.0.0.1:19999 (FAILED)
DNS lookup : 0.01 ms
TCP connect : 0.09 ms
─────────────────────────────
Total : 0.23 ms
✗ connect: [Errno 61] Connection refused
http://10.255.255.1/ (FAILED)
DNS lookup : 0.03 ms
TCP connect : 16.50 ms
─────────────────────────────
Total : 16.64 ms
✗ connect: [Errno 61] Connection refused
http://192.0.2.1/ (FAILED)
DNS lookup : 0.01 ms
TCP connect : 2001.27 ms
─────────────────────────────
Total : 2001.56 ms
✗ timeout: timed out
https://expired.badssl.com/ (FAILED)
DNS lookup : 16.37 ms
TCP connect : 1125.42 ms
TLS handshake : 329.12 ms
─────────────────────────────
Total : 1471.48 ms
✗ tls: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1081)
https://self-signed.badssl.com/ (FAILED)
DNS lookup : 13.27 ms
TCP connect : 1127.27 ms
TLS handshake : 325.84 ms
─────────────────────────────
Total : 1466.90 ms
✗ tls: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1081)
https://www.google.com/this-page-does-not-exist-at-all-1234567890 (404 ✗)
DNS lookup : 0.96 ms
TCP connect : 7.61 ms
TLS handshake : 25.25 ms
Server (TTFB) : 0.01 ms
Transfer : 99.63 ms
─────────────────────────────
Total : 142.38 ms
https://github.com/this-repo-does-not-exist-abcxyz123/no-way (404 ✗)
DNS lookup : 1.23 ms
TCP connect : 19.57 ms
TLS handshake : 22.02 ms
Server (TTFB) : 209.47 ms
Transfer : 67.20 ms
─────────────────────────────
Total : 339.82 ms
─────────────────────────────────────────────────
Summary: 30 URLs — 20 ok, 10 failed
✗ dns : 2
✗ connect : 3
✗ timeout : 1
✗ tls : 2
✗ http : 2
→ exit 6 (http)
exit: 6
```
The individual URL blocks above are unchanged in shape from every other
example on this page — this is purely a matter of scale. The summary footer
is where scale actually pays off: instead of scanning 30 blocks to count
outcomes, `Summary: 30 URLs — 20 ok, 10 failed` plus the per-class breakdown
answers "what happened" at a glance, and `→ exit 6 (http)` explains *why*
that particular exit code came back (the two 404s under `--fail`, at
`EXIT_HTTP = 6`, outrank every other class present).
---
## `--fail` flag — exit non-zero on HTTP 4xx/5xx
```sh
uv run python -m hxprobe --fail https://www.google.com/this-page-does-not-exist-at-all-1234567890
echo "exit: $?"
```
```
https://www.google.com/this-page-does-not-exist-at-all-1234567890 (404 ✗)
DNS lookup : 2.96 ms
TCP connect : 11.57 ms
TLS handshake : 24.95 ms
Server (TTFB) : 0.01 ms
Transfer : 104.22 ms
─────────────────────────────
Total : 152.77 ms
exit: 6
```
Without `--fail`, 4xx/5xx responses are shown normally and the exit code is `0`.
---
## JSON output
```sh
uv run python -m hxprobe --json https://example.com | python3.14 -m json.tool
```
```json
[
{
"url": "https://example.com",
"status": 200,
"succeeded": 1,
"failed": 0,
"phases": {
"dns": {"min_ms": 2.80, "avg_ms": 2.80, "max_ms": 2.80},
"connect": {"min_ms": 11.68, "avg_ms": 11.68, "max_ms": 11.68},
"tls": {"min_ms": 14.52, "avg_ms": 14.52, "max_ms": 14.52},
"ttfb": {"min_ms": 6.49, "avg_ms": 6.49, "max_ms": 6.49},
"transfer": {"min_ms": 0.81, "avg_ms": 0.81, "max_ms": 0.81},
"total": {"min_ms": 45.59, "avg_ms": 45.59, "max_ms": 45.59}
}
}
]
```
Same shape as `latprobe`'s JSON — `phases` is omitted when all samples
failed, `tls` is omitted for `http://` URLs.
---
## JSON + verbose
```sh
uv run python -m hxprobe --verbose --json https://example.com
```
```json
[
{
"url": "https://example.com",
"status": 200,
"succeeded": 1,
"failed": 0,
"phases": { "...": "..." },
"verbose": {
"ip": "104.20.23.154",
"http_version": "HTTP/2",
"redirect_count": 0,
"tls_version": "TLSv1.3",
"tls_cipher": "TLS_AES_256_GCM_SHA384",
"tls_bits": 256,
"cert": {
"cn": "example.com",
"sans": ["example.com", "*.example.com"],
"expiry": "2026-08-29",
"issuer_cn": "SSL Corporation",
"verified": true
},
"headers": {
"content-type": "text/html",
"server": "cloudflare",
"cf-cache-status": "HIT"
}
}
}
]
```
`"http_version"`/`"redirect_count"` are the two keys `latprobe`'s JSON never
has (its `VerboseDetail.http_version` stays `""`). `"headers"` contains
**all** parsed response headers (the text view shows only a priority list).
---
## Timeout
```sh
uv run python -m hxprobe --timeout 500ms http://192.0.2.1/
echo "exit: $?"
```
```
http://192.0.2.1/ (FAILED)
DNS lookup : 3.11 ms
TCP connect : 501.51 ms
─────────────────────────────
Total : 505.20 ms
✗ timeout: timed out
exit: 4
```
`192.0.2.1` is in RFC 5737's TEST-NET-1 range — reserved for documentation,
guaranteed unreachable, and the kernel gets no reply so the connect phase
runs the full `--timeout` before giving up. `--timeout` accepts `ms`, `s`,
`m` suffixes or a bare number of seconds.
---
## Exit codes
| Code | Meaning |
|------|---------|
| 0 | All probes succeeded (or HTTP 4xx without `--fail`) |
| 1 | Usage / argument error |
| 2 | DNS failure |
| 3 | TCP connect failure |
| 4 | Timeout |
| 5 | TLS error |
| 6 | HTTP status ≥ 400 with `--fail` |
The **highest** exit code across all URLs is returned as the process exit —
identical scheme to `latprobe` and the Go implementation. This scalar is kept
unchanged for compatibility; for multi-URL runs, the "Multi-URL summary
footer" section above shows the full per-URL breakdown behind it.
---
## Makefile shortcuts
From inside this directory:
```sh
make run ARGS="--verbose https://example.com" # run hxprobe
make test # hermetic tests only
make test-integration # live internet tests
make lint # ruff check
make fmt # ruff format
make check # lint + hermetic tests
```
From the parent repo's root, the equivalent shortcuts are prefixed `hx-`:
```sh
make hx-run ARGS="--verbose https://example.com"
make hx-test
make hx-test-integration
make hx-check
```
Both Makefiles are independent — neither calls into the other — so either
works whether you're inside a checkout of the full monorepo or a standalone
copy of just `hxprobe/`.