# Plan: `--verbose` / `-v` flag for `latprobe` **Timestamp:** 2026-07-01 12:55 **Status:** Planned ## Goal Add a `--verbose` / `-v` flag that surfaces diagnostic detail beyond timing: which IP was used, what TLS version and cipher was negotiated, certificate metadata (CN, expiry, issuer), and the most useful response headers. Useful for debugging *why* a probe failed, not just *that* it did. --- ## What verbose mode shows ### Text output — successful HTTPS request ``` https://example.com (200) DNS lookup : 18.87 ms TCP connect : 9.76 ms TLS handshake : 14.71 ms Server (TTFB) : 67.07 ms Transfer : 0.14 ms ───────────────────────────── Total : 118.39 ms IP : 93.184.216.34 TLS : TLSv1.3 TLS_AES_256_GCM_SHA384 256 bit Cert : CN=www.example.com valid until 2025-06-14 DigiCert Inc Server : ECS (dcb/7F84) Content-Type : text/html; charset=UTF-8 ``` ### Text output — TLS failure (with diagnostic cert pass) ``` https://expired.badssl.com/ (FAILED) DNS lookup : 15.39 ms TCP connect : 124.98 ms TLS handshake : 305.30 ms ───────────────────────────── Total : 473.17 ms ✗ tls: certificate verify failed: certificate has expired IP : 104.154.89.105 Cert (unverified) : CN=*.badssl.com EXPIRED 2015-04-09 COMODO RSA Domain Validation ``` ### Text output — DNS failure (minimal; nothing to show after DNS) ``` http://no.such.host.invalid (FAILED) Total : 0.65 ms ✗ dns: [Errno 8] nodename nor servname provided, or not known ``` No verbose block — IP is unknown, no TLS, no headers. ### Text output — HTTP 4xx with redirect header ``` https://www.google.com/this-page-does-not-exist (404) ... Total : 145.50 ms IP : 142.251.36.4 TLS : TLSv1.3 TLS_AES_128_GCM_SHA256 128 bit Cert : CN=*.google.com valid until 2026-01-20 Google Trust Services Server : gws Content-Type : text/html; charset=UTF-8 ``` ### Text output — 3xx redirect (Location shown) ``` https://google.com (301) ... IP : 142.251.36.4 TLS : TLSv1.3 TLS_AES_128_GCM_SHA256 128 bit Cert : CN=*.google.com valid until 2026-01-20 Google Trust Services Location : https://www.google.com/ Server : gws Content-Type : text/html; charset=UTF-8 ``` ### Aggregate verbose (multi-sample `-n N`) ``` https://example.com (200, 5 samples) min avg max ... Total : 97.10 ms 101.20 ms 109.80 ms IP : 93.184.216.34 TLS : TLSv1.3 TLS_AES_256_GCM_SHA384 256 bit Cert : CN=www.example.com valid until 2025-06-14 DigiCert Inc Server : ECS (dcb/7F84) Content-Type : text/html; charset=UTF-8 ``` Verbose block taken from the **last successful sample**. If DNS resolves to multiple IPs and different samples hit different addresses, they are listed as `IP : 93.184.216.34, 93.184.216.35` (deduped, insertion-ordered). --- ## Verbose block format rules - Label column: **14 chars**, left-padded with spaces, matching the phase labels. - Separator: `" : "` (4 chars) — same as phase rows. - Value: free-form string, no fixed width. - Verbose block appears immediately after the last line of the standard block (after `Total` or after the `✗ …` error line). - Block is omitted entirely when there is nothing to show (e.g. pure DNS failure where IP is unknown). - No separator line before the verbose block — the visual break from `Total` and `✗` is enough. Label strings (14 chars each): ``` "IP " # resolved IP address "TLS " # version + cipher + bits "Cert " # CN, expiry, issuer (verified) "Cert (unvrf.) " # same fields, cert not trusted (TLS failure) "Location " # for 3xx responses "Server " # Server response header "Content-Type " # Content-Type response header "X-Cache " # CDN cache status (if present) "Via " # proxy chain (if present) ``` Additional headers shown only when present (see priority list in `_VERBOSE_HEADERS` below). --- ## JSON extension When `--verbose --json` are both set, each entry gets a top-level `"verbose"` object (omitted when `--verbose` is absent): ```json { "url": "https://example.com", "status": 200, "succeeded": 1, "failed": 0, "phases": { ... }, "verbose": { "ip": "93.184.216.34", "tls_version": "TLSv1.3", "tls_cipher": "TLS_AES_256_GCM_SHA384", "tls_bits": 256, "cert": { "cn": "www.example.com", "sans": ["www.example.com", "example.com"], "expiry": "2025-06-14", "issuer_cn": "DigiCert SHA2 Secure Server CA", "verified": true }, "headers": { "Server": "ECS (dcb/7F84)", "Content-Type": "text/html; charset=UTF-8", "X-Cache": "HIT" } } } ``` - `"verbose"` is omitted when `--verbose` is not set. - `"cert"` is omitted for `http://` URLs (no TLS). - For TLS failures: `"cert"` contains what the diagnostic pass found, with `"verified": false`. If the diagnostic pass itself failed, `"cert"` is omitted. - `"headers"` contains **all** parsed response headers (not just the priority list used in text mode). - When `succeeded == 0` the `"verbose"` object may still contain `"ip"` if DNS resolved, but will lack `"tls_version"`, `"cert"`, and `"headers"`. --- ## Data model changes ### `probe.py` — new dataclasses ```python @dataclass class CertInfo: cn: str = "" sans: list[str] = field(default_factory=list) expiry: str = "" # ISO date "YYYY-MM-DD" issuer_cn: str = "" verified: bool = False # True = TLS handshake passed verification @dataclass class VerboseDetail: resolved_ip: str = "" tls_version: str = "" tls_cipher: str = "" tls_bits: int = 0 cert: CertInfo | None = None headers: dict[str, str] = field(default_factory=dict) # all parsed headers ``` ### `probe.py` — `Options` change ```python @dataclass class Options: timeout: float = 10.0 verbose: bool = False # NEW ``` ### `probe.py` — `Result` change ```python @dataclass class Result: ...existing fields... detail: VerboseDetail | None = None # NEW; None when opts.verbose=False ``` --- ## Capture points in `probe.py` ### Resolved IP After `socket.getaddrinfo` succeeds: ```python if opts.verbose: r.detail = VerboseDetail(resolved_ip=str(infos[0][4][0])) ``` `infos[0][4][0]` is the first resolved address string (works for both IPv4 and IPv6 since `[4]` is the full sockaddr tuple and `[0]` is the address). ### TLS version, cipher, and certificate After `ctx.wrap_socket()` succeeds: ```python if opts.verbose and r.detail: r.detail.tls_version = sock.version() or "" cipher_name, _, bits = sock.cipher() r.detail.tls_cipher = cipher_name or "" r.detail.tls_bits = bits or 0 r.detail.cert = _parse_cert(sock.getpeercert(), verified=True) ``` `sock.getpeercert()` returns a dict with `subject`, `issuer`, `notAfter`, `subjectAltName` when called after a successful handshake. ### Response headers The current code reads the first non-empty chunk then drains the body. For verbose mode (and more correctly in general), the probe must accumulate bytes until `\r\n\r\n` is found before recording `t_first_byte`, so that the full header section is available. Modified TTFB loop (replaces the current `while not first_chunk:` block): ```python buf = b"" t_first_byte: float | None = None while b"\r\n\r\n" not in buf: chunk = sock.recv(4096) if not chunk: raise OSError("server closed connection before headers complete") if t_first_byte is None: t_first_byte = time.perf_counter() # first byte — unchanged semantics buf += chunk t_first_byte = t_first_byte or time.perf_counter() r.ttfb = _p(t_wrote, t_first_byte) r.status_code = _parse_status(buf) if opts.verbose and r.detail: r.detail.headers = _parse_response_headers(buf) ``` TTFB semantics are **unchanged** — `t_first_byte` is captured on the first `recv()` call that returns data, not after all headers arrive. The transfer drain loop needs to also drain `buf` bytes that follow `\r\n\r\n` (the body prefix already read into the buffer). ### Certificate inspection on TLS failure When `opts.verbose=True` and `r.fail_phase == "tls"`, call a helper that opens a second, non-verifying connection purely to fetch the certificate: ```python def _fetch_cert_unverified( host: str, port: int, addr, family: int, timeout: float ) -> CertInfo | None: ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) ctx.check_hostname = False ctx.verify_mode = ssl.CERT_NONE sock = socket.socket(family, socket.SOCK_STREAM) sock.settimeout(timeout) try: sock.connect(addr) ssl_sock = ctx.wrap_socket(sock, server_hostname=host) peer = ssl_sock.getpeercert() ssl_sock.close() return _parse_cert(peer, verified=False) if peer else None except OSError: return None finally: sock.close() ``` This runs **after** `measure()` records timing, so it does not affect any phase durations. It adds a short additional wall-clock delay (one more TLS round trip) only in verbose mode on TLS failures — acceptable for a diagnostic path. ### Certificate parsing helper ```python from email.utils import parsedate # for "Jun 14 00:00:00 2025 GMT" import datetime def _parse_cert(peer: dict, verified: bool) -> CertInfo: def _cn(rdns): for rdn in rdns: for k, v in rdn: if k == "commonName": return v return "" sans = [v for k, v in peer.get("subjectAltName", ()) if k == "DNS"] expiry = "" not_after = peer.get("notAfter", "") if not_after: t = parsedate(not_after) # returns time.struct_time or None if t: expiry = f"{t[0]:04d}-{t[1]:02d}-{t[2]:02d}" return CertInfo( cn = _cn(peer.get("subject", ())), sans = sans, expiry = expiry, issuer_cn = _cn(peer.get("issuer", ())), verified = verified, ) ``` ### Response header parsing helper ```python _VERBOSE_HEADERS = [ "Location", "Server", "Content-Type", "X-Cache", "CF-Cache-Status", "Cache-Control", "Via", "X-Powered-By", "Strict-Transport-Security", ] def _parse_response_headers(buf: bytes) -> dict[str, str]: """Parse all headers from a raw HTTP response buffer (up to \\r\\n\\r\\n).""" header_section = buf.split(b"\r\n\r\n", 1)[0] lines = header_section.decode("latin-1", errors="replace").splitlines() headers: dict[str, str] = {} for line in lines[1:]: # skip status line if ":" in line: name, _, value = line.partition(":") headers[name.strip()] = value.strip() return headers ``` --- ## `cli.py` changes ### New flag ```python parser.add_argument( "-v", "--verbose", action="store_true", help="show resolved IP, TLS details, certificate, and response headers", ) ``` Passed into `Options(verbose=ns.verbose)`. ### Text rendering — verbose block ```python def _print_verbose_block(detail: VerboseDetail, out: IO) -> None: rows: list[tuple[str, str]] = [] if detail.resolved_ip: rows.append(("IP ", detail.resolved_ip)) if detail.tls_version: tls_val = detail.tls_version if detail.tls_cipher: tls_val += f" {detail.tls_cipher}" if detail.tls_bits: tls_val += f" {detail.tls_bits} bit" rows.append(("TLS ", tls_val)) if detail.cert: c = detail.cert label = "Cert (unvrf.) " if not c.verified else "Cert " parts = [f"CN={c.cn}"] if c.cn else [] if c.expiry: tag = "EXPIRED" if _cert_expired(c.expiry) else "valid until" parts.append(f"{tag} {c.expiry}") if c.issuer_cn: parts.append(c.issuer_cn) rows.append((label, " ".join(parts))) # Response headers — show priority list, in order, if present for name in _VERBOSE_HEADERS: val = detail.headers.get(name) if val: label = f"{name:<14}" rows.append((label, val)) for label, value in rows: out.write(f" {label} : {value}\n") ``` `_cert_expired(expiry: str) -> bool` compares the ISO date string against today's date using `datetime.date.fromisoformat`. Call site: `_print_verbose_block` is called from `_print_single`, `_print_all_failed`, and `_print_aggregate` — after the last line written by each function — but only when `result.detail` (or `agg`'s last-sample detail) is not `None`. For aggregate rendering: collect `detail` from the last successful result before calling `summarize`. Pass it alongside `agg` to `_print_aggregate`. ### JSON extension In `_build_json_entry`, when `detail` is not `None`: ```python v: dict = {} if detail.resolved_ip: v["ip"] = detail.resolved_ip if detail.tls_version: v["tls_version"] = detail.tls_version v["tls_cipher"] = detail.tls_cipher v["tls_bits"] = detail.tls_bits if detail.cert: c = detail.cert v["cert"] = { "cn": c.cn, "sans": c.sans, "expiry": c.expiry, "issuer_cn": c.issuer_cn, "verified": c.verified, } if detail.headers: v["headers"] = dict(detail.headers) if v: entry["verbose"] = v ``` --- ## Files changed | File | Change | |------|--------| | `python/latprobe/probe.py` | `CertInfo`, `VerboseDetail` dataclasses; `Options.verbose`; `Result.detail`; capture points for IP, TLS, cert, headers; `_fetch_cert_unverified`; `_parse_cert`; `_parse_response_headers`; modified TTFB loop | | `python/latprobe/cli.py` | `-v`/`--verbose` flag; `_print_verbose_block`; verbose call sites in all 3 print functions; `_build_json_entry` extension; `_cert_expired` helper | | `python/tests/test_probe.py` | Tests for verbose fields on success (IP, TLS, cert, headers), on TLS failure (unverified cert), and that non-verbose leaves `detail=None` | | `python/tests/test_cli.py` | Tests for `--verbose` text layout (IP/TLS/cert/header rows), `--verbose --json` schema, verbose absent when flag not set | | `python/tests/test_integration.py` | Tests for real site verbose output (expiry date format, header values, actual IPs), TLS failure cert inspection against badssl.com | | `docs/usage/py-latprobe.md` | New `--verbose` section with example output | --- ## Implementation order 1. **`probe.py` — data model**: add `CertInfo`, `VerboseDetail`, `Options.verbose`, `Result.detail`. No behaviour change yet — all `None` by default. 2. **`probe.py` — capture IP**: set `r.detail.resolved_ip` after `getaddrinfo`. Easiest capture; good checkpoint. 3. **`probe.py` — modify TTFB loop**: accumulate to `\r\n\r\n`, update body drain. This is the riskiest change (touches the hot path); verify existing tests still pass before continuing. 4. **`probe.py` — parse headers**: add `_parse_response_headers`, `_VERBOSE_HEADERS`; populate `r.detail.headers` when verbose. 5. **`probe.py` — TLS details on success**: add `_parse_cert`; populate `tls_version`, `tls_cipher`, `tls_bits`, `cert` after `wrap_socket`. 6. **`probe.py` — TLS failure cert**: add `_fetch_cert_unverified`; call it at the end of `measure` when `opts.verbose and r.fail_phase == "tls"`. 7. **`cli.py` — flag + text rendering**: add `-v`, `_print_verbose_block`, `_cert_expired`; wire into all print functions. 8. **`cli.py` — JSON**: extend `_build_json_entry`. 9. **Tests**: hermetic tests for each capture point; CLI text/JSON tests; integration tests against real sites. 10. **Docs**: update `docs/usage/py-latprobe.md`. Steps 1–2 and 7 can be skipped ahead and demonstrated early (IP line shows up immediately); steps 3–6 build the richer detail progressively. --- ## Known limitations / out of scope - **Aggregate verbose from last sample only** — no per-sample IP list unless sampling actually returned multiple distinct IPs (rare; only with round-robin DNS between samples). - **DNS timeout not controllable** — `getaddrinfo` does not accept a Python timeout; the `--timeout` flag applies only from TCP connect onward. Already documented in existing usage doc. - **No redirect following** — 3xx responses show the `Location` header in the verbose block but do not probe the target. Consistent with non-verbose behaviour. - **Diagnostic cert pass adds latency** — only on TLS failures in verbose mode; documented inline in output (future: could suppress with a flag). - **HTTP/2, HTTP/3 not supported** — raw socket drives HTTP/1.1 only; TLS ALPN negotiation may result in some servers rejecting the connection. Out of scope for this tool.