http-latency-prober/docs/plans/2026-07-01-12-55-py-latprobe-verbose.md

Plan: --verbose / -v flag for latprobe

Timestamp: 2026-07-01 12:55 Status: Planned

Goal

Add a --verbose / -v flag that surfaces diagnostic detail beyond timing: which IP was used, what TLS version and cipher was negotiated, certificate metadata (CN, expiry, issuer), and the most useful response headers. Useful for debugging why a probe failed, not just that it did.

---

What verbose mode shows

Text output — successful HTTPS request

https://example.com  (200)
  DNS lookup     :     18.87 ms
  TCP connect    :      9.76 ms
  TLS handshake  :     14.71 ms
  Server (TTFB)  :     67.07 ms
  Transfer       :      0.14 ms
  ─────────────────────────────
  Total          :    118.39 ms
  IP             :  93.184.216.34
  TLS            :  TLSv1.3  TLS_AES_256_GCM_SHA384  256 bit
  Cert           :  CN=www.example.com  valid until 2025-06-14  DigiCert Inc
  Server         :  ECS (dcb/7F84)
  Content-Type   :  text/html; charset=UTF-8

Text output — TLS failure (with diagnostic cert pass)

https://expired.badssl.com/  (FAILED)
  DNS lookup     :     15.39 ms
  TCP connect    :    124.98 ms
  TLS handshake  :    305.30 ms
  ─────────────────────────────
  Total          :    473.17 ms
  ✗ tls: certificate verify failed: certificate has expired
  IP             :  104.154.89.105
  Cert (unverified) :  CN=*.badssl.com  EXPIRED 2015-04-09  COMODO RSA Domain Validation

Text output — DNS failure (minimal; nothing to show after DNS)

http://no.such.host.invalid  (FAILED)
  Total          :      0.65 ms
  ✗ dns: [Errno 8] nodename nor servname provided, or not known

No verbose block — IP is unknown, no TLS, no headers.

Text output — HTTP 4xx with redirect header

https://www.google.com/this-page-does-not-exist  (404)
  ...
  Total          :    145.50 ms
  IP             :  142.251.36.4
  TLS            :  TLSv1.3  TLS_AES_128_GCM_SHA256  128 bit
  Cert           :  CN=*.google.com  valid until 2026-01-20  Google Trust Services
  Server         :  gws
  Content-Type   :  text/html; charset=UTF-8

Text output — 3xx redirect (Location shown)

https://google.com  (301)
  ...
  IP             :  142.251.36.4
  TLS            :  TLSv1.3  TLS_AES_128_GCM_SHA256  128 bit
  Cert           :  CN=*.google.com  valid until 2026-01-20  Google Trust Services
  Location       :  https://www.google.com/
  Server         :  gws
  Content-Type   :  text/html; charset=UTF-8

Aggregate verbose (multi-sample -n N)

https://example.com  (200, 5 samples)
                          min        avg        max
  ...
  Total          :    97.10 ms  101.20 ms  109.80 ms
  IP             :  93.184.216.34
  TLS            :  TLSv1.3  TLS_AES_256_GCM_SHA384  256 bit
  Cert           :  CN=www.example.com  valid until 2025-06-14  DigiCert Inc
  Server         :  ECS (dcb/7F84)
  Content-Type   :  text/html; charset=UTF-8

Verbose block taken from the last successful sample. If DNS resolves to multiple IPs and different samples hit different addresses, they are listed as IP : 93.184.216.34, 93.184.216.35 (deduped, insertion-ordered).

---

Verbose block format rules

• Label column: 14 chars, left-padded with spaces, matching the phase labels.
• Separator: " : " (4 chars) — same as phase rows.
• Value: free-form string, no fixed width.
• Verbose block appears immediately after the last line of the standard block (after Total or after the ✗ … error line).
• Block is omitted entirely when there is nothing to show (e.g. pure DNS failure where IP is unknown).
• No separator line before the verbose block — the visual break from Total and ✗ is enough.

Label strings (14 chars each):

"IP            "   # resolved IP address
"TLS           "   # version + cipher + bits
"Cert          "   # CN, expiry, issuer (verified)
"Cert (unvrf.) "   # same fields, cert not trusted (TLS failure)
"Location      "   # for 3xx responses
"Server        "   # Server response header
"Content-Type  "   # Content-Type response header
"X-Cache       "   # CDN cache status (if present)
"Via           "   # proxy chain (if present)

Additional headers shown only when present (see priority list in _VERBOSE_HEADERS below).

---

JSON extension

When --verbose --json are both set, each entry gets a top-level "verbose" object (omitted when --verbose is absent):

{
  "url": "https://example.com",
  "status": 200,
  "succeeded": 1,
  "failed": 0,
  "phases": { ... },
  "verbose": {
    "ip": "93.184.216.34",
    "tls_version": "TLSv1.3",
    "tls_cipher": "TLS_AES_256_GCM_SHA384",
    "tls_bits": 256,
    "cert": {
      "cn": "www.example.com",
      "sans": ["www.example.com", "example.com"],
      "expiry": "2025-06-14",
      "issuer_cn": "DigiCert SHA2 Secure Server CA",
      "verified": true
    },
    "headers": {
      "Server": "ECS (dcb/7F84)",
      "Content-Type": "text/html; charset=UTF-8",
      "X-Cache": "HIT"
    }
  }
}

• "verbose" is omitted when --verbose is not set.
• "cert" is omitted for http:// URLs (no TLS).
• For TLS failures: "cert" contains what the diagnostic pass found, with "verified": false. If the diagnostic pass itself failed, "cert" is omitted.
• "headers" contains all parsed response headers (not just the priority list used in text mode).
• When succeeded == 0 the "verbose" object may still contain "ip" if DNS resolved, but will lack "tls_version", "cert", and "headers".

---

Data model changes

probe.py — new dataclasses

@dataclass
class CertInfo:
    cn: str = ""
    sans: list[str] = field(default_factory=list)
    expiry: str = ""       # ISO date "YYYY-MM-DD"
    issuer_cn: str = ""
    verified: bool = False # True = TLS handshake passed verification


@dataclass
class VerboseDetail:
    resolved_ip: str = ""
    tls_version: str = ""
    tls_cipher: str = ""
    tls_bits: int = 0
    cert: CertInfo | None = None
    headers: dict[str, str] = field(default_factory=dict)  # all parsed headers

probe.py — Options change

@dataclass
class Options:
    timeout: float = 10.0
    verbose: bool = False   # NEW

probe.py — Result change

@dataclass
class Result:
    ...existing fields...
    detail: VerboseDetail | None = None   # NEW; None when opts.verbose=False

---

Capture points in probe.py

Resolved IP

After socket.getaddrinfo succeeds:

if opts.verbose:
    r.detail = VerboseDetail(resolved_ip=str(infos[0][4][0]))

infos[0][4][0] is the first resolved address string (works for both IPv4 and IPv6 since [4] is the full sockaddr tuple and [0] is the address).

TLS version, cipher, and certificate

After ctx.wrap_socket() succeeds:

if opts.verbose and r.detail:
    r.detail.tls_version = sock.version() or ""
    cipher_name, _, bits = sock.cipher()
    r.detail.tls_cipher = cipher_name or ""
    r.detail.tls_bits   = bits or 0
    r.detail.cert       = _parse_cert(sock.getpeercert(), verified=True)

sock.getpeercert() returns a dict with subject, issuer, notAfter, subjectAltName when called after a successful handshake.

Response headers

The current code reads the first non-empty chunk then drains the body. For verbose mode (and more correctly in general), the probe must accumulate bytes until \r\n\r\n is found before recording t_first_byte, so that the full header section is available.

Modified TTFB loop (replaces the current while not first_chunk: block):

buf = b""
t_first_byte: float | None = None
while b"\r\n\r\n" not in buf:
    chunk = sock.recv(4096)
    if not chunk:
        raise OSError("server closed connection before headers complete")
    if t_first_byte is None:
        t_first_byte = time.perf_counter()   # first byte — unchanged semantics
    buf += chunk

t_first_byte = t_first_byte or time.perf_counter()
r.ttfb = _p(t_wrote, t_first_byte)
r.status_code = _parse_status(buf)
if opts.verbose and r.detail:
    r.detail.headers = _parse_response_headers(buf)

TTFB semantics are unchanged — t_first_byte is captured on the first recv() call that returns data, not after all headers arrive.

The transfer drain loop needs to also drain buf bytes that follow \r\n\r\n (the body prefix already read into the buffer).

Certificate inspection on TLS failure

When opts.verbose=True and r.fail_phase == "tls", call a helper that opens a second, non-verifying connection purely to fetch the certificate:

def _fetch_cert_unverified(
    host: str, port: int, addr, family: int, timeout: float
) -> CertInfo | None:
    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
    ctx.check_hostname = False
    ctx.verify_mode   = ssl.CERT_NONE
    sock = socket.socket(family, socket.SOCK_STREAM)
    sock.settimeout(timeout)
    try:
        sock.connect(addr)
        ssl_sock = ctx.wrap_socket(sock, server_hostname=host)
        peer = ssl_sock.getpeercert()
        ssl_sock.close()
        return _parse_cert(peer, verified=False) if peer else None
    except OSError:
        return None
    finally:
        sock.close()

This runs after measure() records timing, so it does not affect any phase durations. It adds a short additional wall-clock delay (one more TLS round trip) only in verbose mode on TLS failures — acceptable for a diagnostic path.

Certificate parsing helper

from email.utils import parsedate   # for "Jun 14 00:00:00 2025 GMT"
import datetime

def _parse_cert(peer: dict, verified: bool) -> CertInfo:
    def _cn(rdns):
        for rdn in rdns:
            for k, v in rdn:
                if k == "commonName":
                    return v
        return ""

    sans = [v for k, v in peer.get("subjectAltName", ()) if k == "DNS"]
    expiry = ""
    not_after = peer.get("notAfter", "")
    if not_after:
        t = parsedate(not_after)  # returns time.struct_time or None
        if t:
            expiry = f"{t[0]:04d}-{t[1]:02d}-{t[2]:02d}"

    return CertInfo(
        cn        = _cn(peer.get("subject", ())),
        sans      = sans,
        expiry    = expiry,
        issuer_cn = _cn(peer.get("issuer", ())),
        verified  = verified,
    )

Response header parsing helper

_VERBOSE_HEADERS = [
    "Location", "Server", "Content-Type", "X-Cache",
    "CF-Cache-Status", "Cache-Control", "Via",
    "X-Powered-By", "Strict-Transport-Security",
]

def _parse_response_headers(buf: bytes) -> dict[str, str]:
    """Parse all headers from a raw HTTP response buffer (up to \\r\\n\\r\\n)."""
    header_section = buf.split(b"\r\n\r\n", 1)[0]
    lines = header_section.decode("latin-1", errors="replace").splitlines()
    headers: dict[str, str] = {}
    for line in lines[1:]:   # skip status line
        if ":" in line:
            name, _, value = line.partition(":")
            headers[name.strip()] = value.strip()
    return headers

---

cli.py changes

New flag

parser.add_argument(
    "-v", "--verbose", action="store_true",
    help="show resolved IP, TLS details, certificate, and response headers",
)

Passed into Options(verbose=ns.verbose).

Text rendering — verbose block

def _print_verbose_block(detail: VerboseDetail, out: IO) -> None:
    rows: list[tuple[str, str]] = []

    if detail.resolved_ip:
        rows.append(("IP            ", detail.resolved_ip))

    if detail.tls_version:
        tls_val = detail.tls_version
        if detail.tls_cipher:
            tls_val += f"  {detail.tls_cipher}"
        if detail.tls_bits:
            tls_val += f"  {detail.tls_bits} bit"
        rows.append(("TLS           ", tls_val))

    if detail.cert:
        c = detail.cert
        label = "Cert (unvrf.) " if not c.verified else "Cert          "
        parts = [f"CN={c.cn}"] if c.cn else []
        if c.expiry:
            tag = "EXPIRED" if _cert_expired(c.expiry) else "valid until"
            parts.append(f"{tag} {c.expiry}")
        if c.issuer_cn:
            parts.append(c.issuer_cn)
        rows.append((label, "  ".join(parts)))

    # Response headers — show priority list, in order, if present
    for name in _VERBOSE_HEADERS:
        val = detail.headers.get(name)
        if val:
            label = f"{name:<14}"
            rows.append((label, val))

    for label, value in rows:
        out.write(f"  {label} :  {value}\n")

_cert_expired(expiry: str) -> bool compares the ISO date string against today's date using datetime.date.fromisoformat.

Call site: _print_verbose_block is called from _print_single, _print_all_failed, and _print_aggregate — after the last line written by each function — but only when result.detail (or agg's last-sample detail) is not None.

For aggregate rendering: collect detail from the last successful result before calling summarize. Pass it alongside agg to _print_aggregate.

JSON extension

In _build_json_entry, when detail is not None:

v: dict = {}
if detail.resolved_ip:
    v["ip"] = detail.resolved_ip
if detail.tls_version:
    v["tls_version"] = detail.tls_version
    v["tls_cipher"]  = detail.tls_cipher
    v["tls_bits"]    = detail.tls_bits
if detail.cert:
    c = detail.cert
    v["cert"] = {
        "cn":        c.cn,
        "sans":      c.sans,
        "expiry":    c.expiry,
        "issuer_cn": c.issuer_cn,
        "verified":  c.verified,
    }
if detail.headers:
    v["headers"] = dict(detail.headers)
if v:
    entry["verbose"] = v

---

Files changed

File	Change
python/latprobe/probe.py	CertInfo, VerboseDetail dataclasses; Options.verbose; Result.detail; capture points for IP, TLS, cert, headers; _fetch_cert_unverified; _parse_cert; _parse_response_headers; modified TTFB loop
python/latprobe/cli.py	-v/--verbose flag; _print_verbose_block; verbose call sites in all 3 print functions; _build_json_entry extension; _cert_expired helper
python/tests/test_probe.py	Tests for verbose fields on success (IP, TLS, cert, headers), on TLS failure (unverified cert), and that non-verbose leaves detail=None
python/tests/test_cli.py	Tests for --verbose text layout (IP/TLS/cert/header rows), --verbose --json schema, verbose absent when flag not set
python/tests/test_integration.py	Tests for real site verbose output (expiry date format, header values, actual IPs), TLS failure cert inspection against badssl.com
docs/usage/py-latprobe.md	New --verbose section with example output

---

Implementation order

1. probe.py — data model: add CertInfo, VerboseDetail, Options.verbose, Result.detail. No behaviour change yet — all None by default.
2. probe.py — capture IP: set r.detail.resolved_ip after getaddrinfo. Easiest capture; good checkpoint.
3. probe.py — modify TTFB loop: accumulate to \r\n\r\n, update body drain. This is the riskiest change (touches the hot path); verify existing tests still pass before continuing.
4. probe.py — parse headers: add _parse_response_headers, _VERBOSE_HEADERS; populate r.detail.headers when verbose.
5. probe.py — TLS details on success: add _parse_cert; populate tls_version, tls_cipher, tls_bits, cert after wrap_socket.
6. probe.py — TLS failure cert: add _fetch_cert_unverified; call it at the end of measure when opts.verbose and r.fail_phase == "tls".
7. cli.py — flag + text rendering: add -v, _print_verbose_block, _cert_expired; wire into all print functions.
8. cli.py — JSON: extend _build_json_entry.
9. Tests: hermetic tests for each capture point; CLI text/JSON tests; integration tests against real sites.
10. Docs: update docs/usage/py-latprobe.md.

Steps 1–2 and 7 can be skipped ahead and demonstrated early (IP line shows up immediately); steps 3–6 build the richer detail progressively.

---

Known limitations / out of scope

• Aggregate verbose from last sample only — no per-sample IP list unless sampling actually returned multiple distinct IPs (rare; only with round-robin DNS between samples).
• DNS timeout not controllable — getaddrinfo does not accept a Python timeout; the --timeout flag applies only from TCP connect onward. Already documented in existing usage doc.
• No redirect following — 3xx responses show the Location header in the verbose block but do not probe the target. Consistent with non-verbose behaviour.
• Diagnostic cert pass adds latency — only on TLS failures in verbose mode; documented inline in output (future: could suppress with a flag).
• HTTP/2, HTTP/3 not supported — raw socket drives HTTP/1.1 only; TLS ALPN negotiation may result in some servers rejecting the connection. Out of scope for this tool.