http-latency-prober/docs/plans/2026-07-01-12-55-py-latprobe-verbose.md

# Plan: `--verbose` / `-v` flag for `latprobe`

**Timestamp:** 2026-07-01 12:55
**Status:** Planned

## Goal

Add a `--verbose` / `-v` flag that surfaces diagnostic detail beyond timing:
which IP was used, what TLS version and cipher was negotiated, certificate
metadata (CN, expiry, issuer), and the most useful response headers.  Useful
for debugging *why* a probe failed, not just *that* it did.

---

## What verbose mode shows

### Text output — successful HTTPS request

```
https://example.com  (200)
  DNS lookup     :     18.87 ms
  TCP connect    :      9.76 ms
  TLS handshake  :     14.71 ms
  Server (TTFB)  :     67.07 ms
  Transfer       :      0.14 ms
  ─────────────────────────────
  Total          :    118.39 ms
  IP             :  93.184.216.34
  TLS            :  TLSv1.3  TLS_AES_256_GCM_SHA384  256 bit
  Cert           :  CN=www.example.com  valid until 2025-06-14  DigiCert Inc
  Server         :  ECS (dcb/7F84)
  Content-Type   :  text/html; charset=UTF-8
```

### Text output — TLS failure (with diagnostic cert pass)

```
https://expired.badssl.com/  (FAILED)
  DNS lookup     :     15.39 ms
  TCP connect    :    124.98 ms
  TLS handshake  :    305.30 ms
  ─────────────────────────────
  Total          :    473.17 ms
  ✗ tls: certificate verify failed: certificate has expired
  IP             :  104.154.89.105
  Cert (unverified) :  CN=*.badssl.com  EXPIRED 2015-04-09  COMODO RSA Domain Validation
```

### Text output — DNS failure (minimal; nothing to show after DNS)

```
http://no.such.host.invalid  (FAILED)
  Total          :      0.65 ms
  ✗ dns: [Errno 8] nodename nor servname provided, or not known
```
No verbose block — IP is unknown, no TLS, no headers.

### Text output — HTTP 4xx with redirect header

```
https://www.google.com/this-page-does-not-exist  (404)
  ...
  Total          :    145.50 ms
  IP             :  142.251.36.4
  TLS            :  TLSv1.3  TLS_AES_128_GCM_SHA256  128 bit
  Cert           :  CN=*.google.com  valid until 2026-01-20  Google Trust Services
  Server         :  gws
  Content-Type   :  text/html; charset=UTF-8
```

### Text output — 3xx redirect (Location shown)

```
https://google.com  (301)
  ...
  IP             :  142.251.36.4
  TLS            :  TLSv1.3  TLS_AES_128_GCM_SHA256  128 bit
  Cert           :  CN=*.google.com  valid until 2026-01-20  Google Trust Services
  Location       :  https://www.google.com/
  Server         :  gws
  Content-Type   :  text/html; charset=UTF-8
```

### Aggregate verbose (multi-sample `-n N`)

```
https://example.com  (200, 5 samples)
                          min        avg        max
  ...
  Total          :    97.10 ms  101.20 ms  109.80 ms
  IP             :  93.184.216.34
  TLS            :  TLSv1.3  TLS_AES_256_GCM_SHA384  256 bit
  Cert           :  CN=www.example.com  valid until 2025-06-14  DigiCert Inc
  Server         :  ECS (dcb/7F84)
  Content-Type   :  text/html; charset=UTF-8
```
Verbose block taken from the **last successful sample**.  If DNS resolves to
multiple IPs and different samples hit different addresses, they are listed as
`IP : 93.184.216.34, 93.184.216.35` (deduped, insertion-ordered).

---

## Verbose block format rules

- Label column: **14 chars**, left-padded with spaces, matching the phase labels.
- Separator: `" :  "` (4 chars) — same as phase rows.
- Value: free-form string, no fixed width.
- Verbose block appears immediately after the last line of the standard block
  (after `Total` or after the `✗ …` error line).
- Block is omitted entirely when there is nothing to show (e.g. pure DNS
  failure where IP is unknown).
- No separator line before the verbose block — the visual break from `Total`
  and `✗` is enough.

Label strings (14 chars each):
```
"IP            "   # resolved IP address
"TLS           "   # version + cipher + bits
"Cert          "   # CN, expiry, issuer (verified)
"Cert (unvrf.) "   # same fields, cert not trusted (TLS failure)
"Location      "   # for 3xx responses
"Server        "   # Server response header
"Content-Type  "   # Content-Type response header
"X-Cache       "   # CDN cache status (if present)
"Via           "   # proxy chain (if present)
```

Additional headers shown only when present (see priority list in
`_VERBOSE_HEADERS` below).

---

## JSON extension

When `--verbose --json` are both set, each entry gets a top-level `"verbose"`
object (omitted when `--verbose` is absent):

```json
{
  "url": "https://example.com",
  "status": 200,
  "succeeded": 1,
  "failed": 0,
  "phases": { ... },
  "verbose": {
    "ip": "93.184.216.34",
    "tls_version": "TLSv1.3",
    "tls_cipher": "TLS_AES_256_GCM_SHA384",
    "tls_bits": 256,
    "cert": {
      "cn": "www.example.com",
      "sans": ["www.example.com", "example.com"],
      "expiry": "2025-06-14",
      "issuer_cn": "DigiCert SHA2 Secure Server CA",
      "verified": true
    },
    "headers": {
      "Server": "ECS (dcb/7F84)",
      "Content-Type": "text/html; charset=UTF-8",
      "X-Cache": "HIT"
    }
  }
}
```

- `"verbose"` is omitted when `--verbose` is not set.
- `"cert"` is omitted for `http://` URLs (no TLS).
- For TLS failures: `"cert"` contains what the diagnostic pass found, with
  `"verified": false`.  If the diagnostic pass itself failed, `"cert"` is omitted.
- `"headers"` contains **all** parsed response headers (not just the priority
  list used in text mode).
- When `succeeded == 0` the `"verbose"` object may still contain `"ip"` if DNS
  resolved, but will lack `"tls_version"`, `"cert"`, and `"headers"`.

---

## Data model changes

### `probe.py` — new dataclasses

```python
@dataclass
class CertInfo:
    cn: str = ""
    sans: list[str] = field(default_factory=list)
    expiry: str = ""       # ISO date "YYYY-MM-DD"
    issuer_cn: str = ""
    verified: bool = False # True = TLS handshake passed verification


@dataclass
class VerboseDetail:
    resolved_ip: str = ""
    tls_version: str = ""
    tls_cipher: str = ""
    tls_bits: int = 0
    cert: CertInfo | None = None
    headers: dict[str, str] = field(default_factory=dict)  # all parsed headers
```

### `probe.py` — `Options` change

```python
@dataclass
class Options:
    timeout: float = 10.0
    verbose: bool = False   # NEW
```

### `probe.py` — `Result` change

```python
@dataclass
class Result:
    ...existing fields...
    detail: VerboseDetail | None = None   # NEW; None when opts.verbose=False
```

---

## Capture points in `probe.py`

### Resolved IP
After `socket.getaddrinfo` succeeds:
```python
if opts.verbose:
    r.detail = VerboseDetail(resolved_ip=str(infos[0][4][0]))
```
`infos[0][4][0]` is the first resolved address string (works for both IPv4
and IPv6 since `[4]` is the full sockaddr tuple and `[0]` is the address).

### TLS version, cipher, and certificate
After `ctx.wrap_socket()` succeeds:
```python
if opts.verbose and r.detail:
    r.detail.tls_version = sock.version() or ""
    cipher_name, _, bits = sock.cipher()
    r.detail.tls_cipher = cipher_name or ""
    r.detail.tls_bits   = bits or 0
    r.detail.cert       = _parse_cert(sock.getpeercert(), verified=True)
```
`sock.getpeercert()` returns a dict with `subject`, `issuer`, `notAfter`,
`subjectAltName` when called after a successful handshake.

### Response headers
The current code reads the first non-empty chunk then drains the body.
For verbose mode (and more correctly in general), the probe must accumulate
bytes until `\r\n\r\n` is found before recording `t_first_byte`, so that the
full header section is available.

Modified TTFB loop (replaces the current `while not first_chunk:` block):

```python
buf = b""
t_first_byte: float | None = None
while b"\r\n\r\n" not in buf:
    chunk = sock.recv(4096)
    if not chunk:
        raise OSError("server closed connection before headers complete")
    if t_first_byte is None:
        t_first_byte = time.perf_counter()   # first byte — unchanged semantics
    buf += chunk

t_first_byte = t_first_byte or time.perf_counter()
r.ttfb = _p(t_wrote, t_first_byte)
r.status_code = _parse_status(buf)
if opts.verbose and r.detail:
    r.detail.headers = _parse_response_headers(buf)
```

TTFB semantics are **unchanged** — `t_first_byte` is captured on the first
`recv()` call that returns data, not after all headers arrive.

The transfer drain loop needs to also drain `buf` bytes that follow
`\r\n\r\n` (the body prefix already read into the buffer).

### Certificate inspection on TLS failure
When `opts.verbose=True` and `r.fail_phase == "tls"`, call a helper that
opens a second, non-verifying connection purely to fetch the certificate:

```python
def _fetch_cert_unverified(
    host: str, port: int, addr, family: int, timeout: float
) -> CertInfo | None:
    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
    ctx.check_hostname = False
    ctx.verify_mode   = ssl.CERT_NONE
    sock = socket.socket(family, socket.SOCK_STREAM)
    sock.settimeout(timeout)
    try:
        sock.connect(addr)
        ssl_sock = ctx.wrap_socket(sock, server_hostname=host)
        peer = ssl_sock.getpeercert()
        ssl_sock.close()
        return _parse_cert(peer, verified=False) if peer else None
    except OSError:
        return None
    finally:
        sock.close()
```

This runs **after** `measure()` records timing, so it does not affect any
phase durations.  It adds a short additional wall-clock delay (one more TLS
round trip) only in verbose mode on TLS failures — acceptable for a diagnostic
path.

### Certificate parsing helper

```python
from email.utils import parsedate   # for "Jun 14 00:00:00 2025 GMT"
import datetime

def _parse_cert(peer: dict, verified: bool) -> CertInfo:
    def _cn(rdns):
        for rdn in rdns:
            for k, v in rdn:
                if k == "commonName":
                    return v
        return ""

    sans = [v for k, v in peer.get("subjectAltName", ()) if k == "DNS"]
    expiry = ""
    not_after = peer.get("notAfter", "")
    if not_after:
        t = parsedate(not_after)  # returns time.struct_time or None
        if t:
            expiry = f"{t[0]:04d}-{t[1]:02d}-{t[2]:02d}"

    return CertInfo(
        cn        = _cn(peer.get("subject", ())),
        sans      = sans,
        expiry    = expiry,
        issuer_cn = _cn(peer.get("issuer", ())),
        verified  = verified,
    )
```

### Response header parsing helper

```python
_VERBOSE_HEADERS = [
    "Location", "Server", "Content-Type", "X-Cache",
    "CF-Cache-Status", "Cache-Control", "Via",
    "X-Powered-By", "Strict-Transport-Security",
]

def _parse_response_headers(buf: bytes) -> dict[str, str]:
    """Parse all headers from a raw HTTP response buffer (up to \\r\\n\\r\\n)."""
    header_section = buf.split(b"\r\n\r\n", 1)[0]
    lines = header_section.decode("latin-1", errors="replace").splitlines()
    headers: dict[str, str] = {}
    for line in lines[1:]:   # skip status line
        if ":" in line:
            name, _, value = line.partition(":")
            headers[name.strip()] = value.strip()
    return headers
```

---

## `cli.py` changes

### New flag
```python
parser.add_argument(
    "-v", "--verbose", action="store_true",
    help="show resolved IP, TLS details, certificate, and response headers",
)
```
Passed into `Options(verbose=ns.verbose)`.

### Text rendering — verbose block

```python
def _print_verbose_block(detail: VerboseDetail, out: IO) -> None:
    rows: list[tuple[str, str]] = []

    if detail.resolved_ip:
        rows.append(("IP            ", detail.resolved_ip))

    if detail.tls_version:
        tls_val = detail.tls_version
        if detail.tls_cipher:
            tls_val += f"  {detail.tls_cipher}"
        if detail.tls_bits:
            tls_val += f"  {detail.tls_bits} bit"
        rows.append(("TLS           ", tls_val))

    if detail.cert:
        c = detail.cert
        label = "Cert (unvrf.) " if not c.verified else "Cert          "
        parts = [f"CN={c.cn}"] if c.cn else []
        if c.expiry:
            tag = "EXPIRED" if _cert_expired(c.expiry) else "valid until"
            parts.append(f"{tag} {c.expiry}")
        if c.issuer_cn:
            parts.append(c.issuer_cn)
        rows.append((label, "  ".join(parts)))

    # Response headers — show priority list, in order, if present
    for name in _VERBOSE_HEADERS:
        val = detail.headers.get(name)
        if val:
            label = f"{name:<14}"
            rows.append((label, val))

    for label, value in rows:
        out.write(f"  {label} :  {value}\n")
```

`_cert_expired(expiry: str) -> bool` compares the ISO date string against
today's date using `datetime.date.fromisoformat`.

Call site: `_print_verbose_block` is called from `_print_single`,
`_print_all_failed`, and `_print_aggregate` — after the last line written by
each function — but only when `result.detail` (or `agg`'s last-sample detail)
is not `None`.

For aggregate rendering: collect `detail` from the last successful result
before calling `summarize`.  Pass it alongside `agg` to `_print_aggregate`.

### JSON extension

In `_build_json_entry`, when `detail` is not `None`:
```python
v: dict = {}
if detail.resolved_ip:
    v["ip"] = detail.resolved_ip
if detail.tls_version:
    v["tls_version"] = detail.tls_version
    v["tls_cipher"]  = detail.tls_cipher
    v["tls_bits"]    = detail.tls_bits
if detail.cert:
    c = detail.cert
    v["cert"] = {
        "cn":        c.cn,
        "sans":      c.sans,
        "expiry":    c.expiry,
        "issuer_cn": c.issuer_cn,
        "verified":  c.verified,
    }
if detail.headers:
    v["headers"] = dict(detail.headers)
if v:
    entry["verbose"] = v
```

---

## Files changed

| File | Change |
|------|--------|
| `python/latprobe/probe.py` | `CertInfo`, `VerboseDetail` dataclasses; `Options.verbose`; `Result.detail`; capture points for IP, TLS, cert, headers; `_fetch_cert_unverified`; `_parse_cert`; `_parse_response_headers`; modified TTFB loop |
| `python/latprobe/cli.py` | `-v`/`--verbose` flag; `_print_verbose_block`; verbose call sites in all 3 print functions; `_build_json_entry` extension; `_cert_expired` helper |
| `python/tests/test_probe.py` | Tests for verbose fields on success (IP, TLS, cert, headers), on TLS failure (unverified cert), and that non-verbose leaves `detail=None` |
| `python/tests/test_cli.py` | Tests for `--verbose` text layout (IP/TLS/cert/header rows), `--verbose --json` schema, verbose absent when flag not set |
| `python/tests/test_integration.py` | Tests for real site verbose output (expiry date format, header values, actual IPs), TLS failure cert inspection against badssl.com |
| `docs/usage/py-latprobe.md` | New `--verbose` section with example output |

---

## Implementation order

1. **`probe.py` — data model**: add `CertInfo`, `VerboseDetail`, `Options.verbose`,
   `Result.detail`.  No behaviour change yet — all `None` by default.
2. **`probe.py` — capture IP**: set `r.detail.resolved_ip` after `getaddrinfo`.
   Easiest capture; good checkpoint.
3. **`probe.py` — modify TTFB loop**: accumulate to `\r\n\r\n`, update body drain.
   This is the riskiest change (touches the hot path); verify existing tests
   still pass before continuing.
4. **`probe.py` — parse headers**: add `_parse_response_headers`, `_VERBOSE_HEADERS`;
   populate `r.detail.headers` when verbose.
5. **`probe.py` — TLS details on success**: add `_parse_cert`; populate
   `tls_version`, `tls_cipher`, `tls_bits`, `cert` after `wrap_socket`.
6. **`probe.py` — TLS failure cert**: add `_fetch_cert_unverified`; call it at the
   end of `measure` when `opts.verbose and r.fail_phase == "tls"`.
7. **`cli.py` — flag + text rendering**: add `-v`, `_print_verbose_block`,
   `_cert_expired`; wire into all print functions.
8. **`cli.py` — JSON**: extend `_build_json_entry`.
9. **Tests**: hermetic tests for each capture point; CLI text/JSON tests;
   integration tests against real sites.
10. **Docs**: update `docs/usage/py-latprobe.md`.

Steps 1–2 and 7 can be skipped ahead and demonstrated early (IP line shows up
immediately); steps 3–6 build the richer detail progressively.

---

## Known limitations / out of scope

- **Aggregate verbose from last sample only** — no per-sample IP list unless
  sampling actually returned multiple distinct IPs (rare; only with round-robin
  DNS between samples).
- **DNS timeout not controllable** — `getaddrinfo` does not accept a Python
  timeout; the `--timeout` flag applies only from TCP connect onward.  Already
  documented in existing usage doc.
- **No redirect following** — 3xx responses show the `Location` header in the
  verbose block but do not probe the target.  Consistent with non-verbose
  behaviour.
- **Diagnostic cert pass adds latency** — only on TLS failures in verbose mode;
  documented inline in output (future: could suppress with a flag).
- **HTTP/2, HTTP/3 not supported** — raw socket drives HTTP/1.1 only; TLS
  ALPN negotiation may result in some servers rejecting the connection.  Out of
  scope for this tool.