Jan Novak
f487a4b1bd
Plans (docs/plans/): - 2026-07-01-23-47-py-hxprobe-httpx.md — initial httpx probe design - 2026-07-02-09-32 through 14-05 — standalone project, toolchain, usage doc + Makefile, file input (-f), simplification pass, run-summary footer Summaries (docs/summaries/): one per completed feature, recording what was actually built, deviations from the plan, and verification steps Explanations (docs/explanations/): two deep-dives written during review — hxprobe concurrency model and worst-exit-code + render-loop analysis Usage (docs/usage/hxprobe.md): overview with pointer to hxprobe/USAGE.md for the full runnable reference Walkthrough (docs/py-latprobe-walkthrough.md): narrative tour of the latprobe Python package for interview / code-review context CHANGELOG.md: entries for all hxprobe features (toolchain, usage doc, file input, simplification, run-summary footer) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
152 lines
8.5 KiB
Markdown
152 lines
8.5 KiB
Markdown
# Plan: extract `hxprobe` into a fully standalone top-level project
|
|
|
|
## Context
|
|
|
|
`hxprobe` currently lives at `python/hxprobe/` and imports shared code from
|
|
`latprobe` (`latprobe.probe.{Options,Phase,Result,VerboseDetail,_parse_cert}`,
|
|
`latprobe.cli.run`). That coupling was a deliberate reuse choice at the time,
|
|
but the user now wants `hxprobe` to be a genuinely independent project — no
|
|
`from latprobe import ...` anywhere, duplication accepted — and, per their
|
|
follow-up answer, independent enough that it could be `cp -r`'d into its own
|
|
repo tomorrow: own `pyproject.toml`, own venv, own Makefile section, own
|
|
top-level directory (sibling to `go/` and `python/`), not nested under `python/`.
|
|
|
|
Confirmed via `git status`/`git log`: everything hxprobe-related
|
|
(`python/hxprobe/`, `python/pyproject.toml`, `python/tests/test_hx_*.py`,
|
|
`docs/usage/py-hxprobe.md`, the two `docs/plans`/`docs/summaries` entries) is
|
|
still uncommitted from this session, and `python/latprobe/cli.py` /
|
|
`python/latprobe/probe.py` only diverge from the last commit (`24ea9c9`) by
|
|
the reuse-oriented additions made to support hxprobe. So this is a clean,
|
|
low-risk restructuring: move already-debugged code, revert latprobe with
|
|
`git checkout --`, no git history surgery needed.
|
|
|
|
## Target layout
|
|
|
|
```
|
|
hxprobe/ (new, top-level, sibling of go/ and python/)
|
|
├── pyproject.toml (own manifest: httpx[http2] dependency)
|
|
├── hxprobe/
|
|
│ ├── __init__.py
|
|
│ ├── probe.py (own Options/Phase/Result/VerboseDetail/CertInfo
|
|
│ │ + own _parse_cert/_parse_cert_date + existing
|
|
│ │ _Trace/_TimingStream/_TimingBackend/
|
|
│ │ _TimingTransport/measure() — unchanged logic)
|
|
│ ├── aggregate.py (verbatim copy of latprobe/aggregate.py —
|
|
│ │ its `from .probe import Result` is already
|
|
│ │ package-relative, needs zero edits)
|
|
│ ├── duration.py (verbatim copy of latprobe/duration.py — no
|
|
│ │ imports at all)
|
|
│ ├── cli.py (full standalone CLI — see below)
|
|
│ └── __main__.py (unchanged: `from .cli import run`)
|
|
└── tests/
|
|
├── __init__.py (empty, matches python/tests/__init__.py)
|
|
├── test_probe.py (moved from python/tests/test_hx_probe.py)
|
|
├── test_cli.py (moved from python/tests/test_hx_cli.py)
|
|
└── test_integration.py (moved from python/tests/test_integration_hx.py)
|
|
```
|
|
|
|
`python/` reverts to containing only `latprobe` — zero third-party deps, no
|
|
`pyproject.toml`, no venv, exactly its pre-hxprobe state.
|
|
|
|
## Step-by-step
|
|
|
|
**1. Move already-debugged files (preserve the bug fixes already made):**
|
|
`git mv`/`mv` (untracked, so plain `mv` is fine) `python/hxprobe/{probe.py,__init__.py,__main__.py}`
|
|
to `hxprobe/hxprobe/`, and the three `python/tests/test_hx_*.py` /
|
|
`test_integration_hx.py` files to `hxprobe/tests/` with the `hx_`/`_hx` name
|
|
segments dropped (`test_probe.py`, `test_cli.py`, `test_integration.py`).
|
|
Do **not** rewrite these from scratch — they already have the mark_dns /
|
|
verbose-on-failure-path / Content-Length-on-keep-alive fixes found during
|
|
the original implementation.
|
|
|
|
**2. Inline the dataclasses into `hxprobe/hxprobe/probe.py`:**
|
|
Replace `from latprobe.probe import Options, Phase, Result, VerboseDetail, _parse_cert`
|
|
with local definitions copied verbatim from `python/latprobe/probe.py`:
|
|
`CertInfo`, `VerboseDetail` (its `http_version`/`redirect_count` fields are
|
|
now simply always-meaningful, no more "populated only by hxprobe" caveat
|
|
comment needed), `Options` (with `follow_redirects`/`http2` as normal fields,
|
|
no more "ignored by socket measure()" caveat), `Phase`, `Result`,
|
|
`_parse_cert`, `_parse_cert_date`. Everything else in the file (`_Trace`,
|
|
`_TimingStream`, `_TimingBackend`, `_TimingTransport`, `measure()`,
|
|
`_classify`, `_unwrap`, `_fill_phases`, `_fill_verbose`) is untouched.
|
|
|
|
**3. Create `hxprobe/hxprobe/aggregate.py` and `duration.py`:**
|
|
Verbatim copies of `python/latprobe/aggregate.py` and `duration.py`.
|
|
|
|
**4. Write `hxprobe/hxprobe/cli.py` as a full standalone CLI:**
|
|
Start from the *current* `python/latprobe/cli.py` (it already has 100% of
|
|
the needed logic, including the `--no-http2`/`--no-follow-redirects` flags,
|
|
the verbose "Protocol" row, and the JSON `http_version`/`redirect_count`
|
|
keys — all added earlier specifically for hxprobe). Strip the
|
|
generalization scaffolding that only existed to let `latprobe` share this
|
|
code:
|
|
|
|
- Remove the `MeasureFn` type alias and the `measure_fn` parameter from
|
|
`_run_samples`/`run` — call `measure` directly (module-level import from
|
|
`.probe`).
|
|
- Remove `run()`'s `prog`/`description`/`protocol_flags` parameters —
|
|
hardcode `prog="hxprobe"` and the httpx-specific description.
|
|
- Make the `--no-http2`/`--no-follow-redirects` `add_argument` calls
|
|
unconditional (drop the `if protocol_flags:` guard).
|
|
- Everything else (exit codes, `_ArgExit`/`_Parser`, phase-label/verbose
|
|
constants, all `_print_*`/`_build_json_entry` rendering) carries over
|
|
unchanged — it's already correct standalone logic.
|
|
|
|
**5. Revert `python/latprobe/` to its pre-hxprobe state:**
|
|
`git checkout -- python/latprobe/cli.py python/latprobe/probe.py` (safe:
|
|
confirmed these are the only diffs since the last commit, and both diffs
|
|
are exactly the reuse scaffolding being removed here).
|
|
|
|
**6. Clean up the old shared-package artifacts:**
|
|
Delete `python/hxprobe/`, `python/pyproject.toml`, `python/.venv/`, and the
|
|
three `python/tests/test_hx_*`/`test_integration_hx.py` files (now moved).
|
|
|
|
**7. Makefile:**
|
|
- Remove `PY_VENV`/`PY_VENV_PYTHON` vars and the `py-deps` target; revert
|
|
`py-test`/`py-test-integration`/`py-check` to invoke `$(PYTHON)` directly
|
|
with no `py-deps` prerequisite (their pre-hxprobe form). Remove `hx-run`/
|
|
`hx-test-integration` from the Python section (moving out).
|
|
- Add `HX_DIR := hxprobe`, `HX_VENV := $(HX_DIR)/.venv`,
|
|
`HX_VENV_PYTHON := $(HX_VENV)/bin/$(PYTHON)` and a new "── hxprobe
|
|
(standalone) ──" section: `hx-deps` (idempotent venv+pip install, mirrors
|
|
the removed `py-deps`), `hx-run`, `hx-test` (hermetic, `test_[!i]*.py`
|
|
glob), `hx-test-integration` (`test_integration.py` pattern), `hx-check`,
|
|
`hx-clean`.
|
|
- Fold into the umbrella targets: `test: go-test py-test hx-test`,
|
|
`check: go-check py-check hx-check`, `clean: go-clean py-clean hx-clean`.
|
|
|
|
**8. Docs:**
|
|
- Rename `docs/usage/py-hxprobe.md` → `docs/usage/hxprobe.md` (drop the
|
|
`py-` prefix — it's no longer part of the Python port). Update: remove the
|
|
"reuses `latprobe.cli.run()` in full" claim (now false — say it's a fully
|
|
standalone implementation sharing only the *design*, not the code, with
|
|
`latprobe`); update setup/Makefile-target sections to `make hx-deps`/
|
|
`hx-run`/`hx-test`/`hx-test-integration`; update file paths from
|
|
`python/hxprobe/` to `hxprobe/hxprobe/`. Keep the TCP_NODELAY/Nagle finding
|
|
and the example transcripts (still accurate — behavior is unchanged, only
|
|
location/packaging changed).
|
|
- `CHANGELOG.md`: new entry describing the extraction.
|
|
- `docs/summaries/`: new dated summary per the CLAUDE.md convention
|
|
(leave the original `2026-07-02-00-29-py-hxprobe-httpx.md` as-is — it's a
|
|
historical record of that implementation; this is a follow-up).
|
|
- `docs/plans/`: save this plan as
|
|
`docs/plans/<yyyy-mm-dd-hh-mm>-hxprobe-standalone-project.md` at
|
|
implementation start, per CLAUDE.md.
|
|
|
|
## Verification
|
|
|
|
1. `rm -rf hxprobe/.venv python/.venv` (fresh state) then `make hx-deps` —
|
|
creates `hxprobe/.venv`, installs `httpx[http2]` from `hxprobe/pyproject.toml`.
|
|
2. `grep -rn "latprobe" hxprobe/` — must return nothing (proves the
|
|
decoupling).
|
|
3. `make hx-run ARGS="-v http://github.com"` — same output as before (HTTP/2,
|
|
1 redirect followed, IP/TLS/cert shown).
|
|
4. `make hx-test` — all hermetic hxprobe tests pass standalone.
|
|
5. `make hx-test-integration` — live HTTP/2 + redirect tests still pass.
|
|
6. `make py-test` — confirms `latprobe`'s own suite is back to its original,
|
|
dependency-free form and still green (no `py-deps` needed to run it).
|
|
7. `make check` (top-level) — Go + latprobe + hxprobe all green in one gate.
|
|
8. `diff <(python -m latprobe --help) <(git show 24ea9c9:python/latprobe/cli.py | ...)` —
|
|
or simpler: confirm `python -m latprobe --help` output is byte-identical
|
|
to before this whole feature existed (no leftover `--no-http2` etc.).
|