test(go): step 6 — integration tests covering all exit codes and output
Extract run(args, stdout, stderr) int from main() for in-process testability. Fix TLS failure classification (tlsErr now captured from TLSHandshakeDone hook). Add run_test.go with 14 table-driven in-process tests and cli_test.go with TestMain + 3 subprocess smoke tests. All servers use httptest; .invalid TLD for deterministic DNS failures. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
125
docs/plans/2026-07-01-00-49-integration-tests.md
Normal file
125
docs/plans/2026-07-01-00-49-integration-tests.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# Plan: Step 6 — Integration tests (Go)
|
||||
|
||||
## Context
|
||||
|
||||
The Go `latprobe` tool is feature-complete (Steps 0–5): per-phase latency,
|
||||
multiple URLs, `-n` sampling, `--json`, and classified failure handling with
|
||||
distinct exit codes. There are currently **no automated tests** — every check so
|
||||
far has been manual.
|
||||
|
||||
We want integration tests that demonstrate the application's behaviour across a
|
||||
**successful run and every failure mode**: HTTP success (200), HTTP error status
|
||||
(404/500), DNS failure, connection refused, timeout, and TLS handshake failure —
|
||||
asserting on both the rendered output and the exit code.
|
||||
|
||||
The user chose **both test layers**: fast in-process tests for breadth, plus a
|
||||
few subprocess smoke tests that exercise the real compiled binary.
|
||||
|
||||
## Prerequisite refactor (makes the CLI testable)
|
||||
|
||||
`main()` currently uses the global `flag` package, prints directly to
|
||||
`os.Stdout`/`os.Stderr`, and calls `os.Exit` — none of which is testable.
|
||||
|
||||
Extract the logic into a pure, injectable function in `go/main.go`:
|
||||
|
||||
```go
|
||||
func run(args []string, stdout, stderr io.Writer) int { ... }
|
||||
|
||||
func main() { os.Exit(run(os.Args[1:], os.Stdout, os.Stderr)) }
|
||||
```
|
||||
|
||||
- Use a local `flag.NewFlagSet("latprobe", flag.ContinueOnError)` with
|
||||
`fs.SetOutput(stderr)`; on parse error return `exitUsage`.
|
||||
- Replace every `fmt.Print*` with `fmt.Fprint*(stdout/stderr, …)`.
|
||||
- Thread an `io.Writer` through `printURL`, `printResult`, `printAggregate`,
|
||||
`printFailureSummary`.
|
||||
- Return the exit code instead of calling `os.Exit`.
|
||||
This is a mechanical change; no behaviour changes.
|
||||
|
||||
## Small fix surfaced by the TLS test — `go/internal/probe/probe.go`
|
||||
|
||||
The current `classifyErr` detects TLS failure via "tlsStart set but tlsDone
|
||||
zero". But Go's `httptrace` calls `TLSHandshakeDone` **with the error** on a
|
||||
failed handshake, so `tlsDone` is set even on a cert error — meaning a TLS
|
||||
failure is currently misclassified as `connect`.
|
||||
|
||||
Fix (mirrors the existing `dnsErr` capture): record the handshake error in the
|
||||
`TLSHandshakeDone` hook into a local `tlsErr`, and in `classifyErr` return
|
||||
`"tls"` when `tlsErr != nil`. Classification order: dns → timeout → tls →
|
||||
connect (so a deadline during the handshake still classifies as `timeout`).
|
||||
|
||||
## Test approach
|
||||
|
||||
Stdlib only (`testing`, `net/http/httptest`, `os/exec`, `encoding/json`) — no
|
||||
third-party deps, consistent with the project.
|
||||
|
||||
### How each case is triggered (deterministically)
|
||||
|
||||
| Case | Trigger |
|
||||
|------|---------|
|
||||
| Success 200 | `httptest.NewServer` returning 200 |
|
||||
| HTTP 404 / 500 | `httptest.NewServer` returning the status |
|
||||
| DNS failure | URL with reserved `.invalid` TLD (RFC 6761 — always NXDOMAIN) |
|
||||
| Connection refused | `net.Listen` on `127.0.0.1:0`, capture addr, `Close()`, use that addr |
|
||||
| Timeout | server handler blocks on `<-r.Context().Done()`; client `--timeout 200ms` (handler returns as soon as the client disconnects, so `Close()` doesn't hang) |
|
||||
| TLS failure | `httptest.NewTLSServer` (self-signed cert) → default client rejects → cert error |
|
||||
|
||||
### Layer 1 — in-process (`go/run_test.go`, `package main`)
|
||||
|
||||
Table-driven tests calling `run(args, &stdoutBuf, &stderrBuf)` and asserting on
|
||||
the returned exit code and output substrings. Cases:
|
||||
|
||||
- success 200 → exit 0; output contains `Total`, `DNS lookup`
|
||||
- 500 without `--fail` → exit 0; output contains `(500)`
|
||||
- 404 with `--fail` → exit 6; output contains `404 ✗`
|
||||
- DNS failure → exit 2; output contains `✗ dns:`
|
||||
- connection refused → exit 3; output contains `✗ connect:`
|
||||
- timeout (`--timeout 200ms`) → exit 4; output contains `✗ timeout:`
|
||||
- TLS failure → exit 5; output contains `✗ tls:`
|
||||
- multiple URLs, mixed (200 + `.invalid`) → exit = highest (2)
|
||||
- `-n 3` success → exit 0; output contains `3 samples`
|
||||
- no args → exit 1; stderr contains usage
|
||||
- `--json` success → valid JSON, `phases.total` present, `failed == 0`
|
||||
- `--json` DNS failure → valid JSON, `errors[0].phase == "dns"`, `succeeded == 0`
|
||||
|
||||
JSON cases unmarshal `stdout` into the `[]jsonEntry` shape (or a mirror struct)
|
||||
and assert on fields — verifying phase presence without brittle text matching.
|
||||
|
||||
### Layer 2 — subprocess smoke tests (`go/cli_test.go`, `package main`)
|
||||
|
||||
`TestMain` builds the binary once with `go build -o <tmp>/latprobe` and stores
|
||||
the path; tests `exec.Command` it and read exit code via `*exec.ExitError`.
|
||||
A small representative set (real `os.Exit` path, real binary):
|
||||
|
||||
- success against a local `httptest` server → exit 0, stdout has `Total`
|
||||
- DNS failure (`https://*.invalid`) → exit 2
|
||||
- `--json` DNS failure → stdout parses as JSON with an `errors` entry
|
||||
|
||||
## Files
|
||||
|
||||
- `go/main.go` — refactor to `run(...) int` (+ thread writer through printers)
|
||||
- `go/internal/probe/probe.go` — capture `tlsErr`, fix `classifyErr`
|
||||
- `go/run_test.go` — new, in-process integration matrix
|
||||
- `go/cli_test.go` — new, `TestMain` + subprocess smoke tests
|
||||
- `docs/usage/step-6-integration-tests.md` — how to run the tests
|
||||
- `CHANGELOG.md`, `README.md` — bookkeeping
|
||||
- `docs/plans/2026-07-01-00-49-integration-tests.md` — copy of this plan
|
||||
|
||||
## Verification
|
||||
|
||||
```sh
|
||||
cd go
|
||||
go build ./...
|
||||
go vet ./...
|
||||
go test ./... # all integration + subprocess tests pass
|
||||
go test -v ./... # human-readable per-case results showing behaviour
|
||||
```
|
||||
|
||||
Confirm the matrix covers exit codes 0–6 and that a deliberately broken
|
||||
classification (e.g. revert the TLS fix) makes the TLS case fail — proving the
|
||||
tests actually assert behaviour.
|
||||
|
||||
## Bookkeeping at execution time
|
||||
1. Copy this plan to `docs/plans/2026-07-01-00-49-integration-tests.md`.
|
||||
2. Add `docs/usage/step-6-integration-tests.md`.
|
||||
3. Append a Step 6 entry to `CHANGELOG.md`; add a Step 6 row to `README.md`.
|
||||
Reference in New Issue
Block a user