test(go): step 6 — integration tests covering all exit codes and output
Extract run(args, stdout, stderr) int from main() for in-process testability. Fix TLS failure classification (tlsErr now captured from TLSHandshakeDone hook). Add run_test.go with 14 table-driven in-process tests and cli_test.go with TestMain + 3 subprocess smoke tests. All servers use httptest; .invalid TLD for deterministic DNS failures. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
125
docs/plans/2026-07-01-00-49-integration-tests.md
Normal file
125
docs/plans/2026-07-01-00-49-integration-tests.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# Plan: Step 6 — Integration tests (Go)
|
||||
|
||||
## Context
|
||||
|
||||
The Go `latprobe` tool is feature-complete (Steps 0–5): per-phase latency,
|
||||
multiple URLs, `-n` sampling, `--json`, and classified failure handling with
|
||||
distinct exit codes. There are currently **no automated tests** — every check so
|
||||
far has been manual.
|
||||
|
||||
We want integration tests that demonstrate the application's behaviour across a
|
||||
**successful run and every failure mode**: HTTP success (200), HTTP error status
|
||||
(404/500), DNS failure, connection refused, timeout, and TLS handshake failure —
|
||||
asserting on both the rendered output and the exit code.
|
||||
|
||||
The user chose **both test layers**: fast in-process tests for breadth, plus a
|
||||
few subprocess smoke tests that exercise the real compiled binary.
|
||||
|
||||
## Prerequisite refactor (makes the CLI testable)
|
||||
|
||||
`main()` currently uses the global `flag` package, prints directly to
|
||||
`os.Stdout`/`os.Stderr`, and calls `os.Exit` — none of which is testable.
|
||||
|
||||
Extract the logic into a pure, injectable function in `go/main.go`:
|
||||
|
||||
```go
|
||||
func run(args []string, stdout, stderr io.Writer) int { ... }
|
||||
|
||||
func main() { os.Exit(run(os.Args[1:], os.Stdout, os.Stderr)) }
|
||||
```
|
||||
|
||||
- Use a local `flag.NewFlagSet("latprobe", flag.ContinueOnError)` with
|
||||
`fs.SetOutput(stderr)`; on parse error return `exitUsage`.
|
||||
- Replace every `fmt.Print*` with `fmt.Fprint*(stdout/stderr, …)`.
|
||||
- Thread an `io.Writer` through `printURL`, `printResult`, `printAggregate`,
|
||||
`printFailureSummary`.
|
||||
- Return the exit code instead of calling `os.Exit`.
|
||||
This is a mechanical change; no behaviour changes.
|
||||
|
||||
## Small fix surfaced by the TLS test — `go/internal/probe/probe.go`
|
||||
|
||||
The current `classifyErr` detects TLS failure via "tlsStart set but tlsDone
|
||||
zero". But Go's `httptrace` calls `TLSHandshakeDone` **with the error** on a
|
||||
failed handshake, so `tlsDone` is set even on a cert error — meaning a TLS
|
||||
failure is currently misclassified as `connect`.
|
||||
|
||||
Fix (mirrors the existing `dnsErr` capture): record the handshake error in the
|
||||
`TLSHandshakeDone` hook into a local `tlsErr`, and in `classifyErr` return
|
||||
`"tls"` when `tlsErr != nil`. Classification order: dns → timeout → tls →
|
||||
connect (so a deadline during the handshake still classifies as `timeout`).
|
||||
|
||||
## Test approach
|
||||
|
||||
Stdlib only (`testing`, `net/http/httptest`, `os/exec`, `encoding/json`) — no
|
||||
third-party deps, consistent with the project.
|
||||
|
||||
### How each case is triggered (deterministically)
|
||||
|
||||
| Case | Trigger |
|
||||
|------|---------|
|
||||
| Success 200 | `httptest.NewServer` returning 200 |
|
||||
| HTTP 404 / 500 | `httptest.NewServer` returning the status |
|
||||
| DNS failure | URL with reserved `.invalid` TLD (RFC 6761 — always NXDOMAIN) |
|
||||
| Connection refused | `net.Listen` on `127.0.0.1:0`, capture addr, `Close()`, use that addr |
|
||||
| Timeout | server handler blocks on `<-r.Context().Done()`; client `--timeout 200ms` (handler returns as soon as the client disconnects, so `Close()` doesn't hang) |
|
||||
| TLS failure | `httptest.NewTLSServer` (self-signed cert) → default client rejects → cert error |
|
||||
|
||||
### Layer 1 — in-process (`go/run_test.go`, `package main`)
|
||||
|
||||
Table-driven tests calling `run(args, &stdoutBuf, &stderrBuf)` and asserting on
|
||||
the returned exit code and output substrings. Cases:
|
||||
|
||||
- success 200 → exit 0; output contains `Total`, `DNS lookup`
|
||||
- 500 without `--fail` → exit 0; output contains `(500)`
|
||||
- 404 with `--fail` → exit 6; output contains `404 ✗`
|
||||
- DNS failure → exit 2; output contains `✗ dns:`
|
||||
- connection refused → exit 3; output contains `✗ connect:`
|
||||
- timeout (`--timeout 200ms`) → exit 4; output contains `✗ timeout:`
|
||||
- TLS failure → exit 5; output contains `✗ tls:`
|
||||
- multiple URLs, mixed (200 + `.invalid`) → exit = highest (2)
|
||||
- `-n 3` success → exit 0; output contains `3 samples`
|
||||
- no args → exit 1; stderr contains usage
|
||||
- `--json` success → valid JSON, `phases.total` present, `failed == 0`
|
||||
- `--json` DNS failure → valid JSON, `errors[0].phase == "dns"`, `succeeded == 0`
|
||||
|
||||
JSON cases unmarshal `stdout` into the `[]jsonEntry` shape (or a mirror struct)
|
||||
and assert on fields — verifying phase presence without brittle text matching.
|
||||
|
||||
### Layer 2 — subprocess smoke tests (`go/cli_test.go`, `package main`)
|
||||
|
||||
`TestMain` builds the binary once with `go build -o <tmp>/latprobe` and stores
|
||||
the path; tests `exec.Command` it and read exit code via `*exec.ExitError`.
|
||||
A small representative set (real `os.Exit` path, real binary):
|
||||
|
||||
- success against a local `httptest` server → exit 0, stdout has `Total`
|
||||
- DNS failure (`https://*.invalid`) → exit 2
|
||||
- `--json` DNS failure → stdout parses as JSON with an `errors` entry
|
||||
|
||||
## Files
|
||||
|
||||
- `go/main.go` — refactor to `run(...) int` (+ thread writer through printers)
|
||||
- `go/internal/probe/probe.go` — capture `tlsErr`, fix `classifyErr`
|
||||
- `go/run_test.go` — new, in-process integration matrix
|
||||
- `go/cli_test.go` — new, `TestMain` + subprocess smoke tests
|
||||
- `docs/usage/step-6-integration-tests.md` — how to run the tests
|
||||
- `CHANGELOG.md`, `README.md` — bookkeeping
|
||||
- `docs/plans/2026-07-01-00-49-integration-tests.md` — copy of this plan
|
||||
|
||||
## Verification
|
||||
|
||||
```sh
|
||||
cd go
|
||||
go build ./...
|
||||
go vet ./...
|
||||
go test ./... # all integration + subprocess tests pass
|
||||
go test -v ./... # human-readable per-case results showing behaviour
|
||||
```
|
||||
|
||||
Confirm the matrix covers exit codes 0–6 and that a deliberately broken
|
||||
classification (e.g. revert the TLS fix) makes the TLS case fail — proving the
|
||||
tests actually assert behaviour.
|
||||
|
||||
## Bookkeeping at execution time
|
||||
1. Copy this plan to `docs/plans/2026-07-01-00-49-integration-tests.md`.
|
||||
2. Add `docs/usage/step-6-integration-tests.md`.
|
||||
3. Append a Step 6 entry to `CHANGELOG.md`; add a Step 6 row to `README.md`.
|
||||
90
docs/usage/step-6-integration-tests.md
Normal file
90
docs/usage/step-6-integration-tests.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Step 6 — Integration Tests
|
||||
|
||||
## What this step delivers
|
||||
|
||||
A full integration-test suite covering every success and failure mode of the
|
||||
`latprobe` CLI. Tests are written using the Go standard library only (`testing`,
|
||||
`net/http/httptest`, `os/exec`, `encoding/json`) — no third-party dependencies.
|
||||
|
||||
Two test layers:
|
||||
|
||||
| Layer | File | How it runs |
|
||||
|-------|------|-------------|
|
||||
| In-process | `go/run_test.go` | Calls `run(args, stdout, stderr)` directly; fast, covers the full matrix |
|
||||
| Subprocess | `go/cli_test.go` | Builds the real binary once, `exec.Command`s it; exercises the true `os.Exit` path |
|
||||
|
||||
## Running the tests
|
||||
|
||||
```sh
|
||||
cd go
|
||||
|
||||
# Run all tests (quiet)
|
||||
go test ./...
|
||||
|
||||
# Run with per-case output
|
||||
go test -v ./...
|
||||
|
||||
# Run only in-process tests
|
||||
go test -v -run TestRunMatrix .
|
||||
go test -v -run TestJSON .
|
||||
|
||||
# Run only subprocess smoke tests
|
||||
go test -v -run TestCLI .
|
||||
```
|
||||
|
||||
## Test matrix
|
||||
|
||||
### In-process (`run_test.go`)
|
||||
|
||||
| Test case | Exit code | Assertion |
|
||||
|-----------|-----------|-----------|
|
||||
| Success 200 | 0 | stdout contains `(200)`, `TCP connect`, `Total` |
|
||||
| HTTP 500, no `--fail` | 0 | stdout contains `(500)`, `Total` |
|
||||
| HTTP 404, `--fail` | 6 | stdout contains `404 ✗` |
|
||||
| DNS failure (`.invalid` TLD) | 2 | stdout contains `✗ dns:` |
|
||||
| Connection refused (listen-then-close) | 3 | stdout contains `✗ connect:` |
|
||||
| Timeout (`--timeout 200ms`, blocking handler) | 4 | stdout contains `✗ timeout:` |
|
||||
| TLS failure (self-signed cert) | 5 | stdout contains `✗ tls:` |
|
||||
| Multiple URLs (200 + `.invalid`) | 2 (highest) | stdout contains both `(200)` and `✗ dns:` |
|
||||
| Sampling `-n 3`, all success | 0 | stdout contains `3 samples`, `min`, `avg`, `max` |
|
||||
| No args | 1 | stderr contains `Usage:` |
|
||||
| `-h` | 0 | stderr contains `Usage:` |
|
||||
| `--json` success | 0 | valid JSON, `phases.total` present, `failed == 0` |
|
||||
| `--json` DNS failure | 2 | valid JSON, `errors[0].phase == "dns"`, `succeeded == 0` |
|
||||
| `--json` `-n 3` success | 0 | `succeeded == 3`, `total.min_ms > 0`, `max_ms >= min_ms` |
|
||||
|
||||
### Subprocess smoke tests (`cli_test.go`)
|
||||
|
||||
`TestMain` builds the binary with `go build -o <tmp>/latprobe .` once before
|
||||
any test runs. The binary is deleted on test completion.
|
||||
|
||||
| Test | Checks |
|
||||
|------|--------|
|
||||
| `TestCLISuccess` | Local server, exit 0, stdout has `(200)` and `Total` |
|
||||
| `TestCLIDNSFailure` | `.invalid` host, real binary exits 2 |
|
||||
| `TestCLIJSONDNSFailure` | `.invalid` host, JSON output, `errors[0].phase == "dns"` |
|
||||
|
||||
## Notes
|
||||
|
||||
- The `http: TLS handshake error` log line printed during the TLS test is the
|
||||
**server-side** log of the client correctly rejecting the self-signed cert.
|
||||
It is expected and harmless.
|
||||
- `httptest.NewServer` binds to `127.0.0.1`; Go resolves loopback addresses
|
||||
without a DNS query, so the DNS row does not appear in localhost test output.
|
||||
Tests use `TCP connect` as the success-path phase assertion instead.
|
||||
- The test suite requires no network access for any case except DNS failure,
|
||||
which uses the reserved `.invalid` TLD (RFC 6761 — always NXDOMAIN).
|
||||
|
||||
## Code changes included in this step
|
||||
|
||||
Beyond the tests, two code changes were made:
|
||||
|
||||
1. **`main.go` refactor** — extracted `run(args []string, stdout, stderr io.Writer) int`
|
||||
so the CLI is testable in-process. `main()` is now a one-liner:
|
||||
`os.Exit(run(os.Args[1:], os.Stdout, os.Stderr))`. No behaviour change.
|
||||
|
||||
2. **`probe.go` TLS classification fix** — Go's `httptrace` calls
|
||||
`TLSHandshakeDone` with the error on a failed handshake, so `tlsDone` was
|
||||
set even on cert rejection. The previous classifier ("tlsStart set, tlsDone
|
||||
zero") never matched. Fixed by capturing `tlsErr` from the hook and checking
|
||||
it in `classifyErr`.
|
||||
Reference in New Issue
Block a user