learning-go-for-sre-interview/04-log-topn

04 — Log Crunching: Top-N

Looks like a warm-up. It's actually a trap-density championship.

Trap 1: bufio.Scanner's 64KB guillotine

Scanner refuses tokens above 64KB by default and stops with bufio.ErrTooLong — and if you forget to check sc.Err(), it stops silently, and your report just... misses the tail of the file. Real-world lines that exceed 64KB: JSON-formatted logs, stack traces, requests with fat query strings. The fix:

sc.Buffer(make([]byte, 0, 1024*1024), 1024*1024)

An enormous number of production log processors have shipped with this bug. Checking sc.Err() after the loop is the difference between a tool and a liar.

Trap 2: maps have no order — by design, with a great backstory

Go deliberately randomizes map iteration order (a random seed per map, and the runtime even randomizes the starting bucket) so nobody can accidentally depend on it. To rank by count you must extract into a slice and sort:

sort.Slice(out, func(i, j int) bool { return out[i].Count > out[j].Count })

Since 1.24 maps use a Swiss-table implementation (flat, SIMD-probed, cache friendlier) — fun fact to drop if asked "what's inside a Go map."

Trap 3: strings.Fields vs strings.Split

Split(s, " ") produces empty strings for repeated spaces; Fields splits on any run of whitespace (incl. tabs) and never yields empties. For log parsing, Fields is almost always what you mean.

The senior follow-up: "the file is 500 GB"

Have a ladder ready:

1. This solution already streams — memory is O(distinct endpoints), not O(file). Say that first; many candidates don't realize their answer scales.
2. Cardinality explosion (endpoints contain IDs): normalize paths (/users/123 → /users/:id) or cap the map with count-min sketch / space-saving algorithm — approximate top-K in bounded memory. Naming "count-min sketch" in an SRE interview is a mic-drop; it's what real metrics pipelines do to survive high-cardinality labels.
3. Parallelize: shard by hash(endpoint) across workers, merge partials — map-reduce in one sentence.
4. Only need top-N, not full sort? A min-heap of size N gives O(distinct · log N) instead of O(distinct · log distinct). container/heap exists; nobody remembers its interface; everyone respects you for trying.