kacerrandClaude Fable 5
7219eb5a23
Includes solved 02-worker-pool and 09-safe-counter (mutex, atomic, sharded). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GbnJj9ZqEx9nUvzkjdwfMq
55 lines
2.3 KiB
Markdown
55 lines
2.3 KiB
Markdown
# 04 — Log Crunching: Top-N
|
|
|
|
Looks like a warm-up. It's actually a trap-density championship.
|
|
|
|
## Trap 1: `bufio.Scanner`'s 64KB guillotine
|
|
|
|
`Scanner` refuses tokens above 64KB by default and stops with
|
|
`bufio.ErrTooLong` — and if you forget to check `sc.Err()`, it stops
|
|
*silently*, and your report just... misses the tail of the file. Real-world
|
|
lines that exceed 64KB: JSON-formatted logs, stack traces, requests with fat
|
|
query strings. The fix:
|
|
|
|
```go
|
|
sc.Buffer(make([]byte, 0, 1024*1024), 1024*1024)
|
|
```
|
|
|
|
An enormous number of production log processors have shipped with this bug.
|
|
Checking `sc.Err()` after the loop is the difference between a tool and a liar.
|
|
|
|
## Trap 2: maps have no order — by design, with a great backstory
|
|
|
|
Go **deliberately randomizes** map iteration order (a random seed per map, and
|
|
the runtime even randomizes the starting bucket) so nobody can accidentally
|
|
depend on it. To rank by count you must extract into a slice and sort:
|
|
|
|
```go
|
|
sort.Slice(out, func(i, j int) bool { return out[i].Count > out[j].Count })
|
|
```
|
|
|
|
Since 1.24 maps use a Swiss-table implementation (flat, SIMD-probed, cache
|
|
friendlier) — fun fact to drop if asked "what's inside a Go map."
|
|
|
|
## Trap 3: `strings.Fields` vs `strings.Split`
|
|
|
|
`Split(s, " ")` produces empty strings for repeated spaces; `Fields` splits on
|
|
any run of whitespace (incl. tabs) and never yields empties. For log parsing,
|
|
`Fields` is almost always what you mean.
|
|
|
|
## The senior follow-up: "the file is 500 GB"
|
|
|
|
Have a ladder ready:
|
|
|
|
1. **This solution already streams** — memory is O(distinct endpoints), not
|
|
O(file). Say that first; many candidates don't realize their answer scales.
|
|
2. **Cardinality explosion** (endpoints contain IDs): normalize paths
|
|
(`/users/123` → `/users/:id`) or cap the map with count-min sketch /
|
|
space-saving algorithm — approximate top-K in bounded memory. Naming
|
|
"count-min sketch" in an SRE interview is a mic-drop; it's what real
|
|
metrics pipelines do to survive high-cardinality labels.
|
|
3. **Parallelize**: shard by hash(endpoint) across workers, merge partials —
|
|
map-reduce in one sentence.
|
|
4. **Only need top-N, not full sort?** A min-heap of size N gives
|
|
O(distinct · log N) instead of O(distinct · log distinct). `container/heap`
|
|
exists; nobody remembers its interface; everyone respects you for trying.
|