learning-go-for-sre-interview/02-worker-pool

02 — Worker Pool

The fundamental fan-out/fan-in shape. Everything else (pipelines, errgroup, batch processors) is a decoration of this.

The channel-closing protocol

The whole exercise is really about one question: who closes what, and when?

Rules worth engraving:

1. Only the sender closes. A receiver closing a channel = eventual panic on the sender side (send on closed channel).
2. Close is a broadcast. Every range and every v, ok := <-ch across all receivers unblocks. That's why close(jobs) cleanly retires all workers.
3. With N senders, no single sender may close. That's the "closer goroutine" pattern you need here:

go func() {
    wg.Wait()
    close(out)
}()

All N workers send to out; a dedicated goroutine waits for all of them and then closes. This is the only race-free way to close a multi-producer channel.

4. You don't have to close channels at all. Channels are GC'd like anything else. Closing is signaling, not resource cleanup. You close out here only because the consumer ranges over it.

The subtle leak in the naive version

If you forget close(out), the consumer's range blocks forever — the test catches this with a timeout. In prod this presents as goroutine count creeping up one per request, which you'd see in go_goroutines on your dashboard long before OOM. Well... ideally you would.

Why unbuffered out is fine (and interesting)

With out unbuffered, workers block until the consumer reads → natural backpressure, memory bounded by workers. Buffering out decouples throughput spikes but the buffer is a queue, and queues hide latency — a very SRE trade-off to mention out loud. (Little's law applies to your channels too.)

Interview follow-ups to prep

• "Add context cancellation" → every out <- fn(j) becomes a select with ctx.Done(), and job consumption too.
• "A job can fail" → either a Result{V, Err} envelope type, or switch to errgroup and talk about fail-fast vs collect-all-errors semantics.
• "Jobs arrive faster than they're processed" → bounded jobs channel = backpressure at the producer; discuss shedding vs queueing.