Fan Out, Fan In: The Pattern You've Been Using Without Naming

Ask a room of application developers to name the pattern where you split a big job into independent chunks, process them in parallel, and merge the results, and you’ll get a pause, then “…MapReduce?” Close. Ask a room of data engineers and they won’t pause, because they did it twice before lunch — they just called it “the backfill script”.

The pattern is fan out, fan in, and it is the load-bearing shape of data engineering. It’s also older than all of us (scatter-gather, divide-and-conquer, fork-join — same animal, different collars). This post gives it a proper introduction: what it is, where it earns its keep, and the half of the pattern everyone underestimates.

The shape, in three moves

the bottleneck · one worker, all the work

The bottleneck

One worker, one loop, eleven million rows. It works perfectly and finishes eventually — where "eventually" is a number that grows with the business and doesn't care about your SLA. Every fan-out story starts as a single arrow that got too heavy.

Fan out

Split the work along a partition key — date, customer segment, file, region — into chunks that don't need to talk to each other. That independence clause is the entire entry fee. If your chunks share state or care about order, you don't have a fan-out; you have a queue wearing a disguise.

Fan in

Wait for all the workers, then merge. That wait is called a barrier — the whole job now finishes when the slowest chunk does, and the merge must cope with duplicates, retries, and partial failure. This is the half of the pattern that pages you at 2am, so we'll give it its own section.

Where you already run it

The pattern is platform-agnostic to the point of being platform-invisible. The same shape, wearing different name badges:

You call it	The fan out	The fan in
A backfill	one task per day/month	“all partitions present?” check
Warehouse loading	one load per file/table	the reconciliation query
dbt / transformation DAGs	independent upstream models	the model that joins them
Enrichment at scale	one API call per record batch	merge + dead-letter the failures
Orchestrator task groups	dynamic task mapping	the join/sensor task
Serverless workflows	one function per chunk	the aggregator the framework sells you

Two honest observations from that table. First: you don’t choose whether to use fan-out/fan-in; you only choose whether to do it deliberately. Second: every row’s fan-in column is a euphemism for “the part we improvised”.

The economics: why it works, and when it stops

Sequential 840m

8 workers 130m

32 workers 85m

A composite backfill, fictionalised from several real ones. The win is real; so is the ceiling.

Fan out and the clock time collapses — that’s the demo that sells the pattern. But watch what happens as you keep adding workers:

The curve flattens because parallelism only buys back the divisible work. Two things set the floor: skew — if one customer is 40% of your data, their partition finishes last no matter how many workers you rent — and the fixed cost of the fan-in itself. Past the ceiling, more workers just means paying more for the privilege of waiting on the same straggler. The fixes are unglamorous and effective: pick a key that distributes evenly, split hot partitions again (fan out inside the big chunk), and put your monitoring on the p99 chunk, not the average.

Fan-in: the hard half

The fan-out is a for-loop. The fan-in is a distributed-systems problem. Three questions decide whether yours is sound, and none of them involve a vendor:

How do you know you’re done? Counting “workers that reported success” fails the moment a worker dies silently. Strong fan-ins track expected chunks — a manifest written at split time — and compare against delivered chunks. Completeness is data, not vibes.
What happens when chunk 47 of 200 fails? If the answer is “rerun everything”, your merge isn’t idempotent and your 40-minute job has a 14-hour failure mode. Retries must target the chunk, and replaying a chunk twice must change nothing.
Is the result invisible until it’s whole? Downstream consumers must never see 199/200ths of a dataset. Stage the output, then commit atomically — a partition swap, a pointer flip, a rename. The fan-in isn’t finished when the data arrives; it’s finished when the data is safe to believe.

# The whole pattern in one breath — and the barrier
# that makes it honest. Language is incidental;
# the manifest is not.
chunks = split(job, by=partition_key)        # fan out
manifest.expect([c.id for c in chunks])      # promises made

results = run_parallel(process, chunks)      # the easy half

missing = manifest.missing(results)          # promises kept?
if missing:
    raise Incomplete(missing)                # loudly, precisely

commit_atomically(merge(results))            # fan in — all or nothing

The coaching note

When I teach this to a team, I don’t start with the diagram — I start by asking them to find the fan-out/fan-in they already own. There’s always one, usually improvised, usually with a fan-in held together by a sleep(300) and optimism. Then we ask the three questions above of it, in a design review, together.

Naming the pattern is the actual gift. Once an engineer can say “this is a fan-in problem”, they stop debugging a script and start applying twenty years of prior art — manifests, barriers, atomic commits. That’s what patterns are for: they let you borrow scars instead of earning all of them yourself.

The sleep(300) was real. It worked for two years, which is the most frightening sentence in this post.