AWS Fanout Architecture
Consider this scenario, A user uploads a profile picture to your app. That one action triggers multiple things that need to happen.
You need to:
Generate thumbnails in different sizes.
Scan it for viruses.
You need to update the user's record in the database.
Maybe you want to send it to your ML pipeline for face detection.
Perhaps you log it for analytics.
One upload. Five different processes need to act on it. Each does something completely different with the same image.
That's fanout.
But how to implement it?
It all starts with Simple Notification Serive(SNS) which acts as the announcement system.
When the image lands in S3, SNS publishes one message saying "new image uploaded, here's the location." That message fans out to multiple SQS queues. One queue for thumbnails. One for content moderation. One for database updates. Each Lambda polls its own queue and does its specific job.
"Okay but why not just have SNS trigger the Lambdas directly?"
Because queues give you breathing room. Lambda can only handle so many concurrent executions. If you get a sudden spike, a hundred users uploading profile pictures at once, SNS will try to invoke hundreds of Lambdas simultaneously.
Some will fail. Some will throttle. You lose processing tasks.
With SQS in between, those hundred messages sit safely in the queue. Your Lambdas poll the queue and process images as fast as they can handle. Nothing gets lost. It just waits its turn.
The queue is your buffer. Your safety net.
"So each Lambda gets its own queue?"
Exactly.
Let's say you have three different processes. One Lambda generates thumbnails. Another runs content moderation through an ML model. A third updates your user database with the new image URL.
SNS publishes the image event once. It fans out to three SQS queues. Each queue feeds its own Lambda. Each Lambda does one job well.
They work independently. If the thumbnail Lambda has a bug and crashes, your database still updates. Content moderation still runs. One failure doesn't cascade.
That's the power of decoupling.
"What if I need to add a new process later?"
You just subscribe a new SQS queue to the existing SNS topic. Deploy a new Lambda to poll it. Maybe you want to add image format conversion, or start extracting EXIF data. The rest of your system doesn't even notice. You didn't change anything that was already working.
This is how you scale architecture without breaking things.
One announcement. Multiple listeners. Each doing their own work. Each moving at their own speed. All independent.
That's fanout.
And once you see it this way, you start seeing it everywhere. Because this pattern solves one of the most common problems in distributed systems.
How do you let multiple things react to the same event without coupling them together?
You fan out.