
Your API Doesn’t Need to Do Everything: An Introduction to Asynchronous Processing with Amazon SQS
Understand how Amazon SQS helps decouple services, handle traffic spikes, and improve application resilience with asynchronous processing. Instagram: [https://www.instagram.com/vithortorelli/] LinkedIn: [https://www.linkedin.com/in/vithor-tech/] AWS SBG UVV: Instagram: [https://www.instagram.com/aws.sbg.uvv/] Meetup: [https://www.meetup.com/aws-sbg-at-university-vila-velha/]
Your API Doesn’t Need to Do Everything: An Introduction to Asynchronous Processing with Amazon SQS
Imagine you are developing an API for an order management system.
When a user confirms a purchase, several actions may need to happen:
- register the order in the database;
- update inventory;
- send a confirmation email;
- generate an invoice or document;
- notify other systems;
- start additional processing.
An initial implementation could simply execute all of these tasks one after another before returning a response to the user.
Something like:
Client ↓API ↓Save order ↓Update inventory ↓Send email ↓Notify another system ↓ResponseFor small applications, this can work perfectly well.
The problem begins when the system grows.
The email service may become slow. An external integration may stop responding. A certain task may start taking several seconds to complete.
Suddenly, an operation that should simply register an order becomes dependent on several other components working correctly.
This is where an important software architecture idea comes into play:
not everything needs to happen during the same request.
And one way to solve this problem on AWS is by using Amazon Simple Queue Service (Amazon SQS).
What is Amazon SQS?
Amazon SQS is a fully managed message queuing service from AWS.
Its main purpose is to allow different components of an application to exchange information without being directly dependent on each other.
AWS describes Amazon SQS as a service that helps decouple and scale microservices, distributed systems, and serverless applications.
Instead of having our API execute all processing directly, it can simply register that a certain piece of work needs to be performed.
For example:
┌──→ Process email │Client → API → SQS ──→ Update integration │ └──→ Process taskThe API produces a message.
Another component consumes that message and performs the work.
It may seem like a small change, but architecturally, it is a major one.
From synchronous to asynchronous
Consider our order API again:
POST /ordersIn a fully synchronous architecture, we could have something like:
1. Validate order2. Save order3. Update inventory4. Send email5. Communicate with external system6. Return HTTP 201Now imagine that the email service takes five seconds to respond.
Or worse: imagine that it is unavailable.
Should our API fail to register the order just because the email could not be sent at that moment?
In many systems, the answer is no.
We can separate what really needs to happen immediately from what can be processed later.
Client ↓API ↓Register order ↓Publish message to SQS ↓HTTP 201Meanwhile:
SQS ↓Consumer ↓Execute processingNow the client does not need to wait for all secondary processing to finish.
The API is responsible for what truly belongs to the request context, while later tasks can happen asynchronously.
Producer, Queue, and Consumer
This model usually involves three main elements.
Producer
The Producer is the component that sends the message.
In our example, it could be the API responsible for orders.
It could publish something conceptually similar to:
{ "event": "ORDER_CREATED", "orderId": "92831", "customerId": "501", "createdAt": "2026-08-31T18:30:00Z"}Notice that we are not necessarily sending an instruction such as:
"send an email"We can simply say:
"an order was created"This allows other components to decide what to do based on that event.
Queue
The queue acts as an intermediary between whoever produces the message and whoever processes it.
The Producer does not necessarily need to know the Consumer.
It only needs to be able to publish the message.
Consumer
The Consumer is the component responsible for retrieving or receiving that message and executing the required processing.
For example:
Amazon SQS ↓AWS Lambda ↓Order processingAWS Lambda can use an SQS queue as an event source. In this model, Lambda polls the queue and invokes the function with batches of available messages.
This is a very useful pattern in event-driven architectures.
Why Put a Queue in the Middle?
At first, it may look like we are simply adding another service to the architecture.
But the queue solves several interesting problems.
1. Decoupling
Without a queue:
API → Service BThe API needs Service B to be available at that exact moment.
With a queue:
API → SQS → Service BThe API can publish the work even if the Consumer is not processing it immediately.
This reduces the temporal dependency between components.
The Producer and the Consumer do not necessarily need to be available at exactly the same time.
2. Absorbing Traffic Spikes
Imagine that your system normally receives:
100 orders/minuteBut during a promotion, it suddenly jumps to:
10,000 orders/minuteWithout an intermediate layer, all those orders may hit the processing services directly.
With a queue, we have something like:
spike in messages ↓API → SQS → → → → → → → → Consumer queue accumulatingThe Consumer can process the messages according to its own capacity.
In this scenario, the queue also acts as a kind of buffer between the rate of production and the rate of consumption.
SQS Standard queues are designed to provide very high throughput and scale to large volumes of messages.
3. Handling Failures
Now imagine that our Consumer receives a message but fails during processing.
When a message is received from SQS, it is not necessarily removed immediately.
Instead, it becomes temporarily invisible to other consumers for a period known as the visibility timeout.
If processing completes successfully, the message can be deleted.
If the Consumer fails and the message is not deleted, it can become available again after the visibility timeout expires.
We can visualize it like this:
Message available ↓Consumer receives it ↓Message becomes invisible ↓ Processing / \ success failure ↓ ↓delete timeout ↓ message becomes available againThis behavior significantly improves the resilience of the system.
But it also leads us to an extremely important concept.
Your Application Needs to Be Idempotent
SQS Standard queues use an at-least-once delivery model.
This means that a message will be delivered at least once, but in some situations it may be delivered again.
AWS recommends that consumer applications be designed to be idempotent in order to support this behavior.
Imagine this message:
{ "orderId": "92831", "action": "CHARGE_PAYMENT"}If it is processed twice and our application simply charges the card again every time, we have a serious problem.
A robust Consumer needs to be able to recognize that:
order 92831 has already been processedand prevent duplicate side effects.
This is why working with messaging does not simply mean “adding a queue.”
The application architecture needs to take the message delivery model into account.
What If a Message Keeps Failing?
Imagine a problematic message:
SQS ↓Consumer ↓ERROR ↓SQS again ↓Consumer ↓ERROR ↓...We do not want it to keep being processed forever.
For this, we can use a Dead-Letter Queue, commonly called a DLQ.
After a message exceeds a configured number of attempts, it can be sent to another queue.
success ↓SQS → Consumer → processing ↑ │ failure │ └── retry
After several failures:
SQS → Dead-Letter QueueNow that message can be analyzed separately without blocking or polluting the normal processing flow.
AWS recommends using DLQs to capture messages that cannot be successfully processed after multiple attempts.
This also creates opportunities for monitoring.
For example:
DLQ receives message ↓Amazon CloudWatch ↓Alert ↓Team investigates
SQS Standard or FIFO?
Amazon SQS offers two main types of queues:
- Standard
- FIFO — First-In-First-Out
Standard queues provide very high throughput and use at-least-once delivery. In addition, message ordering is maintained on a best-effort basis, which means messages may occasionally be delivered out of their original sequence.
FIFO queues are useful when processing order matters.
Imagine operations related to the same account:
1. balance = 1002. deposit 503. withdraw 20Depending on the domain, processing these events in the wrong order could create undesirable behavior.
In situations like this, ordering requirements need to be part of the architectural decision.
Therefore, there is no queue that is simply “better.”
There is the queue that best matches the characteristics of the problem.
An Example with AWS Lambda
A simple serverless architecture could look like this:
┌──────────────┐ │ Client │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ API │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ Amazon SQS │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ AWS Lambda │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ Processing │ └──────────────┘The flow would be:
1. The client sends a request.2. The API validates and registers the required operation.3. A message is sent to SQS.4. The API responds to the client.5. Lambda receives messages from the queue.6. Processing happens in the background.7. After success, the message is removed.And what happens if processing fails?
Message ↓Lambda ↓Error ↓Message becomes available again ↓New attemptAfter repeated failures:
Message ↓DLQWe now have a relatively simple architecture that is much better prepared for failures and variations in load.
But Asynchronous Processing Also Has Architectural Costs
Messaging is extremely useful.
But adding SQS does not automatically make an architecture better.
When we move from:
A → Bto:
A → Queue → Bwe also introduce new concerns:
- duplicate messages;
- out-of-order processing;
- retries;
- observability;
- queue monitoring;
- invalid message handling;
- eventual consistency;
- tracing across services;
- visibility timeout configuration;
- DLQ management.
In synchronous systems, we often have:
request → resultIn asynchronous systems, we may have:
request → work accepted → future processing → later resultThis change needs to make sense for the application domain.
When Would I Consider Using SQS?
Some common scenarios include:
Image Processing
Upload → SQS → Resize/CompressThe user does not need to wait for the entire image processing task to finish.
Email Sending
Registration → SQS → Email serviceA temporary email service outage does not need to prevent the account from being created.
Order Processing
Order → SQS → Secondary processingActions that do not need to be part of the HTTP response can happen later.
External Integrations
System A → SQS → Integration → System BInstability in System B does not necessarily need to bring down the main flow of System A.
Heavy Tasks
API → SQS → WorkersTasks can be distributed among multiple Consumers, and processing capacity can scale according to demand.
The Real Role of a Queue
When we first learn about Amazon SQS, it is easy to think:
“It is just a place where messages wait.”
But that definition leaves out the most interesting part.
A queue creates a boundary between components.
Before:
Service A depends directly on Service B.After:
Service A declares that some work needs to happen.
The queue preserves that information.
A Consumer performs the work when it can.This change reduces temporal coupling and allows each part of the application to evolve and scale more independently.
And perhaps this is one of the most important ideas when studying distributed architectures:
not every task needs to happen now, and not every component needs to directly know who will perform that task.
Amazon SQS is one of the tools that makes this separation possible within AWS.
Next Steps
After understanding the role of a queue, some interesting experiments would be:
Amazon SQS + AWS LambdaThen:
SQS + Lambda + Dead-Letter QueueAnd from there, explore even more interesting architectural questions:
When should you use SQS?When should you use Amazon SNS?When should you use Amazon EventBridge?When is synchronous processing still the better choice?Because learning cloud should not simply be about memorizing services.
What really matters is understanding which problem each service solves and which trade-offs appear when we decide to use it.
And that is exactly what turns AWS knowledge into architecture knowledge.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article