AWS Builder Center

Builder Center

Building Resilient Event-Driven Systems with Amazon SQS

A practical guide to using Amazon Simple Queue Service (SQS) to decouple application components, handle traffic spikes, and build reliable asynchronous workflows with retries, visibility timeouts, and dead-letter queues.

Modern applications rarely operate as a single synchronous process. A user request may trigger database updates, notifications, analytics, file processing, or calls to external services. Executing all of these operations synchronously can make systems slow and fragile.
Amazon Simple Queue Service (SQS) provides a straightforward way to decouple these workloads and process them asynchronously.
This article demonstrates how SQS can be used to build a resilient background-processing architecture and explains the design decisions that matter in production.

The Problem: Synchronous Processing Doesn't Scale Well

Consider an application that receives an incident event:
1
2
3
4
5
6
7
8
9
10
Client
│
▼
API
│
├── Save incident
├── Run analysis
├── Generate notifications
├── Update metrics
└── Trigger remediation
If every operation happens during the API request, a slow downstream service can increase the overall response time.
A temporary failure can also cause the entire request to fail.
Instead, we can separate ingestion from processing:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌───────────────┐
│ API │
└───────┬───────┘
│
▼
┌───────────────┐
│ SQS Queue │
└───────┬───────┘
│
asynchronous
│
▼
┌───────────────┐
│ Worker │
└───────┬───────┘
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Database Analysis Notification
The API only needs to accept the event and place a message onto the queue.
The worker can process it independently.

Why Amazon SQS?

SQS provides several capabilities that are useful for asynchronous architectures:
  • Decoupling between producers and consumers
  • Automatic message durability
  • Retry handling
  • Visibility timeouts
  • Dead-letter queues
  • Horizontal scaling of consumers
  • Integration with Lambda and other AWS services
The key architectural benefit is that the producer does not need to know when or how the consumer will process the message.

Standard Queue vs FIFO Queue

SQS provides two primary queue types.

Standard Queue

Standard queues are designed for high throughput and provide at-least-once delivery.
This means consumers should be prepared for the possibility that a message can be delivered more than once.
For example:
1
2
3
4
5
{
"event_id": "inc-1024",
"type": "incident.created",
"severity": "high"
}
The consumer should therefore make processing idempotent.
For example:
1
2
3
4
5
6
if event_id already processed:
return

process event

mark event_id as processed

FIFO Queue

FIFO queues are useful when message ordering and exactly-once processing semantics are important to the application.
A common example could be:
1
2
3
4
5
6
7
OrderCreated
↓
PaymentConfirmed
↓
OrderPacked
↓
OrderShipped
When processing order-dependent events, maintaining the correct sequence can be important.
The queue type should therefore be selected based on the application's actual consistency and ordering requirements rather than simply choosing FIFO by default.

Visibility Timeout

One of the most important SQS concepts is the visibility timeout.
When a worker receives a message, SQS temporarily hides that message from other consumers.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Message arrives
│
▼
Worker receives message
│
▼
Message becomes invisible
│
├── Processing succeeds
│ ↓
│ Delete
│
└── Processing fails
↓
Visibility expires
↓
Message available again
This prevents multiple workers from processing the same message simultaneously under normal conditions.
However, the visibility timeout needs to be configured carefully.
If processing normally takes 60 seconds but the visibility timeout is only 20 seconds, another worker could receive the message before the first worker finishes.
A practical rule is to set the timeout above the expected processing duration and extend it for workloads whose duration can vary significantly.

Dead-Letter Queues

Retries are useful when failures are temporary.
They become problematic when a message is permanently invalid.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Message
↓
Processing fails
↓
Retry
↓
Processing fails
↓
Retry
↓
Processing fails
↓
Retry
↓
...
A malformed message could continue consuming processing capacity indefinitely.
A Dead-Letter Queue (DLQ) provides an isolation mechanism.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌──────────────┐
│ Main Queue │
└──────┬───────┘
│
Worker
│
┌───────┴───────┐
│ │
Success Failure
│ │
▼ ▼
Delete Retry
│
Max retries
│
▼
┌─────────┐
│ DLQ │
└─────────┘
Once a message exceeds the configured retry threshold, it can be moved to the DLQ for investigation.
This separates transient failures from persistent failures.

Lambda + SQS

One powerful serverless pattern is connecting SQS directly to AWS Lambda.
1
2
3
4
5
6
7
8
9
10
11
Producer
│
▼
SQS
│
▼
Lambda
│
├── Validate
├── Process
└── Store result
Lambda polls the queue and invokes the function when messages are available.
This eliminates the need to continuously run worker infrastructure.
It also allows the compute layer to scale according to workload.
For example:
1
2
3
4
5
6
7
8
Low traffic
SQS → Lambda

Traffic spike
SQS → Lambda → Lambda → Lambda → Lambda

Traffic decreases
SQS → Lambda
The queue effectively acts as a buffer between incoming traffic and available compute capacity.

Batch Processing

Instead of processing every message individually, consumers can process multiple messages together.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
100 messages
│
▼
SQS
│
▼
Lambda batch
│
├── Message 1
├── Message 2
├── Message 3
└── ...
Batching can reduce invocation overhead and improve throughput for suitable workloads.
However, the failure strategy becomes more important because individual messages within a batch may have different outcomes.

Designing Idempotent Consumers

A reliable queue architecture should assume that duplicate processing can happen.
Consider a payment event:
1
2
3
4
{
"event_id": "payment-8392",
"amount": 500
}
If the worker successfully processes the payment but crashes before deleting the SQS message, the message may be delivered again.
Without idempotency:
1
2
3
4
5
6
7
8
9
₹500 payment
↓
Processed
↓
Worker crashes
↓
Message redelivered
↓
₹500 processed again
With idempotency:
1
2
3
4
5
6
7
8
event_id = payment-8392

Already processed?
│
┌───┴───┐
YES NO
│ │
Ignore Process
An idempotency key stored in DynamoDB or another persistent datastore can prevent duplicate side effects.

Observability Matters

Simply putting messages into a queue does not automatically make the system observable.
Important metrics include:
  • Number of messages available
  • Number of messages being processed
  • Approximate age of the oldest message
  • Number of messages moved to the DLQ
  • Processing failures
  • Lambda errors and duration
A particularly useful signal is message age.
If the age of the oldest message keeps increasing, consumers may not be processing messages quickly enough.
This can indicate:
1
2
3
4
5
6
7
Traffic increase
↓
Queue depth increases
↓
Processing capacity insufficient
↓
Message age increases
This provides an operational signal before users necessarily notice a failure.

Security Considerations

SQS should also follow the principle of least privilege.
A producer generally needs permission to send messages:
1
sqs:SendMessage
A consumer may need permissions such as:
1
2
3
sqs:ReceiveMessage
sqs:DeleteMessage
sqs:GetQueueAttributes
Applications should avoid granting broad SQS permissions when only specific queue operations are required.
Sensitive information should also not be placed into messages unnecessarily.
A message should contain the minimum information required to perform the operation.

A Practical Architecture

A production-oriented serverless workflow could look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
┌─────────────┐
│ API Gateway│
└──────┬──────┘
│
▼
┌─────────────┐
│ Lambda │
│ Producer │
└──────┬──────┘
│
▼
┌─────────────┐
│ SQS Queue │
└──────┬──────┘
│
Event Source
Mapping
│
▼
┌─────────────┐
│ Lambda │
│ Worker │
└──────┬──────┘
│
┌─────────┼─────────┐
▼ ▼ ▼
DynamoDB External CloudWatch
Service

│
Processing
failures
│
▼
┌─────────────┐
│ DLQ │
└─────────────┘
This architecture separates ingestion, processing, storage, and failure handling.

Key Takeaways

Amazon SQS is more than a mechanism for moving messages between services. Used correctly, it becomes an architectural boundary that helps systems absorb traffic spikes, isolate failures, and scale consumers independently.
The most important design principles are:
  1. Decouple producers and consumers.
  2. Design consumers to be idempotent.
  3. Configure visibility timeouts based on real processing behavior.
  4. Use DLQs for messages that repeatedly fail.
  5. Monitor queue depth and message age.
  6. Apply least-privilege IAM permissions.
  7. Choose Standard or FIFO based on actual application requirements.
The result is an architecture where a temporary downstream failure does not necessarily become an immediate user-facing failure.
For event-driven and serverless applications, that separation can significantly improve resilience and operational scalability.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article