AWS Builder Center

Building Reliable Event-Driven Architectures with Amazon SQS and AWS Lambda

Event-driven architecture is widely used in modern cloud-native applications.

AWS provides services such as Amazon Simple Queue Service (SQS) and AWS Lambda to build asynchronous, scalable, and loosely coupled application architectures.
However, asynchronous processing introduces engineering challenges:
  • Message delivery failures.
  • Duplicate message processing.
  • Retry management.
  • Visibility timeout configuration.
  • Poison messages.
  • Partial batch failures.
  • Downstream service availability.
A production-grade architecture must handle these failures without losing messages or executing business operations multiple times.
In this article, we will explore how to build reliable message-processing workflows using Amazon SQS and AWS Lambda.

1. Understanding the Architecture

Amazon SQS acts as a message broker between application producers and consumers.
AWS Lambda processes messages asynchronously through an SQS event source mapping.

Architecture Flow

Application / Producer
|
v
Amazon SQS
|
v
Lambda Event Source Mapping
|
v
AWS Lambda
|
v
Database / API
|
v
Processing Result

Failed Messages
|
v
SQS DLQ

Component Responsibilities

Amazon SQS
Stores messages until they are successfully processed or expire.
Lambda Event Source Mapping
Polls SQS and invokes Lambda with batches of messages.
AWS Lambda
Executes application processing logic.
Dead-Letter Queue
Stores messages that exceed the configured receive threshold.
Amazon CloudWatch
Provides monitoring and operational visibility.

2. Understanding SQS Visibility Timeout

Visibility Timeout is the period during which a received SQS message becomes temporarily invisible to other consumers.
Consider this scenario:
  1. Lambda receives 10 messages.
  1. SQS hides those messages.
  1. Lambda starts processing the batch.
  1. One message causes an error.
  1. The invocation fails.
  1. Without partial batch responses, all 10 messages can become visible again after the visibility timeout.
This can cause previously successful messages to be processed again.

Production Configuration

AWS recommends configuring:
Visibility Timeout >= 6 × Lambda Timeout
+ Maximum Batching Window
Example:
Lambda Timeout: 30 seconds
Batching Window: 5 seconds

Visibility Timeout:
(6 × 30) + 5 = 185 seconds
Therefore, configure the SQS visibility timeout to at least 185 seconds.
The purpose is to provide sufficient time for processing and retry behavior, including throttling scenarios.

3. Understanding Partial Batch Responses

Partial batch responses solve an important problem in SQS-Lambda integrations.
By default, when a batch fails, successfully processed messages can also be retried.
AWS provides a feature called:
 ReportBatchItemFailures 
When enabled, Lambda can report individual failed messages instead of treating the entire batch as failed.

Example Scenario

Suppose Lambda receives five messages:
Message
Processing Result
MSG-001
SUCCESS
MSG-002
SUCCESS
MSG-003
FAILED
MSG-004
SUCCESS
MSG-005
SUCCESS
Without partial batch responses, all five messages can be retried.
With partial batch responses, only MSG-003 is reported as failed.
This reduces unnecessary processing and improves throughput.

Enable Partial Batch Responses

First, retrieve the event source mapping UUID:
aws lambda list-event-source-mappings \
--function-name order-processor
Then enable partial batch responses:
aws lambda update-event-source-mapping \
--uuid YOUR_MAPPING_UUID \
--function-response-types ReportBatchItemFailures
Replace  YOUR_MAPPING_UUID  with the actual mapping UUID.

Python Lambda Example

import json

def lambda_handler(event, context):

batch_item_failures = []

for record in event["Records"]:

message_id = record["messageId"]

try:
body = json.loads(record["body"])

print(f"Processing message: {message_id}")

process_order(body)

except Exception as error:

print(f"Failed message: {message_id}")
print(str(error))

batch_item_failures.append({
"itemIdentifier": message_id
})

return {
"batchItemFailures": batch_item_failures
}

def process_order(order):

print(f"Order received: {order}")

# Application business logic goes here.
Important: Returning an empty  batchItemFailures  list indicates that all records were processed successfully.
The event source mapping must also have  ReportBatchItemFailures  enabled. If the function throws an unhandled exception, Lambda treats the entire batch as failed.

4. Understanding Dead-Letter Queues (DLQ)

A Dead-Letter Queue isolates messages that repeatedly fail processing.
Consider an invalid payment event that consistently fails because required fields are missing.
Retrying the same invalid message indefinitely wastes processing resources.
A DLQ provides a separate location for investigating these failures.

Example Architecture

Main SQS Queue
|
v
AWS Lambda
|
Processing Error
|
v
Retry Message
|
maxReceiveCount
|
v
SQS DLQ
|
v
Investigation
|
v
Manual Redrive

Configure a Redrive Policy

Example configuration:
{
"deadLetterTargetArn": "arn:aws:sqs:ap-south-1:123456789012:orders-dlq",
"maxReceiveCount": 5
}
This is an illustrative policy. Replace the ARN with your actual DLQ ARN.
AWS recommends setting  maxReceiveCount  to at least 5 as a starting point for Lambda-SQS integrations.
The final value should reflect the application's failure characteristics and recovery requirements.

5. Why Idempotency Matters

SQS and Lambda event source mappings provide at-least-once processing.
This means duplicate message delivery can occur.
Consider a payment-processing workflow:
  1. Lambda receives a payment event.
  1. The payment operation succeeds.
  1. Lambda experiences a timeout before successful completion is acknowledged.
  1. SQS makes the message available again.
  1. Lambda receives the same message.
Without idempotency, the customer might be charged twice.

Implementing Idempotency

A common approach is to maintain a unique event identifier in a database.
Example:
def process_payment(event):

event_id = event["event_id"]

existing = get_processed_event(event_id)

if existing:
print("Duplicate event detected")
return

execute_payment(event)

save_processed_event(event_id)
This is a simplified illustration, not a production-safe transaction implementation.
For payment processing, the idempotency record and business operation must be coordinated atomically wherever possible. A database transaction or conditional write can help prevent concurrent duplicate execution.

6. Handling Poison Messages

A poison message is a message that repeatedly fails processing.
Common causes include:
  • Invalid JSON.
  • Missing required attributes.
  • Unsupported event versions.
  • Business validation failures.
  • Unexpected data types.
  • Incompatible application schemas.
A reliable consumer should distinguish between transient and permanent failures.
Transient Failure
Examples:
  • Database temporarily unavailable.
  • Network timeout.
  • Downstream API throttling.
These failures may succeed on retry.
Permanent Failure
Examples:
  • Invalid message structure.
  • Missing mandatory fields.
  • Unsupported event type.
These failures usually require investigation or correction rather than repeated immediate processing.

7. Monitoring SQS and Lambda with CloudWatch

Monitoring is essential for understanding message-processing reliability.
Important SQS metrics include:
Metric
Purpose
ApproximateNumberOfMessagesVisible
Measures available message backlog
ApproximateNumberOfMessagesNotVisible
Measures messages currently in flight
ApproximateAgeOfOldestMessage
Identifies processing delays
NumberOfMessagesDeleted
Tracks successful message deletions
NumberOfMessagesReceived
Tracks message receive activity
For DLQs, monitor:
 ApproximateNumberOfMessagesVisible 
A growing DLQ backlog should trigger investigation.

Recommended Alarms

  1. DLQ message count greater than zero.
  1. Oldest message age exceeds the processing SLA.
  1. Main queue backlog remains elevated.
  1. Lambda error rate increases.
  1. Lambda throttling increases.
Remember that SQS CloudWatch metrics are approximate and should be interpreted with message lifecycle behavior in mind.

8. Production Best Practices

For production-grade SQS-Lambda integrations:
  1. Configure visibility timeout according to AWS recommendations.
  1. Enable partial batch responses.
  1. Configure a DLQ and appropriate  maxReceiveCount .
  1. Implement idempotent message consumers.
  1. Separate transient and permanent failures.
  1. Select batch size according to processing duration and payload.
  1. Monitor queue backlog and message age.
  1. Apply least-privilege IAM permissions.
  1. Use structured logging with message identifiers.
  1. Test duplicate delivery and failure scenarios.
  1. Monitor Lambda concurrency and throttling.
  1. Establish a controlled DLQ investigation and redrive process.

9. Conclusion

Amazon SQS and AWS Lambda provide powerful building blocks for asynchronous application architectures.
However, simply connecting a queue to a Lambda function does not guarantee reliable message processing.
Production reliability requires proper visibility timeout configuration, partial batch responses, controlled retries, DLQs, idempotency, and monitoring.
The most important engineering principle is:
Design every message consumer with the assumption that a message can be delivered more than once and that any individual message can fail independently.
Building around these assumptions makes event-driven systems more resilient, predictable, and easier to operate.

Discussion Question

Imagine a production Lambda function receives a batch of 10 messages.
Nine messages are processed successfully, but one fails because a downstream database becomes temporarily unavailable.
How would you design the retry mechanism?
Would you use partial batch responses, exponential backoff, a DLQ, or a combination of these approaches?
Share your production experience with AWS serverless architectures.
#AWS #AWSBuilders #AmazonSQS #AWSLambda #ServerlessArchitecture #DevOps #CloudEngineering #EventDrivenArchitecture #CloudWatch #SRE
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article