AWS Builder Center

Builder Center

AWS Serverless Architecture: Building Reliable Applications with Lambda and SQS

AWS Serverless Architecture: Building Reliable Applications with Lambda and SQS

AWS provides serverless services such as AWS Lambda and Amazon SQS to build applications that can process tasks efficiently while remaining reliable and scalable. In this article, we will explore how Lambda and SQS work together, why message queues are important, and how Dead Letter Queues (DLQs) and partial batch processing can make serverless applications more resilient.

Series: Cloud Computing & AWS Learning Journey. (6 articles)

  1. 1
    AWS Serverless Architecture: Building Reliable Applications with Lambda and SQS This article
  2. 2
    Beyond Prompts: Agentic AI and the Rise of Autonomous Digital Workers
  3. 3
    Beyond Chatbots: How Agentic AI and Machine Learning Are Transforming the Future

ntroduction

Imagine you are running an online shopping application.
A customer places an order, and your application needs to:
  • Process the order
  • Update inventory
  • Send a notification
  • Store transaction information
Now imagine that one of these operations suddenly fails.
Should the entire application stop?
A well-designed cloud application should not fail completely because of one problematic request. This is where serverless architecture and event-driven processing become extremely useful.
AWS Lambda and Amazon SQS can work together to create a system where tasks are placed into a queue and processed independently. This approach can improve reliability, scalability, and resource efficiency.

What is AWS Lambda?

AWS Lambda is a serverless compute service that allows developers to run code without managing servers.
Instead of continuously running a server, you upload your function to AWS Lambda, and AWS runs it when an event occurs.
For example:
Customer places an order → Message enters SQS → Lambda processes the order.
Lambda can automatically handle multiple requests, making it suitable for applications where workloads can change over time.

What is Amazon SQS?

Amazon Simple Queue Service (SQS) is a managed message queuing service.
Think of SQS as a waiting line for application tasks.
For example:
1
2
3
4
5
6
7
Application
↓
Amazon SQS
↓
AWS Lambda
↓
Process Task
Instead of sending a task directly to a processing application, the application places the task into SQS. Lambda can then process messages from the queue.
This creates a separation between the application producing the work and the service processing it.

Why Combine SQS with Lambda?

Using SQS and Lambda together provides several advantages.
1. Better reliability
If Lambda temporarily cannot process a message, the message can remain available for retry instead of being immediately lost.
2. Automatic scaling
Lambda can process messages as the workload increases.
3. Loose coupling
The application producing messages does not need to directly communicate with the processing logic.
4. Failure handling
Failed messages can be retried and eventually moved to a Dead Letter Queue.
5. Cost efficiency
With a serverless model, you do not need to keep dedicated servers running continuously.

The Problem with Batch Processing

Suppose Lambda receives a batch containing 10 messages.
1
2
3
4
5
6
7
8
9
10
Message 1 ✓
Message 2 ✓
Message 3 ✓
Message 4 ✓
Message 5 ✗
Message 6 ✓
Message 7 ✓
Message 8 ✓
Message 9 ✓
Message 10 ✓
If the system treats the entire batch as successful or failed, one problematic message could cause successful messages to be processed again.
This may result in:
  • Duplicate processing
  • Additional compute usage
  • Increased cost
  • Unnecessary retries
A better approach is to identify exactly which message failed.

Partial Batch Failure

AWS supports ReportBatchItemFailures, which allows the Lambda function to report only the messages that failed.
The idea is simple:
1
2
3
4
5
6
7
10 messages received
↓
9 messages processed successfully
↓
1 message failed
↓
Only failed message is retried
This prevents successful messages from being unnecessarily processed again.
The source article demonstrates this approach using SQSBatchResponse and a list of failed message IDs.

What is a Dead Letter Queue?

Sometimes a message continues to fail.
For example:
1
2
3
4
Attempt 1 → Failed
Attempt 2 → Failed
Attempt 3 → Failed
Attempt 4 → Failed
Instead of retrying the same message forever, we can move it to a Dead Letter Queue (DLQ).
A DLQ acts like a separate storage area for messages that could not be successfully processed.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
┌──────────────┐
│ Amazon SQS │
└──────┬───────┘
↓
AWS Lambda
↓
┌──────┴───────┐
│ │
Success Failure
│ ↓
│ Retry
│ ↓
│ Retry Limit
│ ↓
│ DLQ
↓
Completed
In the example source architecture, the queue uses a maximum receive count of 3 before a message is moved to the DLQ.

Why DLQs Are Important

A DLQ is more than just a place for failed messages.
It helps developers investigate problems without blocking the rest of the application.
For example, a failed message might contain:
  • Invalid data
  • Missing information
  • An unexpected format
  • A temporary downstream problem
  • A software bug
Developers can inspect the failed message, identify the problem, fix the application, and then reprocess the message.

Reprocessing Failed Messages

One of the useful features of Amazon SQS is the ability to move messages from a DLQ back to the source queue.
The workflow becomes:
1
2
3
4
5
6
7
8
9
10
11
12
13
Failed Message
↓
DLQ
↓
Investigate Problem
↓
Fix Application
↓
SQS Redrive
↓
Original Queue
↓
Lambda
This allows failed messages to be processed again without writing a separate custom recovery script. The source article demonstrates SQS redrive using start-message-move-task.

Infrastructure as Code with Terraform

Creating AWS resources manually can become difficult when an application grows.
Terraform allows developers to define infrastructure using configuration files.
For this type of architecture, Terraform can create:
  • SQS queues
  • Dead Letter Queues
  • IAM roles
  • Lambda functions
  • Event source mappings
The source project separates application code from infrastructure code using src and infra directories.
A simplified project structure looks like this:
1
2
3
4
5
6
7
8
9
10
aws-serverless-project/
│
├── src/
│ └── LambdaProcessor/
│ └── Function.cs
│
└── infra/
├── main.tf
├── variables.tf
└── outputs.tf
This makes the project easier to maintain and reproduce.

Monitoring with CloudWatch

Building a reliable application is not enough. We also need to know what is happening inside it.
Amazon CloudWatch can help developers monitor Lambda executions and investigate failures.
For example, logs can help answer questions such as:
  • Was the message received?
  • Was it processed successfully?
  • Why did processing fail?
  • How many messages failed?
  • Is the same message being retried?
The source project validates the Lambda/SQS workflow by inspecting CloudWatch logs and checking queue attributes with the AWS CLI.

A Real-World Example

Consider an online food-delivery application.
When a customer places an order:
1
2
3
4
5
6
7
8
9
10
11
Customer
↓
Order Application
↓
Amazon SQS
↓
AWS Lambda
↓
Process Order
↓
Update Order Status
If the order cannot be processed because of a temporary problem, SQS can retry the message.
If the message repeatedly fails, it can be moved to the DLQ.
The development team can then investigate the problem and redrive the message after fixing the issue.
This creates a more fault-tolerant application without requiring developers to manage traditional servers.

Key Takeaways

AWS Lambda and Amazon SQS provide a powerful foundation for event-driven serverless applications.
The most important ideas are:
  • Lambda executes application logic without requiring server management.
  • SQS provides reliable message queuing.
  • Partial batch failure reporting prevents successful messages from being unnecessarily retried.
  • DLQs isolate messages that repeatedly fail.
  • SQS redrive allows failed messages to be reprocessed.
  • Terraform can automate infrastructure deployment.
  • CloudWatch helps monitor and troubleshoot the application.

Conclusion

Building a cloud application is not only about making it work when everything goes right. A truly reliable application must also know how to handle failure.
By combining AWS Lambda, Amazon SQS, Dead Letter Queues, CloudWatch, and Terraform, developers can create serverless systems that are scalable, maintainable, and resilient.
The most important lesson is simple:
Don't design your application only for success. Design it to recover from failure.
That mindset is one of the foundations of building production-ready applications on AWS.
# ai-ml# amazon-forecast# aws-api# aws-b2b-data-interchange# aws-batch

Series: Cloud Computing & AWS Learning Journey. (6 articles)

  1. 1
    AWS Serverless Architecture: Building Reliable Applications with Lambda and SQS This article
  2. 2
    Beyond Prompts: Agentic AI and the Rise of Autonomous Digital Workers
  3. 3
    Beyond Chatbots: How Agentic AI and Machine Learning Are Transforming the Future
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article