AWS Builder Center

Builder Center

Building Event-Driven Apps with AWS SQS, Lambda, and DynamoDB

Build a serverless order-processing system with SQS, Lambda, and DynamoDB. Learn about retries, dead-letter queues, idempotency, and monitoring.

Building an Event-Driven Serverless Application with Amazon SQS, AWS Lambda, and DynamoDB

Introduction

Modern cloud applications often need to process requests without making users wait for every operation to finish. Consider an e-commerce application that receives hundreds of orders in a short period. Processing every order synchronously can make the application slower and harder to scale.
An event-driven architecture offers another approach: accept the request, place a message in a queue, and process it asynchronously.
AWS provides several services that make this architecture possible. Amazon Simple Queue Service (Amazon SQS) stores messages, AWS Lambda processes them without requiring servers, and Amazon DynamoDB provides a managed NoSQL database for application data.
In this tutorial, we will design a small order-processing system using these services. We will explore the architecture, implementation, permissions, retries, dead-letter queues, and reliability considerations.
The goal is not just to connect three AWS services, but to understand how they work together to build a more resilient application.

1. Understanding the Architecture

Our example uses four main components:
  • Amazon SQS: Receives and temporarily stores order messages.
  • AWS Lambda: Consumes messages from the queue and processes orders.
  • Amazon DynamoDB: Stores the resulting order records.
  • Amazon CloudWatch: Provides logs and metrics to help monitor the application.
The processing flow is straightforward:
  1. An application submits an order message to Amazon SQS.
  2. SQS stores the message until it is processed.
  3. An event source mapping polls the queue and invokes the Lambda function with a batch of messages.
  4. Lambda validates each message and writes the order to DynamoDB.
  5. Successfully processed messages are deleted from the queue by the event source mapping.
  6. Messages that repeatedly fail processing can eventually be moved to a dead-letter queue according to the configured redrive policy.
This architecture separates message production from message processing.
If the application receives a temporary spike in traffic, SQS can buffer messages while Lambda processes them at a rate supported by the configured concurrency, downstream capacity, and service limits.
However, a queue does not make a system automatically reliable. Message duplication, partial failures, permissions, and retry behaviour must all be handled deliberately.

2. Create the Required AWS Resources

Before implementing the function, create the following resources in your AWS account:
  1. An SQS standard queue named orders-queue.
  2. A second SQS standard queue named orders-dead-letter-queue.
  3. A DynamoDB table named Orders.
  4. A Lambda function named process-orders.
Use the same AWS Region for these resources unless your architecture specifically requires otherwise.

Configure the dead-letter queue

Open orders-dead-letter-queue and note its ARN.
Next, open the main orders-queue and configure a redrive policy that points to the dead-letter queue. For a demonstration, you could start with a maximum receive count of five and adjust it based on your workload.
The receive count is a configuration choice, not a universal recommendation. A message may be received again after a processing failure, so the appropriate threshold depends on your retry strategy and the nature of the errors.

Configure the DynamoDB table

Create a table named Orders with a partition key called orderId of type String.
For this example, each item represents an order and contains fields such as customerId, amount, and status.
Using a clear partition key makes it possible to retrieve an order directly by its identifier.

3. Implement the Lambda Function

The following Python example processes messages delivered by SQS and stores their data in DynamoDB.
The code validates incoming messages, uses the order ID as the DynamoDB partition key, and handles each record separately.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
import json
import logging
import os
from decimal import Decimal, InvalidOperation

import boto3

logger = logging.getLogger()
logger.setLevel(logging.INFO)

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(os.environ["ORDERS_TABLE"])

def process_order(order):
order_id = order.get("orderId")
customer_id = order.get("customerId")
raw_amount = order.get("amount")

if not isinstance(order_id, str) or not order_id.strip():
raise ValueError("A valid orderId is required")

if not isinstance(customer_id, str) or not customer_id.strip():
raise ValueError("A valid customerId is required")

if isinstance(raw_amount, bool) or raw_amount is None:
raise ValueError("A valid amount is required")

try:
amount = Decimal(str(raw_amount))
except (InvalidOperation, ValueError):
raise ValueError("Amount must be numeric")

if not amount.is_finite() or amount <= 0:
raise ValueError("Amount must be a positive finite number")

table.put_item(
Item={
"orderId": order_id,
"customerId": customer_id,
"amount": amount,
"status": "PROCESSED"
},
ConditionExpression="attribute_not_exists(orderId)"
)

def lambda_handler(event, context):
batch_item_failures = []

for record in event.get("Records", []):
message_id = record["messageId"]

try:
order = json.loads(record["body"])
process_order(order)

logger.info(
"Order processed successfully: %s",
message_id
)

except dynamodb.meta.client.exceptions.ConditionalCheckFailedException:
# The order ID already exists. Treat this as a
# duplicate only if order IDs are guaranteed to be unique.
logger.info(
"Order already exists; skipping duplicate: %s",
message_id
)

except Exception:
logger.exception(
"Failed to process SQS message: %s",
message_id
)
batch_item_failures.append({
"itemIdentifier": message_id
})

return {
"batchItemFailures": batch_item_failures
}

How the code works

The function receives an event containing one or more SQS messages.
For each message, it:
  • Parses the message body as JSON.
  • Validates the order ID, customer ID, and amount.
  • Writes a record to DynamoDB.
  • Records failures and returns the identifiers of unsuccessful messages.
The conditional write prevents a second item with the same order ID from being created. In this example, an existing order is treated as a duplicate and skipped.
This approach assumes that order IDs are globally unique and that an existing record means the same order was already processed. In a real application, you should distinguish a genuine duplicate from a conflicting request that reuses an order ID with different data.
The function also uses SQS partial batch responses. To enable them, configure the Lambda event source mapping with ReportBatchItemFailures. Otherwise, the returned failure list will not provide the intended per-message retry behaviour.

4. Configure Lambda and SQS Integration

After creating the Lambda function, configure its integration with SQS.
  1. Open the Lambda console and select process-orders.
  2. Choose Add trigger.
  3. Select Amazon SQS.
  4. Choose orders-queue.
  5. Configure the batch size and other event source settings.
  6. Enable ReportBatchItemFailures for partial batch failure handling.
  7. Save the trigger.
The Lambda execution role needs permission to consume messages from the queue and write records to DynamoDB.
Typical SQS consumer permissions include:
  • sqs:ReceiveMessage
  • sqs:DeleteMessage
  • sqs:GetQueueAttributes
The role also needs dynamodb:PutItem on the intended Orders table and the required CloudWatch Logs permissions.
Restrict permissions to the relevant resources wherever possible. Avoid granting broad administrative access simply to make a demonstration work.
The event source mapping manages polling and invokes the function as messages become available. You do not need to run a separate server to poll SQS yourself.

5. Test the Application

Open the SQS console, select orders-queue, and send a message with the following body:
1
2
3
4
5
{
"orderId": "ORD-1001",
"customerId": "CUST-501",
"amount": 1499.50
}
Wait for the event source mapping to invoke Lambda.
Then open the DynamoDB console and inspect the Orders table. A successfully processed message should result in an item similar to this:
1
2
3
4
5
6
{
"orderId": "ORD-1001",
"customerId": "CUST-501",
"amount": 1499.50,
"status": "PROCESSED"
}
DynamoDB stores numeric values using its Number type, so the amount is not stored as a string.
You can also test invalid input by sending a message without an order ID. The function should record the failure, and the message should become eligible for retry according to the SQS visibility timeout and event source mapping behaviour.
Observe the Lambda logs and SQS queue metrics to confirm that processing is working as expected.

6. Understand Retries and Dead-Letter Queues

Distributed systems encounter temporary failures. DynamoDB throttling, dependency errors, invalid messages, and unexpected exceptions can interrupt processing.
SQS and Lambda provide mechanisms to handle these failures, but the details matter.

Visibility timeout

When a message is received, SQS hides it from other consumers for the visibility timeout period.
If processing succeeds, the event source mapping deletes the message. If processing fails and the message is not deleted, it can become visible again after the timeout.
The queue's visibility timeout should be configured with the Lambda function's execution time and retry behaviour in mind. AWS recommends setting it to at least six times the function timeout for a Lambda SQS event source mapping, with additional time if a batch window is configured.

Dead-letter queue

A dead-letter queue stores messages that have been received repeatedly without successful processing, according to the source queue's redrive policy.
This allows developers to inspect problematic messages without leaving them in the main processing flow indefinitely.
Before redriving messages, identify the cause of the failure. Otherwise, the same issue may simply occur again.

Partial batch failures

Lambda can receive multiple SQS messages in a single invocation.
Without partial batch responses, a failed batch can lead to successfully processed messages being delivered again. With ReportBatchItemFailures, the function can identify individual messages that failed.
This improves processing efficiency, but it does not eliminate duplicate deliveries.

7. Why Idempotency Matters

Amazon SQS standard queues provide at-least-once delivery. A message may be delivered more than once, even when the application usually processes messages successfully.
For that reason, applications should be designed to handle duplicate processing safely.
In our example, the DynamoDB conditional write prevents an existing order ID from being inserted again. This is a simple form of idempotency for the order-creation operation.
However, a real order-processing workflow may involve more than writing one database item. It might also charge a payment method, send an email, or update inventory.
Repeating those side effects could create serious problems.
A production system should use a deliberate idempotency strategy, such as:
  • Assigning a stable unique identifier to each business operation.
  • Recording processing state with conditional database operations.
  • Making downstream operations idempotent where possible.
  • Tracking external side effects separately when necessary.
  • Handling partial completion and retries explicitly.
The important principle is that retries are useful only when the operation can be repeated safely.

8. Monitor the Application

Amazon CloudWatch helps you understand the behaviour of the queue, Lambda function, and database.
Useful metrics include:
For Amazon SQS
  • ApproximateNumberOfMessagesVisible: Messages waiting to be processed.
  • ApproximateAgeOfOldestMessage: Approximate age of the oldest available message.
  • NumberOfMessagesSent: Messages sent to the queue.
For AWS Lambda
  • Invocations: Number of function invocations.
  • Errors: Number of failed invocations.
  • Duration: Time taken to execute the function.
  • Throttles: Invocation attempts that were throttled.
For Amazon DynamoDB
Monitor relevant read and write activity, throttling, and capacity-related metrics based on the table's configuration.
A growing queue backlog may indicate that incoming traffic exceeds processing capacity. A rising error count may point to invalid input, application bugs, or downstream problems.
Create CloudWatch alarms for meaningful conditions and use logs to investigate the underlying cause rather than relying on a single metric.

9. Production Improvements

This tutorial provides a foundation, but production workloads need additional safeguards.
  • Input validation: Validate the complete message schema before processing.
  • Idempotency: Prevent duplicate business operations and conflicting order IDs.
  • Retry strategy: Distinguish transient errors from permanent validation failures.
  • Security: Use least-privilege IAM roles and encrypt data appropriately.
  • Observability: Add structured logs, useful metrics, and alarms.
  • Capacity management: Configure concurrency and batch sizes according to downstream limits.
  • Data protection: Avoid logging sensitive customer information.
  • Testing: Test duplicate messages, malformed input, timeouts, and partial batch failures.
  • Recovery: Define a process for inspecting and safely redriving dead-letter queue messages.
For high-volume systems, consider how batch size, Lambda concurrency, DynamoDB capacity, and queue retention interact.
Also remember that an SQS queue does not guarantee business-level ordering. If strict ordering is required, evaluate Amazon SQS FIFO queues and their associated processing constraints.

10. Cost Considerations and Cleanup

The services used in this architecture have pricing models based on usage and configuration. Costs can arise from SQS requests, Lambda invocations and execution duration, DynamoDB operations and storage, and CloudWatch logging.
Review the current AWS pricing pages and your account's applicable Free Tier benefits before deploying the system.
When finished experimenting, delete the resources you created if they are no longer needed. Check the Lambda event source mapping, SQS queues, DynamoDB table, and related monitoring resources.
Deleting a queue permanently removes its messages, so inspect any data you want to retain before cleanup.

Conclusion

Amazon SQS, AWS Lambda, and Amazon DynamoDB provide a practical foundation for building event-driven applications without managing servers.
SQS decouples message producers from consumers, Lambda processes messages on demand, and DynamoDB stores application data. Together, these services can help applications handle traffic spikes and isolate processing failures.
However, reliability comes from more than choosing managed services. Correct retry configuration, idempotent operations, least-privilege permissions, monitoring, and a clear recovery strategy are all essential parts of the design.
Discussion: When designing an order-processing system, which would you prioritize first: idempotency, retry handling, or monitoring? Why?
#aws #amazon-sqs #aws-lambda #amazon-dynamodb #serverless
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article