Building an Event-Driven Order Processing System with Amazon SQS and AWS Lambda
Learn how to build a resilient order-processing workflow using Amazon SQS and AWS Lambda, with a focus on decoupling, retries, failure handling, and scalable event-driven architecture.
Building an Event-Driven Order Processing System with Amazon SQS and AWS Lambda
As applications grow, processing every operation synchronously can create unnecessary dependencies between services.
Imagine an e-commerce application where an order request needs to trigger several operations:
- Store the order
- Process payment
- Update inventory
- Send a confirmation
- Generate an invoice
If all of these operations happen inside a single synchronous request, a temporary failure in one component can affect the entire workflow.
An event-driven architecture can help separate these responsibilities.
In this article, we'll build a simplified order-processing system using Amazon SQS and AWS Lambda.
Architecture
The basic architecture looks like this:
┌─────────────────┐ │ Client │ └────────┬────────┘ │ v ┌─────────────────┐ │ Order Service │ └────────┬────────┘ │ v ┌─────────────────┐ │ Amazon SQS │ │ Order Queue │ └────────┬────────┘ │ v ┌─────────────────┐ │ AWS Lambda │ │ Order Processor │ └────────┬────────┘ │ ┌────────────┼────────────┐ v v v Database Inventory NotificationThe important idea is that the order service does not need to wait for downstream processing to finish.
1. Why Use a Queue?
Without a queue, the application might look like this:
Client | vOrder API | +----> Payment | +----> Inventory | +----> NotificationIf one downstream service becomes slow or unavailable, the entire request can become slow or fail.
With Amazon SQS:
Client | vOrder API | vSQS Queue | vOrder ProcessorThe producer and consumer are now decoupled.
The producer can place the message into the queue and continue processing without requiring the consumer to be immediately available.
2. Create the SQS Queue
Create a queue named:
order-processing-queueFor a production workload, consider the appropriate queue configuration based on your processing requirements.
Important concepts include:
- Visibility timeout
- Message retention
- Long polling
- Dead-letter queues
- Maximum receives
A queue should be configured based on how long the consumer needs to process a message and how failures should be handled.
3. Define the Order Message
A message placed into the queue could contain:
{ "orderId": "ORD-1001", "customerId": "CUST-501", "amount": 1499.00, "currency": "USD", "items": [ { "productId": "PROD-101", "quantity": 2 } ]}Keeping the event structure well defined makes it easier for consumers to process messages consistently.
4. Create the Lambda Consumer
Create an AWS Lambda function that processes messages from the SQS queue.
A simplified Python example:
import json
def lambda_handler(event, context):
for record in event["Records"]:
message = json.loads(record["body"])
order_id = message["orderId"] customer_id = message["customerId"] amount = message["amount"]
print(f"Processing order {order_id}") print(f"Customer: {customer_id}") print(f"Amount: {amount}")
return { "statusCode": 200 }Lambda can be configured with an SQS event source mapping so that messages from the queue trigger the function.
5. Understand Message Visibility
One important SQS concept is the visibility timeout.
When Lambda receives a message, SQS temporarily hides that message from other consumers.
If processing succeeds, the message is removed from the queue.
If processing fails and the message becomes visible again, it can be processed again.
This provides a mechanism for handling transient failures.
The processing flow can therefore look like:
Message | vLambda receives message | +---- Success ----> Delete message | +---- Failure ----> Message becomes available again
6. Handle Failed Messages
What happens if a message repeatedly fails?
You don't want a permanently problematic message to continuously consume processing capacity.
This is where a dead-letter queue (DLQ) becomes useful.
┌─────────────────┐ │ SQS Queue │ └────────┬────────┘ │ v AWS Lambda │ Processing / \ Success Failure | | v v Delete Retry | v DLQConfigure a maximum receive count so that messages that repeatedly fail are eventually moved to the dead-letter queue.
The DLQ can then be monitored separately for investigation.
7. Make Processing Idempotent
Retries introduce an important engineering challenge.
A message may potentially be processed more than once.
Therefore, the consumer should ideally be idempotent.
For example, suppose an order-processing operation charges a customer.
If the same message is processed twice, the customer should not accidentally be charged twice.
One approach is to maintain a unique transaction or order identifier and check whether the operation has already been completed.
Conceptually:
Receive order | vCheck order ID | +---- Already processed ---> Skip | +---- New order -----------> ProcessIdempotency is one of the most important considerations when designing reliable event-driven applications.
8. Least-Privilege IAM
The Lambda execution role should have only the permissions required to process the queue and interact with the resources it actually needs.
Avoid granting broad permissions such as unrestricted access to every SQS queue or every AWS service.
A production architecture should explicitly define:
- Which queue Lambda can consume from
- Which databases it can access
- Which resources it can modify
- Which logging permissions it requires
This follows the principle of least privilege.
9. Monitor the System
A queue-based architecture should be observable.
Useful metrics include:
- Number of messages available
- Number of messages being processed
- Lambda invocation count
- Lambda errors
- Processing duration
- DLQ message count
A particularly useful signal is the number of messages accumulating in the queue.
If the queue continuously grows, consumers may not be processing messages quickly enough.
You can then investigate whether the issue is related to:
- Increased traffic
- Lambda concurrency
- Slow downstream services
- Application errors
- Resource limits
10. Scaling the Consumer
One advantage of this architecture is that the processing layer can scale independently from the producer.
For example:
SQS | ┌──────────┼──────────┐ v v v Lambda Lambda Lambda | | | v v v Order Order Order Processing Processing ProcessingAs the workload increases, Lambda can process additional batches of messages according to the configured concurrency and event-source settings.
This allows the system to absorb traffic spikes more effectively than a tightly coupled synchronous architecture.
11. When Should You Use This Pattern?
An SQS + Lambda architecture can be useful when:
- Work does not need to complete during the user's request.
- You need to absorb traffic spikes.
- A producer and consumer should be independently scalable.
- Temporary downstream failures are expected.
- You need retry capabilities.
- Background processing is appropriate.
Examples include:
- Order processing
- Image processing
- Email workflows
- Report generation
- Data transformation
- Asynchronous notifications
Conclusion
Amazon SQS and AWS Lambda provide a simple foundation for building decoupled, event-driven applications.
The key ideas are not just about connecting a queue to Lambda. A reliable implementation should also consider:
- Decoupling — separate producers from consumers.
- Retries — handle transient failures.
- Dead-letter queues — isolate repeatedly failing messages.
- Idempotency — safely handle duplicate processing.
- Least privilege — restrict IAM permissions.
- Observability — monitor queue depth and processing failures.
- Scalability — allow consumers to scale independently.
Starting with these principles can help turn a simple asynchronous workflow into a more resilient production architecture.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article