Stop Dropping Messages: Decouple Your Architecture with SQS and Lambda
When building APIs or microservices, junior developers often write synchronous workflows: a user submits a form, the backend processes an image, writes to a database, sends a welcome email, and only then responds with 200 OK. The problem? If the email provider is down, the user gets an error. If 500 users sign up at the same instant, the backend crashes from sudden connection exhaustion. The industry-standard solution is asynchronous decoupling using Amazon SQS (Simple Queue Service) and AWS Lambda.
The Architecture Pattern
Instead of doing all heavy lifting inside your user-facing request:
- API / Producer: Receives the request and immediately drops a JSON job message into an SQS Queue.
- Immediate Response: Returns
202 Acceptedto the client in milliseconds. - Consumer (AWS Lambda): Automatically polls the queue in batches, processes jobs in the background, and retries gracefully if a downstream service fails.
Step 1: Create an SQS Queue with a Dead-Letter Queue (DLQ)
A Dead-Letter Queue (DLQ) catches messages that fail multiple times, preventing broken payloads from choking your pipeline.
Run these AWS CLI commands to create both queues:
1
2
3
4
5
6
7
8
9
10
11
12
13
# 1. Create the DLQ first
aws sqs create-queue --queue-name signup-tasks-dlq
# 2. Get the DLQ ARN
DLQ_ARN=$(aws sqs get-queue-attributes \
--queue-name signup-tasks-dlq \
--attribute-names QueueArn \
--query "Attributes.QueueArn" --output text)
# 3. Create the main processing queue pointing to the DLQ after 3 failures
aws sqs create-queue \
--queue-name signup-tasks-queue \
--attributes "{\"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"$DLQ_ARN\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\"}"
Step 2: Push Messages to the Queue (Producer)
In your web server, API route, or lightweight ingress Lambda, send messages to the queue:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import json
import boto3
sqs = boto3.client("sqs", region_name="us-east-1")
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/signup-tasks-queue"
def send_task(user_email: str, plan_type: str):
payload = {
"email": user_email,
"plan": plan_type,
"action": "SEND_WELCOME_KIT"
}
response = sqs.send_message(
QueueUrl=QUEUE_URL,
MessageBody=json.dumps(payload),
MessageAttributes={
"JobType": {"DataType": "String", "StringValue": "UserRegistration"}
}
)
return response["MessageId"]
# Quick execution test
msg_id = send_task("builder@example.com", "ProDeveloper")
print(f"Message published successfully! Message ID: {msg_id}")
Step 3: Process the Queue with Lambda (Consumer)
AWS manages the polling loop for you. When new messages land in SQS, AWS Lambda scales up, fetches a batch, and passes it to your handler function.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import json
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
"""
Processes messages sent from SQS.
Using reportBatchItemFailures ensures that if 1 of 5 items fails,
only that specific item is retried instead of the whole batch.
"""
failed_message_ids = []
for record in event.get("Records", []):
message_id = record["messageId"]
try:
body = json.loads(record["body"])
logger.info(f"Processing task for: {body.get('email')}")
# Simulate work (e.g., generating PDFs, sending emails)
# If an error happens here:
if not body.get("email"):
raise ValueError("Missing email address")
logger.info(f"Successfully finished job {message_id}")
except Exception as err:
logger.error(f"Failed processing message {message_id}: {str(err)}")
# Add failed message ID to report back to SQS
failed_message_ids.append({"itemIdentifier": message_id})
# Return partial batch failure response
return {"batchItemFailures": failed_message_ids}
Why This Matters for Cloud Engineers
- Resilience: If your database or email API experiences an outage, messages sit safely in SQS for up to 14 days without data loss.
- Cost Efficiency: SQS provides 1,000,000 free requests per month, and Lambda charges only for the fractions of a second it runs to process items.
- Traffic Spikes: If 10,000 signups occur during a marketing launch, SQS acts as a buffer—smoothing traffic so your databases never get overwhelmed.
Over to You
Builders: Have you used SQS with Lambda in your architectures? Do you prefer standard queues or FIFO queues for strict ordering? Let's discuss in the comments below!
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article