Resilient Event Pipelines: AWS Lambda & SQS
Prevent pipeline failures in serverless architectures. Learn how to handle partial batch errors using AWS Lambda, Amazon SQS, and Dead-Letter Queues.
When building serverless architectures on AWS, decoupling workloads using Amazon Simple Queue Service (SQS) and AWS Lambda is one of the most reliable and widely adopted architectural patterns.
However, handling batch-driven messages introduces subtle challenges that can degrade pipeline reliability if not configured properly. When an error occurs while processing a multi-message batch, default behaviors can cause duplicate executions, pipeline stalls, and unnecessary downstream load.
Understanding how SQS and Lambda coordinate batch processing—and implementing partial failure reporting—is essential for running production-grade event-driven systems.
The Default Batch Failure Problem
When you configure an SQS queue as an event source for a Lambda function, Lambda reads messages in batches (up to 10,000 for standard queues, commonly set between 10 and 100 to optimize throughput and cost).
By default, Lambda treats the entire batch as an all-or-nothing unit:
- Invocation: Lambda polls the queue and pulls a batch of 10 records.
- Processing: The function iterates through the records. Messages 1 through 6 succeed, but message 7 throws an unhandled exception or unhandled Promise rejection.
- Execution Failure: The entire Lambda invocation terminates with an error status.
- Visibility Expiration: Because Lambda did not return a successful response, none of the 10 messages are deleted from the queue.
- Redelivery: When the queue's Visibility Timeout expires, all 10 messages become visible again. Messages 1 through 6 are reprocessed, risking duplicate side effects (e.g., duplicate charges or double database writes).
Default Behavior:
[ Msg 1: OK ] ──> [ Msg 2: OK ] ──> [ Msg 3: FAIL ] ──> Entire Batch Fails
│
All 3 messages return to queue ─────────────────────────────┘
(Msg 1 & 2 re-run needlessly)
[ Msg 1: OK ] ──> [ Msg 2: OK ] ──> [ Msg 3: FAIL ] ──> Entire Batch Fails
│
All 3 messages return to queue ─────────────────────────────┘
(Msg 1 & 2 re-run needlessly)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
## Solution 1: Partial Batch Failures via `ReportBatchItemFailures`
Rather than failing the complete batch or manually calling the SQS `DeleteMessage` API for each success, you can configure the event source mapping with **`ReportBatchItemFailures`**.
Under this model:
- The Lambda function returns a list of specific `messageId` values that failed.
- Lambda automatically deletes the successful messages from SQS.
- Only the failed `messageId` records remain in the queue to be retried.
### 1. Enabling via Infrastructure as Code
#### AWS CLI
```bash
aws lambda update-event-source-mapping \
--uuid "<YOUR-EVENT-SOURCE-MAPPING-UUID>" \
--function-response-types "ReportBatchItemFailures"
AWS CloudFormation / SAM
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Resources:
ProcessOrderFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: src/
Handler: app.handler
Runtime: nodejs20.x
Events:
OrderQueueEvent:
Type: SQS
Properties:
Queue: !GetAtt OrderQueue.Arn
BatchSize: 10
FunctionResponseTypes:
- ReportBatchItemFailures
Solution 2: Handler Implementation Patterns
Node.js / TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import { SQSEvent, SQSBatchResponse, SQSBatchItemFailure } from "aws-lambda";
export const handler = async (event: SQSEvent): Promise<SQSBatchResponse> => {
const batchItemFailures: SQSBatchItemFailure[] = [];
for (const record of event.Records) {
try {
await processSingleMessage(record.body);
} catch (error) {
console.error(`Error processing message ${record.messageId}:`, error);
// Capture the messageId so only this record is retried
batchItemFailures.push({ itemIdentifier: record.messageId });
}
}
// Return the structure expected by the Lambda Event Source Mapping
return { batchItemFailures };
};
async function processSingleMessage(body: string): Promise<void> {
const payload = JSON.parse(body);
if (!payload.id) {
throw new Error("Validation Error: Missing entity ID");
}
// Business logic execution
}
Python (using AWS Lambda Powertools)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
from aws_lambda_powertools.utilities.batch import (
BatchProcessor,
EventType,
process_partial_response,
)
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord
processor = BatchProcessor(event_type=EventType.SQS)
def record_handler(record: SQSRecord):
payload = record.json_body
if "id" not in payload:
raise ValueError("Validation Error: Missing entity ID")
# Business logic execution
def lambda_handler(event, context):
return process_partial_response(
event=event,
record_handler=record_handler,
processor=processor,
context=context,
)
Isolating Poison Pills with a Dead-Letter Queue (DLQ)
A poison pill is a record that cannot be processed successfully regardless of how many retries occur (such as a malformed schema or invalid credentials). Without intervention, a poison pill will exhaust system retries and cause unnecessary invocations.
- Create a dedicated DLQ: Set up a secondary SQS queue (e.g.,
OrderQueue-DLQ). - Configure a Redrive Policy: Associate the DLQ with your primary queue.
- Set
maxReceiveCount: Specify how many times a message can be delivered before it moves to the DLQ (typically between 3 and 5).
1
2
3
4
{
"deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:OrderQueue-DLQ",
"maxReceiveCount": 3
}
1
2
3
4
5
[ Primary SQS Queue ] ──(Receive Count > 3)──> [ Dead-Letter Queue (DLQ) ]
│ │
Lambda Invocations CloudWatch Alarm
│
Ops Team / PagerDuty
Production Reliability Checklist
| Configuration | Recommended Setting | Architectural Reason |
|---|---|---|
| Visibility Timeout | ≥ 6x Lambda Timeout | Prevents SQS from making in-flight messages visible to other consumers while Lambda is still running. |
| Idempotency | Keys / Hash Validation | Distributed systems guarantee at-least-once delivery. Ensure consumers handle duplicate messages gracefully via unique transactional keys. |
| DLQ Monitoring | CloudWatch Metric | Set an alarm on ApproximateNumberOfMessagesVisible > 0 on the DLQ to alert the team when unprocessable messages arrive. |
| Max Concurrency | Set on Event Source | Protects downstream databases (like Amazon RDS) from connection exhaustion during traffic bursts. |
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article