The Lambda Worked. The Queue Kept Replaying Messages.

A practical AWS SQS and Lambda guide to visibility timeouts, batch failures, partial batch responses, poison messages, dead-letter queues, FIFO ordering, idempotent consumers, and production testing.

Ashish kumar jha

Software Devloper

Published Oct 6, 2026

Last modified Oct 6, 2026

The Lambda Worked. The Queue Kept Replaying Messages.

Your Lambda function processed the message successfully.

The database update completed.

Then the same message appeared again.

A few minutes later, the application processed it a second time.

This is one of the most misunderstood behaviors in event-driven AWS architectures.

Developers often think:

"The Lambda succeeded, so the message is done."

But SQS-based processing is a distributed system.

There is a gap between:

1
2
3
4
5
6
7
Message received
      |
      v
Application side effect completed
      |
      v
Message successfully acknowledged/deleted

A failure can happen inside that gap.

The consequence is simple:

A message may be processed more than once.

The solution is not to assume duplicates never happen.

The solution is to make the system safe when they happen.

This article walks through a production-oriented SQS + Lambda design, including visibility timeouts, batch processing, partial batch responses, dead-letter queues, FIFO behavior, idempotent consumers, and failure testing.

The Incident

Imagine:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
                    +-------------+
                    | API / Event |
                    +------+------+
                           |
                           v
                     +-----------+
                     |    SQS    |
                     +-----+-----+
                           |
                           v
                    +-------------+
                    |   Lambda    |
                    +------+------+ 
                           |
                           v
                     +-----------+
                     |    RDS    |
                     +-----------+

A message arrives:

1
2
3
4
{
  "orderId": "ORD-1001",
  "type": "PAYMENT_CAPTURED"
}

Lambda receives it and updates the database:

1
2
3
4
5
6
7
8
9
10
SQS message
    |
    v
Lambda
    |
    v
RDS update
    |
    v
SUCCESS

But now imagine the Lambda completes the database write and something fails before the queue processing flow completes.

The message becomes visible again.

Lambda receives it again.

Now:

1
2
3
4
5
PAYMENT_CAPTURED
        |
        +--> update payment
        |
        +--> update payment AGAIN

If the consumer is not idempotent, the duplicate can become a business incident.

SQS Is Not "Run This Exactly Once"

For standard SQS queues, delivery is at least once.

That means duplicate delivery is possible.

The consumer must therefore tolerate:

1
2
3
4
same message
same event
same business operation
more than once

AWS explicitly recommends making Lambda code idempotent when processing SQS events because duplicate processing can occur. citeturn522517search3

This is the correct mental model:

1
2
3
4
5
6
7
8
9
10
Producer
   |
   v
SQS
   |
   +--> Consumer attempt #1
   |
   +--> Consumer attempt #2
   |
   +--> Consumer attempt #3

The message is not the guarantee.

Your consumer's correctness is the guarantee.

Visibility Timeout Is Not a Processing Guarantee

When Lambda receives an SQS message, the message becomes temporarily invisible to other consumers.

That period is the visibility timeout.

Conceptually:

1
2
3
4
5
6
7
8
9
10
11
12
Message received
      |
      v
Hidden from normal receives
      |
      +------ processing ------+
      |                        |
      v                        v
Delete                   Visibility timeout
                              |
                              v
                         Message visible

If processing finishes and the message is deleted, everything is fine.

If processing takes too long, the message can become visible again while the original invocation is still working.

That can create concurrent duplicate processing.

A Dangerous Example

Suppose:

1
2
Lambda timeout = 30 sec
SQS visibility timeout = 20 sec

The function starts processing.

At 20 seconds:

1
Visibility timeout expires

The message can become available again.

At 25 seconds:

1
2
Original Lambda still processing
Second Lambda can receive the message

Now:

1
2
3
4
5
6
7
8
9
10
11
            Message
               |
        +------+------+
        |             |
        v             v
   Lambda #1      Lambda #2
        |             |
        +------+------+
               |
               v
        Duplicate side effect

That is why visibility timeout must be designed around real processing behavior.

For Lambda event source mappings, AWS currently recommends configuring the SQS visibility timeout to at least six times the Lambda function timeout, with additional time when a batch window is used. citeturn522517search0

Example Visibility Timeout Calculation

Suppose:

1
2
Lambda timeout = 30 seconds
Maximum batching window = 5 seconds

A conservative starting point from the AWS guidance is:

1
2
6 × 30 + 5
= 185 seconds

The exact setting should still be validated against the workload.

The principle is more important than the arithmetic:

Give the event source enough time for the Lambda invocation and retry behavior without making failures invisible for an unnecessarily long time.

The Batch Problem

Lambda can receive multiple SQS messages in one invocation.

For example:

1
2
3
4
5
6
7
8
Batch
+-----------------------+
| Message A             |
| Message B             |
| Message C             |
| Message D             |
| Message E             |
+-----------------------+

Now suppose:

1
2
3
4
5
A -> success
B -> success
C -> success
D -> failure
E -> success

By default, a failed batch can cause the batch's messages to become visible again.

That means:

1
2
3
4
A -> processed again
B -> processed again
C -> processed again
E -> processed again

Even though only D failed.

AWS provides partial batch responses to avoid this unnecessary reprocessing. citeturn522517search1

Partial Batch Response

Enable:

1
ReportBatchItemFailures

Then your Lambda reports only the failed message IDs.

Conceptually:

1
2
3
4
5
6
7
8
Batch
A B C D E
| | | | |
S S S F S

Return:

D = failed

Lambda can then make only the failed message visible again.

The result is:

1
2
3
4
5
A -> no retry
B -> no retry
C -> no retry
D -> retry
E -> no retry

This is especially useful for large batches.

Java Example

A simplified Java Lambda handler can return failed message IDs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
public SQSBatchResponse handleRequest(
        SQSEvent event,
        Context context) {

    List<SQSBatchResponse.BatchItemFailure> failures =
            new ArrayList<>();

    for (SQSEvent.SQSMessage message : event.getRecords()) {

        try {
            process(message);

        } catch (Exception ex) {
            failures.add(
                new SQSBatchResponse.BatchItemFailure(
                    message.getMessageId()
                )
            );
        }
    }

    return new SQSBatchResponse(failures);
}

The important behavior is not the class name.

It is the contract:

1
2
successful messages -> do not retry
failed messages     -> retry

For FIFO queues, AWS recommends stopping after the first failure and returning the failed and unprocessed messages so ordering is preserved. citeturn522517search1

The Poison Message Problem

Some messages are temporarily failing.

Others are permanently broken.

For example:

1
2
3
4
{
  "orderId": null,
  "amount": -500
}

If the consumer always fails this message:

1
2
3
4
5
6
Attempt 1 -> fail
Attempt 2 -> fail
Attempt 3 -> fail
Attempt 4 -> fail
Attempt 5 -> fail
...

The message keeps returning.

This is a poison message.

Without a dead-letter strategy, the system can waste processing capacity repeatedly trying the same impossible work.

AWS recommends using an SQS dead-letter queue for messages that repeatedly fail processing. citeturn522517search0turn522517search5

Dead-Letter Queue

The architecture becomes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
                 +-------------+
                 | Source SQS  |
                 +------+------+
                        |
                        v
                    +-------+
                    |Lambda |
                    +---+---+
                        |
                success | failure
                        |
                        v
                 +-------------+
                 |     DLQ     |
                 +-------------+

A message might follow:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Attempt 1
   |
   v
Attempt 2
   |
   v
Attempt 3
   |
   v
Attempt 4
   |
   v
Attempt 5
   |
   v
DLQ

The exact maxReceiveCount should match the workload, but AWS currently recommends at least 5 for an SQS queue used with Lambda. citeturn522517search0

Why the DLQ Matters More Than It Looks

A DLQ is not just an error bucket.

It is a control mechanism.

Without a DLQ:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Bad message
   |
   v
Retry
   |
   v
Bad message
   |
   v
Retry
   |
   v
More compute
   |
   v
More retries

A growing population of permanently failing messages can consume capacity and make the overall system less healthy.

AWS calls this the snowball anti-pattern in its SQS/Lambda guidance. citeturn522517search5

A DLQ breaks that loop.

But a DLQ Does Not Fix the Business Problem

Putting messages in a DLQ is not the end of the process.

You need an operational workflow:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Source Queue
    |
    v
Lambda
    |
 failure
    v
DLQ
    |
    v
Investigation
    |
    +---- fix data
    |
    +---- fix code
    |
    +---- replay safely

A useful DLQ strategy should answer:

1
2
3
4
5
6
Who monitors it?
How is an incident detected?
How do we inspect the failed message?
How do we decide whether to replay it?
How do we prevent another duplicate?
How do we track replay status?

The Consumer Must Still Be Idempotent

A DLQ reduces repeated failures.

It does not eliminate duplicate delivery.

The consumer should still be safe when it sees the same event twice.

Suppose the message is:

1
2
3
4
5
{
  "eventId": "EVT-123",
  "type": "ORDER_CREATED",
  "orderId": "ORD-1001"
}

A simple database pattern is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
eventId
   |
   v
Already processed?
   |
 +---+---+
 |       |
YES     NO
 |       |
 v       v
skip   process
        |
        v
   mark processed

The uniqueness constraint belongs close to the state change.

For example:

1
2
CREATE UNIQUE INDEX idx_processed_events_event_id
ON processed_events(event_id);

Then concurrent consumers cannot silently create the same business effect.

SQS FIFO Is Not a Universal Duplicate Solution

FIFO queues provide:

ordered processing within message groups
deduplication behavior
stricter delivery semantics than standard queues

But FIFO does not mean:

1
Your business operation can never execute twice.

For example, consumer failure can still cause a message to be delivered again after visibility timeout.

AWS documents that even FIFO consumers must account for visibility timeout and possible duplicate processing. citeturn522517search6

Also, SQS FIFO producer deduplication is time-sensitive.

AWS documents a 5-minute deduplication interval. A retry after that interval can result in a new message. citeturn522517search7

Therefore:

1
2
3
FIFO
  +
idempotent consumer

is much stronger than:

1
2
FIFO
  alone

Ordering Has a Cost

FIFO queues are useful when ordering is a business requirement.

For example:

1
2
3
4
5
6
7
Account A

Deposit
   |
Withdrawal
   |
Transfer

The order matters.

With SQS FIFO:

1
MessageGroupId = ACCOUNT-A

The messages for that group can be processed in order.

But you should not use one single message group for the entire system unless you actually need global serialization.

Instead, partition by a business key when appropriate:

1
2
3
ACCOUNT-A -> group A
ACCOUNT-B -> group B
ACCOUNT-C -> group C

Now independent groups can be processed in parallel while maintaining order inside each group.

A Reliable Architecture

A production-oriented design often looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
                         +----------------+
                         | Producer/API   |
                         +-------+--------+
                                 |
                                 v
                         +---------------+
                         |    SQS        |
                         | standard/FIFO |
                         +-------+-------+
                                 |
                           Lambda event
                           source mapping
                                 |
                                 v
                         +---------------+
                         |    Lambda     |
                         +-------+-------+
                                 |
                    +------------+------------+
                    |                         |
                 success                    failure
                    |                         |
                    v                         v
               side effect                  retry
                                              |
                                              v
                                            DLQ

Inside Lambda:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Receive batch
     |
     v
Validate message
     |
     v
Check idempotency
     |
     v
Perform business operation
     |
     v
Commit business state
     |
     v
Return success

The Most Important Failure Window

Consider this:

1
2
3
4
5
1. Lambda receives message
2. Database update succeeds
3. Lambda crashes
4. Message becomes visible again
5. Lambda processes it again

If step 2 is not idempotent:

1
Database update x2

If step 2 is idempotent:

1
2
Database update
Retry -> duplicate detected

That is the difference between:

1
duplicate delivery

and:

1
duplicate business effect

The first is normal distributed-systems behavior.

The second is an application design bug.

Partial Batch Responses + Idempotency

These solve different problems.

Partial batch response

Answers:

Which messages should be retried from this batch?

Idempotency

Answers:

What should happen if this message is processed again?

DLQ

Answers:

What should happen when this message keeps failing?

They work together:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
             SQS
              |
              v
         Lambda Batch
              |
       +------+------+
       |             |
     success        failure
       |             |
       v             v
  no retry       retry only
                     |
                +----+----+
                |         |
              success    fail
                          |
                          v
                         DLQ

And every successful business operation should remain safe to repeat.

Observability

Monitor the queue, Lambda, and DLQ together.

Useful SQS signals include:

1
2
3
4
ApproximateNumberOfMessagesVisible
ApproximateAgeOfOldestMessage
NumberOfMessagesReceived
NumberOfMessagesDeleted

Also monitor:

1
2
3
4
5
DLQ message count
Lambda Errors
Lambda Throttles
Lambda Duration
Lambda ConcurrentExecutions

A useful alert could be:

1
DLQ messages > 0

Another useful signal is:

1
Age of oldest message keeps increasing

That usually means:

1
Arrival rate > processing capacity

or that messages are repeatedly failing.

A Queue Is Also a Shock Absorber

One of the benefits of SQS is that it decouples producers from consumers.

Without a queue:

1
2
3
4
5
6
7
Traffic spike
    |
    v
API
    |
    v
Database

With a queue:

1
2
3
4
5
6
7
8
9
10
11
12
13
Traffic spike
    |
    v
Producer
    |
    v
SQS  <--- absorbs burst
    |
    v
Consumer
    |
    v
Database

But queues do not eliminate capacity problems.

They move them.

Instead of:

1
HTTP latency

you may see:

1
2
3
Queue age
Queue depth
Consumer lag

So the system should monitor both synchronous latency and asynchronous backlog.

Load Testing the Failure Paths

Do not test only:

1
Message arrives -> Lambda succeeds

Test:

1
2
3
4
5
6
7
8
9
10
1. Lambda succeeds normally
2. Lambda fails halfway through
3. Database is slow
4. Visibility timeout is too short
5. One message in a batch fails
6. Same message is delivered twice
7. Poison message repeatedly fails
8. DLQ receives message
9. Message is replayed
10. Consumer is restarted during processing

One particularly valuable test is:

1
2
3
Process message
+
force failure after business side effect

Then verify:

1
2
3
Message can reappear
+
Business state is still correct

That test validates the actual distributed-systems behavior.

Production Checklist

Before putting an SQS + Lambda workflow into production:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[ ] Consumer is idempotent
[ ] Visibility timeout matches Lambda runtime
[ ] Batch size is intentional
[ ] Partial batch response is enabled where useful
[ ] FIFO is used only when ordering is actually required
[ ] Message groups are partitioned sensibly
[ ] DLQ is configured
[ ] maxReceiveCount is intentional
[ ] DLQ is monitored
[ ] Poison messages have an investigation path
[ ] Replay process is safe
[ ] Business side effects have uniqueness/idempotency protection
[ ] Queue age is monitored
[ ] Lambda errors and throttling are monitored
[ ] Failure paths have been load-tested

The Bigger Lesson

The dangerous assumption in event-driven systems is:

"If my Lambda succeeded once, the message is finished forever."

The stronger model is:

1
2
3
4
5
Messages can be delivered again.
Processing can fail after a side effect.
Batches can partially fail.
Bad messages can repeatedly fail.
Consumers must be safe to run again.

That is why a reliable SQS/Lambda architecture uses several controls together:

1
2
3
4
5
6
7
8
9
10
11
Visibility Timeout
        +
Partial Batch Response
        +
Idempotent Consumer
        +
Dead-Letter Queue
        +
Monitoring
        +
Failure Testing

The goal is not to make retries disappear.

The goal is to make retries boring.

When the same message arrives twice, the system should say:

1
"Already processed."

not:

1
"Create another payment."

# amazon-sqs # aws-lambda # java # scalability

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

The Lambda Worked. The Queue Kept Replaying Messages.

Ashish kumar jha

The Lambda Worked. The Queue Kept Replaying Messages.

The Incident

SQS Is Not "Run This Exactly Once"

Visibility Timeout Is Not a Processing Guarantee

A Dangerous Example

Example Visibility Timeout Calculation

The Batch Problem

Partial Batch Response

Java Example

The Poison Message Problem

Dead-Letter Queue

Why the DLQ Matters More Than It Looks

But a DLQ Does Not Fix the Business Problem

The Consumer Must Still Be Idempotent

SQS FIFO Is Not a Universal Duplicate Solution

Ordering Has a Cost

A Reliable Architecture

The Most Important Failure Window

Partial Batch Responses + Idempotency

Partial batch response

Idempotency

DLQ

Observability

A Queue Is Also a Shock Absorber

Load Testing the Failure Paths

Production Checklist

The Bigger Lesson