AWS Lambda and SQS: Why Retries Need More Than a Queue

A queue helps separate an application from its background work. But adding Amazon SQS and AWS Lambda does not automatically make that work safe to retry.

Consider an application that queues a request to generate a document. Lambda processes the request, saves the document, and then fails before the message is successfully acknowledged. The same request may be processed again.

That possibility changes how the worker should be designed.

Lambda’s SQS event source mapping processes events at least once. Duplicate processing can occur, so the worker should be idempotent: repeating the same request should not create an unintended additional effect.

For the document example, a stable request ID could identify the job across retries. The implementation must also handle concurrent attempts and failures between creating the document and recording completion. A simple “check whether it exists” is not enough if two workers can pass that check together.

Lambda can receive multiple SQS messages in one batch. By default, a failed batch can cause successfully processed messages to become available again.

Partial batch responses let the function identify failed messages so those messages can be retried separately. This requires enabling ReportBatchItemFailures on the event source mapping and returning the failed message identifiers correctly.

Partial batch responses reduce unnecessary reprocessing. They do not remove the need for idempotency.

A useful test is not just “Does the function process a message?”

It is: “What happens if the work succeeds, but the function fails before completion is acknowledged?”

That scenario exposes whether a retry produces a duplicate document, notification, or other side effect.

The architecture lesson is straightforward: a queue separates components, but the application still needs to define what happens when the same work arrives again.