Stop Making Users Wait: Background Jobs with Amazon SQS and AWS Lambda
Move slow tasks out of your API request path and learn how to handle retries, duplicate messages, and failed jobs with Amazon SQS and AWS Lambda.
A user clicks “Generate report.” Your application starts querying data, formatting a document, and uploading the result. Meanwhile, the browser waits.
As the report becomes more complex, a normal request can turn into a frustrating experience. A traffic spike makes the problem worse: every request now competes for the same processing resources.
One useful approach is to accept the job quickly and process it in the background. Amazon SQS provides a managed queue between the component accepting the request and the component doing the work.
This article uses report generation as a hypothetical example. The same design can support image processing, document conversion, and other tasks that do not need to finish before the initial response.
Separate acceptance from completion
In this design, an API validates the request, creates a job identifier, and sends a message to an SQS queue. After SQS accepts the message, the API returns an HTTP
202 Accepted response containing that identifier.A worker processes the request later. The application exposes job status so the user can check whether the report is pending, running, completed, or failed.
These statuses are application behavior you implement. SQS does not provide a user-facing job status API.
For a first version, use a standard queue when independent report requests can finish in any order. Standard queues provide at-least-once delivery and best-effort ordering. If your workflow needs ordering, evaluate FIFO queues and message groups. [1]
Let Lambda handle queue polling
An SQS event source mapping connects the queue to your Lambda function. Lambda polls for messages and invokes the function with a batch.
Messages remain in the queue while processing, temporarily hidden by the visibility timeout. After successful processing, Lambda removes them. By default, a batch failure makes the batch available again after that timeout. Duplicate processing can occur, so the worker must tolerate repeated delivery. [2]
The queue and function must be in the same AWS Region. Give the function's execution role permission to consume the queue and write logs. AWS provides the
AWSLambdaSQSQueueExecutionRole managed policy for a starting setup; review permission scope for your application. [3]Keep the job message small and explicit
A report request might look like this:
1
2
3
4
5
6
{
"schemaVersion": 1,
"jobId": "report-2026-0042",
"reportType": "monthly-sales",
"period": "2026-09"
}Use a stable job identifier across retries. Include enough information for the worker to understand the request, and keep larger inputs in durable storage when appropriate.
Treat message contents as input that still needs validation. A queue decouples components; it does not make every producer's payload correct.
Retry individual failures
Suppose a batch contains five jobs and one is malformed. Repeating all five wastes work.
Enable
ReportBatchItemFailures on the event source mapping and return the identifiers of failed records. Lambda can then retry those records without retrying the successful ones because of that batch failure. Returning the response alone is insufficient: the mapping setting must also be enabled. [4]This Python handler demonstrates validation and partial failure reporting for a standard queue:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import json
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def process_job(job):
if job.get("schemaVersion") != 1:
raise ValueError("Unsupported schema version")
job_id = job.get("jobId")
if not isinstance(job_id, str) or not job_id:
raise ValueError("Missing or invalid jobId")
# Demonstration only: replace this log with your job processing.
logger.info("Validated job %s", job_id)
def lambda_handler(event, context):
failures = []
for record in event["Records"]:
try:
job = json.loads(record["body"])
if not isinstance(job, dict):
raise ValueError("Message body must be an object")
process_job(job)
except Exception:
logger.exception("Failed message %s", record["messageId"])
failures.append({
"itemIdentifier": record["messageId"]
})
return {"batchItemFailures": failures}This example validates and logs jobs; it does not generate a report or implement durable job status.
If the whole invocation raises an exception, the entire batch fails. FIFO processing also needs different failure handling: stop after the first failure and report failed and unprocessed messages to preserve ordering. [4]
Design for repeated execution
Partial batch responses reduce unnecessary retries, but they do not eliminate duplicates.
Make the actual business operation idempotent: repeating a job should not create an unintended second result. AWS recommends idempotent Lambda code. [5]
For report generation, a possible design uses
jobId to identify the output and a durable job record. Coordinate concurrent attempts with an atomic claim or equivalent mechanism. Also plan recovery if the worker crashes after writing the report but before marking the job complete.A simple “check whether completed, then run” sequence is insufficient when two workers can pass the check together. Design the state transitions around both concurrency and interruption.
Give processing time and failures a destination
Choose the function timeout for the entire batch. AWS recommends a queue visibility timeout of at least six times the function timeout, plus the batching window when one is configured. For example, a 30-second function timeout and no batching window suggest at least 180 seconds of visibility. [3]
Configure an SQS dead-letter queue through the source queue's redrive policy. Messages that exceed the receive threshold move there for investigation. AWS recommends a
maxReceiveCount of at least five for this integration. [3]A dead-letter queue is a recovery tool. Inspect the failure, fix the cause, and decide whether replay is appropriate. Set its retention longer than the source queue's retention so failures remain available for investigation. [6]
Monitor the waiting work
Use CloudWatch to watch the source queue's
ApproximateNumberOfMessagesVisible for backlog and ApproximateAgeOfOldestMessage for waiting time.For the dead-letter queue, alarm on
ApproximateNumberOfMessagesVisible. Automatically transferred messages do not contribute to the DLQ's NumberOfMessagesSent metric. Queue metrics are approximate, and the age metric has special behavior for repeatedly failing messages, so interpret them alongside worker logs. [7]Define an application goal such as “reports normally finish within two minutes.” That gives your alarms a purpose beyond tracking message counts.
Try the failure path before expanding
Start with a small sandbox:
- Create a standard queue and an SQS dead-letter queue.
- Connect a Lambda worker and enable partial batch responses.
- Send one valid message and one malformed message.
- Confirm the valid message completes and the malformed one eventually reaches the DLQ.
- Repeat a job identifier to verify your real processing handles duplicates.
- Remove the test resources and associated logs when finished.
Moving work behind a queue can shorten the initial request and absorb bursts. The quality of the experience still depends on what happens afterward: clear job status, safe retries, and a recovery path for work that cannot complete.
What task in your application would benefit most from background processing?
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article