Your Queue Is a Waiting Room, Not a Black Hole: An SQS Support Group
A fictional support group for overworked messages, with practical Amazon SQS and Lambda workflows for duplicates, partial batch failures, and a dead-letter queue that actually gets attention.
A fictional scene, a real engineering problem. This is a design walkthrough for an Amazon SQS standard queue with AWS Lambda, not a report of a deployed system.
Welcome to Queue Anonymous
“Hello. My name is Message 42, and I have been processed three times.”
“Hello, Message 42,” replied the room.
A Lambda function shuffled its execution logs.
“In my defence, nobody told me you were the same order.”
From the back, a dead-letter queue cleared its throat.
“At least somebody visits you.”
Distributed systems are wonderfully efficient at turning one small misunderstanding into a recurring calendar event. A queue buys time between producing work and doing it. The engineering begins when we decide what happens during that time—and what happens when time runs out.
Let’s give our messages a better workplace.
Meet the cast
Character | Actual responsibility | Unhelpful personality trait |
|---|---|---|
Producer | Publishes a task with a stable business operation ID | Thinks “sent” means “finished” |
SQS source queue | Holds work until consumers can handle it | Keeps no diary of your business intentions |
Lambda worker | Validates and processes records | Occasionally meets an unavailable dependency |
Idempotency mechanism | Prevents a repeated operation from repeating its effect | Demands that you define “the same operation” |
SQS dead-letter queue | Isolates repeatedly unsuccessful messages | Becomes furniture if nobody monitors it |
Our example task is generating an order summary. The producer assigns an operation ID such as
summary:order-42:v1 . That ID follows the work through processing, logs, and recovery.Workflow 1: the ordinary day, including the unusual bits
- The producer sends a task to the source queue.
- Lambda’s event source mapping polls SQS and invokes the worker with a batch.
- The worker validates each record and checks the business operation ID before applying its effect.
- Completed records are acknowledged through the successful response. Lambda removes them from the queue.
- Failed records are reported for retry. They can become available again after the visibility timeout.
- With a source queue redrive policy configured, repeatedly received messages can move to the SQS dead-letter queue.
While records are being processed, they remain in SQS but are temporarily hidden. Think of the visibility timeout as the receptionist saying, “This customer is being served.” It is not a certificate proving the job finished.
AWS documents at-least-once processing for this integration. Duplicate processing can happen. Design the worker accordingly. [1]
The duplicate who wore a fake moustache
A retry is not necessarily a new customer.
Use a stable operation ID tied to the business action, rather than assuming a fresh delivery represents fresh work. If the producer republishes the same action, the business ID should still identify it.
The tricky part is coordinating the effect and its completion record. “Check a database, do the work, then save done” has a crash window: the work might succeed just before the completion write fails.
For a real implementation, choose an idempotency approach that handles concurrent attempts, in-progress work, expiry, and failures. Where the downstream service supports an idempotency key, use it. Where one database owns both the effect and the record, consider an atomic transaction.
A sticky note saying “probably done” is an observability strategy only in a very small office.
Workflow 2: one bad record should not ruin the school trip
Imagine a batch containing three tasks:
- Order A: completed.
- Order B: malformed payload.
- Order C: completed.
By default, a failed batch can cause successfully processed records to be retried too. Now A and C are back in the waiting room, asking for the manager.
Enable
ReportBatchItemFailures on the event source mapping and return only the failed message identifiers:JSON
{"batchItemFailures": [{ "itemIdentifier": "message-id-for-order-B" }]}
This response shape is illustrative; substitute the failed SQS record’s actual
messageId .Implement record-level error handling so the response is actually returned. An uncaught exception makes the entire batch fail. Partial batch reporting reduces unnecessary repeats; it does not replace idempotency. [2]
FIFO readers: ordering needs additional care. AWS says to stop after the first failure and include failed and unprocessed messages in the response. This article’s main workflow assumes a standard queue. [2]
The dead-letter queue wants an owner, not sympathy
“Why am I called dead-letter?” asked the DLQ.
“Because ‘Tasks Requiring Accountable Follow-Up’ was less dramatic.”
A DLQ needs an operational workflow:
- Notice: alert when work arrives or remains unresolved.
- Inspect: use the operation ID and logs to identify the cause. Avoid dumping private payloads into logs.
- Classify: distinguish a temporary dependency failure from invalid input or a code defect.
- Repair: fix the underlying issue. Replaying unchanged bad input is optimism with API calls.
- Redrive cautiously: start with a small amount of work, check downstream capacity, and observe the results.
- Close the loop: confirm the business operation completed and record what prevented recurrence.
For this integration, configure the SQS source queue’s redrive policy. Do not confuse it with Lambda’s separate asynchronous-invocation DLQ configuration. The source queue policy supplies the receive limit used for moving work to the SQS DLQ. [3]
The DLQ is a repair desk. A repair desk without an owner is a museum.
Settings with actual consequences
AWS recommends a source queue visibility timeout of at least six times the Lambda function timeout, plus the batching window when one is configured. It also recommends a
maxReceiveCount of at least five for this integration. Treat these as documented starting guidance and test your workload. [3]For example, a 20-second function timeout and a 5-second batching window gives a starting visibility timeout of at least 125 seconds.
Also decide how quickly the downstream system can safely accept work. A queue can absorb a burst; it cannot make an exhausted dependency enthusiastic.
Rehearse the awkward conversations
Before calling the design ready, exercise these scenarios:
Experiment | What you want to observe |
|---|---|
Deliver the same business operation twice | One intended business effect |
Put one failing record beside valid records | Only failed records reported for retry |
Make the downstream dependency unavailable | Bounded pressure and visible failures |
Exhaust the configured receive limit | Work isolated in the SQS DLQ |
Fix the cause and redrive a small batch | Successful processing without duplicate effects |
These are proposed checks, not test results. The best time to discover your recovery plan is missing is before the incident opens a group chat.
Send your messages home
A healthy queue has a clear path through waiting, processing, retry, investigation, and recovery.
Give every operation an identity. Give failures a route. Give the DLQ an owner.
And if Message 42 comes back wearing a moustache, recognise the order before generating another summary.
What is the funniest retry-related surprise you have encountered—and what did you change afterwards?
References
Written with AI assistance. Cover illustration generated with AI; workflow artwork is a conceptual illustration of the design.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article