A Safer S3-to-Lambda Pipeline with SQS, Idempotency, and DLQs

S3 event notifications are convenient, but a direct event-to-function path leaves little room to absorb bursts or inspect failures. For uploads that trigger image processing, document scans, or data imports, put a queue between S3 and Lambda so retries and backpressure are explicit.

Architecture

S3 ObjectCreated event → SQS standard queue → Lambda event source mapping → processor
└→ redrive to a dead-letter queue after repeated failures

Keep processed outputs in a different bucket or a prefix that does not match the notification filter. Otherwise a function that writes back to its source path can trigger itself.

Configure the bucket notification for only the required event types and input prefix, such as ObjectCreated under incoming/. Use an SQS standard queue as the destination. Standard queues provide high throughput, but delivery is at least once and ordering is not guaranteed; design the handler for duplicates and out-of-order events.

The queue policy should allow the S3 service principal to call sqs:SendMessage, constrained by the exact source bucket ARN and your AWS account ID. Avoid a broad Principal with no source condition. If the queue uses a customer-managed KMS key, grant the required service and consumer permissions there too, and test the path before rollout.

An S3 notification is a signal to inspect an object, not a guarantee that the signal appears exactly once. Build an idempotency key from the bucket, decoded object key, and version ID when bucket versioning is enabled. Use a DynamoDB conditional write such as attribute_not_exists(pk) to claim work before making non-idempotent changes. Set a TTL only for cleanup; the conditional write is the correctness check.

If the claim already exists, treat the event as a duplicate and return success. If a later object version arrives, its version ID gives it a distinct key. For unversioned buckets, define how an overwrite should be handled and include the event sequencer or object metadata in that decision rather than assuming arrival order.

Connect Lambda with an SQS event source mapping. Start with a modest batch size and a visibility timeout comfortably longer than the function timeout; AWS recommends at least six times the function timeout, plus any maximum batching window. A message that becomes visible while the first invocation is still running can be processed twice.

Enable partial batch responses so one bad record does not force successful records in the same batch to run again. Return only failed message identifiers. Configure the source queue's redrive policy with a maximum receive count and a dead-letter queue. Alarm on a nonzero DLQ depth and on the age of the oldest source-queue message.

Give the function only the S3 read actions and object prefix it needs, plus the specific DynamoDB table actions used by the idempotency check. Keep deployment permissions separate from runtime permissions. Validate the bucket, key, event type, and expected file format before processing; send malformed or unsupported inputs to a controlled failure path.

Track Lambda errors, throttles, duration, SQS ApproximateAgeOfOldestMessage, and DLQ visible messages in CloudWatch. Log a correlation key and processing outcome, but avoid logging file contents or sensitive object metadata. A small dashboard makes it easier to tell a traffic spike from a poison message.

Cost and tradeoffs

SQS adds request charges and another resource to operate, while buffering smooths bursts and makes retry behavior easier to manage. DynamoDB adds a small per-event cost for idempotency but prevents duplicate side effects. Batches can reduce invocation overhead, though very large batches increase retry scope and processing latency. Start small, load-test representative files, and adjust using queue age and failure metrics.

The useful property of this design is that each layer has one job: S3 announces, SQS absorbs, Lambda processes, DynamoDB guards side effects, and the DLQ preserves failures for investigation. That makes recovery a normal operating path instead of a manual replay guess.