AWS Builder Center

Builder Center

Build a Reliable S3 Processing Pipeline with SQS and Lambda

Learn how to build a reliable, retryable file-processing pipeline using Amazon S3, SQS, and Lambda, with dead-letter queues and idempotency.

Anupam

Build a Reliable S3 Processing Pipeline with SQS and Lambda

File uploads are a common starting point for cloud applications. A user uploads an image, document, or data file, and the application needs to process it.
A simple implementation might trigger an AWS Lambda function directly whenever a file arrives in Amazon S3. This works for small workloads, but it can become unreliable when processing takes longer, failures occur, or many files arrive simultaneously.
A more resilient design separates file storage from file processing by introducing Amazon SQS between Amazon S3 and Lambda.
This article demonstrates how to build that architecture, why it is useful, and what to consider before deploying it in production.

Architecture

The pipeline consists of four primary components:
  • Amazon S3 — Stores uploaded files.
  • Amazon SQS — Buffers processing requests and enables retries.
  • AWS Lambda — Processes files asynchronously.
  • Amazon CloudWatch — Provides logs, metrics, and operational visibility.
The flow looks like this:
User uploads a file
        |
        v
   Amazon S3
        |
        | Object-created event
        v
   Amazon SQS
        |
        | Event source mapping
        v
   AWS Lambda
        |
        v
 File processing
        |
        +---- Success
        |
        +---- Failure after retries
                    |
                    v
              Dead-letter queue
The important design decision is that S3 does not need to wait for processing to finish. It only needs to store the object and publish an event.

Why Use SQS Between S3 and Lambda?

A direct S3-to-Lambda integration is convenient, but it does not provide the same level of control as a queue-based design.
Adding SQS provides several benefits:

1. Buffering

If thousands of files are uploaded within a short period, SQS can temporarily hold processing requests while Lambda scales to handle the workload.

2. Automatic retries

If a Lambda invocation fails, the message can become visible again after the visibility timeout and be retried.

3. Dead-letter queues

Messages that repeatedly fail can be moved to a dead-letter queue instead of being retried indefinitely.

4. Better failure isolation

A problematic file should not prevent other files from being processed.

5. Independent scaling

File uploads and file processing can scale independently.
However, SQS does not automatically make a system exactly-once. A message may be delivered more than once, so the processing logic must be designed to be idempotent.

Step 1: Create the S3 Bucket

Create a bucket to store incoming files.
For production workloads, consider enabling:
  • Block Public Access
  • Server-side encryption
  • Versioning
  • Lifecycle rules
  • Access logging or CloudTrail data events where appropriate
For example, a bucket might use the following prefix structure:
uploads/
    images/
    documents/
    incoming/
Using prefixes makes it easier to separate different types of processing.
Avoid allowing public access to uploaded files unless public access is an explicit and carefully reviewed requirement.

Step 2: Create the SQS Queues

Create two queues:
  1. Processing queue — Receives file-processing requests.
  1. Dead-letter queue (DLQ) — Stores messages that fail repeatedly.
The DLQ helps operators investigate failures without losing the original processing request.
A simplified AWS CloudFormation configuration looks like this:
Resources:
  ProcessingDLQ:
    Type: AWS::SQS::Queue

  ProcessingQueue:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 180
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt ProcessingDLQ.Arn
        maxReceiveCount: 5
The  VisibilityTimeout  should be longer than the maximum expected Lambda processing time.
For example, if processing normally takes 30 seconds but occasionally takes 90 seconds, a timeout of 180 seconds provides additional room.
The exact value should be based on measurements rather than guesswork.

Step 3: Configure S3 Event Notifications

Configure Amazon S3 to publish object-created events to the processing queue.
A typical event contains information about the bucket and object:
{
  "Records": [
    {
      "eventName": "ObjectCreated:Put",
      "s3": {
        "bucket": {
          "name": "example-upload-bucket"
        },
        "object": {
          "key": "uploads/example.pdf",
          "size": 102400
        }
      }
    }
  ]
}
When using direct S3-to-SQS notifications, configure an SQS queue policy that allows the specific S3 bucket to call  sqs:SendMessage . Restrict the policy using the bucket ARN and, where appropriate, the  aws:SourceAccount  condition. The S3 bucket and SQS queue must also be in the same AWS Region. The event contains metadata about the object, not the complete file contents. The Lambda function can use the bucket name and object key to retrieve the file from S3 when processing begins.

Important: URL-Encoded Object Keys

S3 object keys in event notifications can be URL-encoded.
For example:
uploads/my document.pdf
may appear as:
uploads%2Fmy+document.pdf
Always decode the key before using it with the S3 API.
In Python:
from urllib.parse import unquote_plus

key = unquote_plus(record["s3"]["object"]["key"])
Failing to decode the key can cause processing errors for filenames containing spaces or special characters.

Step 4: Create the Lambda Function

The Lambda function reads messages from SQS and processes the corresponding S3 objects.
A simplified Python example:
import json
import logging
import os
import boto3

logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3 = boto3.client("s3")

def process_file(bucket: str, key: str) -> None:
    """
    Replace this function with your actual processing logic.
    Examples:
    - Extract text from a document
    - Generate image thumbnails
    - Validate a data file
    - Run a malware scanning workflow
    - Transform and store metadata
    """
    response = s3.head_object(Bucket=bucket, Key=key)

    logger.info(
        "Processing file",
        extra={
            "bucket": bucket,
            "key": key,
            "size": response.get("ContentLength")
        }
    )

    # Add application-specific processing here.

def lambda_handler(event, context):
    for record in event["Records"]:
        message = json.loads(record["body"])

        for s3_record in message.get("Records", []):
            bucket = s3_record["s3"]["bucket"]["name"]
            key = s3_record["s3"]["object"]["key"]

            process_file(bucket, key)

    return {
        "statusCode": 200,
        "body": "Processing completed"
    }
This example focuses on the architecture. In a real application, the processing function should validate the event, handle expected errors, and record useful metadata.

Step 5: Connect SQS to Lambda

Lambda can poll the SQS queue through an event source mapping.
The event source mapping automatically:
  1. Polls the queue.
  1. Retrieves messages.
  1. Invokes the Lambda function.
  1. Deletes successfully processed messages.
  1. Leaves failed messages available for retry.
A simplified configuration might look like this:
FileProcessor:
  Type: AWS::Lambda::Function
  Properties:
    Runtime: python3.12
    Handler: index.lambda_handler
    Timeout: 30
    Role: !GetAtt LambdaExecutionRole.Arn
    Code:
      ZipFile: |
        def lambda_handler(event, context):
            print(event)
            return {"statusCode": 200}

ProcessingQueue:
  Type: AWS::SQS::Queue
  Properties:
    VisibilityTimeout: 180
In this example, the Lambda timeout is 30 seconds and the queue visibility timeout is 180 seconds. For production workloads, calculate the visibility timeout using the Lambda timeout, batching window, and AWS recommendations rather than choosing an arbitrary value. The  BatchSize  controls how many messages Lambda receives per invocation. A larger batch can improve throughput, but it also increases the amount of work involved when failures occur.

Partial Batch Failure Handling

One important production improvement is enabling partial batch responses.
Without partial batch handling, a single failed message can cause the entire batch to be retried.
For example:
Batch:
    Message A → Success
    Message B → Success
    Message C → Failure
    Message D → Success
With partial batch failure handling, only Message C needs to be retried.
A more complete Lambda example:
import json
import logging
import boto3

logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3 = boto3.client("s3")

def process_file(bucket: str, key: str) -> None:
    logger.info("Processing s3://%s/%s", bucket, key)

    # Replace with actual processing logic.
    s3.head_object(Bucket=bucket, Key=key)

def lambda_handler(event, context):
    batch_item_failures = []

    for record in event["Records"]:
        try:
            message = json.loads(record["body"])

            for s3_record in message.get("Records", []):
                bucket = s3_record["s3"]["bucket"]["name"]
                key = s3_record["s3"]["object"]["key"]

                process_file(bucket, key)

        except Exception:
            logger.exception(
                "Failed to process message %s",
                record["messageId"]
            )

            batch_item_failures.append({
                "itemIdentifier": record["messageId"]
            })

    return {
        "batchItemFailures": batch_item_failures
    }
This allows successfully processed messages to be acknowledged while failed messages remain available for retry.

Step 6: Make Processing Idempotent

This is one of the most important parts of the design.
SQS provides at-least-once delivery. That means the same message may occasionally be delivered more than once.
For example:
S3 upload
    |
    v
SQS message
    |
    v
Lambda processes file successfully
    |
    X Network interruption before message deletion
    |
    v
SQS delivers the message again
If the Lambda function generates a thumbnail, sends an email, or charges a customer, duplicate processing could create incorrect results.

Strategies for Idempotency

Use the S3 Object Key as an Identifier

For some workflows, the combination of bucket, object key, and version ID can identify a unique processing request.

Store Processing State

Use a database such as DynamoDB to record whether an object has already been processed.
Example conceptual record:
{
  "id": "example-upload-bucket/uploads/example.pdf",
  "status": "COMPLETED",
  "processedAt": "2026-09-14T10:00:00Z"
}
Before performing a non-idempotent operation, check the processing state.

Use Deterministic Outputs

If processing always writes to the same output key, retries can overwrite the same result instead of creating duplicates.
For example:
Input:
uploads/example.pdf

Output:
processed/example.pdf.json
The output location is predictable and repeatable.
Idempotency should be designed into the application rather than added after duplicate processing becomes a problem.

Step 7: Configure IAM Permissions

The Lambda execution role should follow the principle of least privilege.
For example, if the function only needs to read files from a specific bucket and write results to another prefix, avoid granting unrestricted S3 access.
A simplified policy might look like this:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::example-upload-bucket/uploads/*"
    }
  ]
}
The Lambda execution role must also allow the permissions required to consume messages from the SQS queue, including  sqs:ReceiveMessage ,  sqs:DeleteMessage , and  sqs:GetQueueAttributes . The role also needs CloudWatch Logs permissions. In production, scope these permissions to the specific queue and S3 resources used by the function.
You may also need permissions for:
  • Writing processed results
  • Reading object metadata
  • Writing logs to CloudWatch
  • Accessing DynamoDB for idempotency tracking
  • Decrypting objects when using customer-managed KMS keys
Keep permissions scoped to the resources and actions the function actually requires.

Step 8: Monitor the Pipeline

A reliable architecture is not complete without observability.
Useful CloudWatch metrics include:

SQS Metrics

  •  ApproximateNumberOfMessagesVisible 
  •  ApproximateNumberOfMessagesNotVisible 
  •  ApproximateAgeOfOldestMessage 
  • Number of messages sent
  • Number of messages received

Lambda Metrics

  • Invocations
  • Errors
  • Duration
  • Throttles
  • Concurrent executions

DLQ Metrics

  • Number of messages sent to the dead-letter queue
  • Age of the oldest DLQ message
A growing queue may indicate that:
  • Lambda concurrency is too low.
  • Processing is slower than expected.
  • A downstream service is unavailable.
  • Messages are repeatedly failing.
The age of the oldest message is often more useful than queue depth alone because it indicates how long requests have been waiting.

Common Mistakes to Avoid

Mistake 1: Setting the Visibility Timeout Too Low

If Lambda takes longer than the visibility timeout, SQS may make the message visible again while it is still being processed.
This can lead to duplicate processing.

Mistake 2: Assuming Exactly-Once Processing

SQS does not guarantee exactly-once processing for standard queues.
Design for retries and duplicate deliveries.

Mistake 3: Ignoring the Dead-Letter Queue

A DLQ is not just a place where failed messages disappear.
Create an operational process to inspect, diagnose, and replay messages when appropriate.

Mistake 4: Using Unbounded Lambda Concurrency

If every incoming message triggers expensive processing, Lambda may overwhelm downstream services.
Use reserved concurrency, batch sizing, and appropriate scaling controls when necessary.

Mistake 5: Logging Sensitive File Contents

Avoid logging passwords, personal data, access tokens, or entire document contents.
Log identifiers, processing status, and error details that are useful for debugging without exposing sensitive information.

Mistake 6: Forgetting S3 Event Filtering

If the Lambda function writes processed files back into the same bucket, it may accidentally trigger itself again.
Use separate prefixes or buckets, and configure event filters carefully.
For example:
uploads/       → Input files
processed/     → Output files
Only configure the S3 notification for the  uploads/  prefix.

Production Improvements

Once the basic pipeline works, consider adding the following improvements.

Use AWS SAM or AWS CDK

Infrastructure as code makes the architecture repeatable and easier to review.

Add Structured Logging

Include fields such as:
  • Request ID
  • S3 bucket
  • Object key
  • Processing status
  • Error type
  • Processing duration

Add Alarms

Create CloudWatch alarms for:
  • Lambda errors
  • Queue age
  • DLQ message count
  • Lambda throttling

Encrypt Data

Use S3 server-side encryption and SQS encryption. If your compliance requirements require customer-managed keys, configure AWS KMS appropriately.

Validate Uploaded Files

Do not trust file extensions alone. Validate file types, sizes, and content before processing.

Consider Step Functions for Complex Workflows

If processing involves multiple stages, long-running tasks, approvals, or branching logic, AWS Step Functions may be a better fit than putting everything inside one Lambda function.
For example:
S3 Upload
    |
    v
SQS
    |
    v
Lambda: Validate
    |
    v
Step Functions
    |
    +--> Extract metadata
    |
    +--> Transform file
    |
    +--> Store results
    |
    v
Completed

When Should You Use This Architecture?

This pattern is useful when:
  • File processing can happen asynchronously.
  • Upload traffic is unpredictable.
  • Processing may fail temporarily.
  • You need retries and failure isolation.
  • You want to scale storage and processing independently.
  • You need better operational visibility.
It may be unnecessary for very small applications where direct S3-to-Lambda processing is sufficient and the failure-handling requirements are minimal.
The goal is not to add services for the sake of complexity. The goal is to introduce the right level of resilience for the workload.

Final Thoughts

A reliable file-processing pipeline is not just about connecting Amazon S3 to AWS Lambda.
The more important design decisions are:
  1. Buffer work with SQS.
  1. Retry transient failures.
  1. Use a dead-letter queue for messages that cannot be processed.
  1. Design Lambda processing to be idempotent.
  1. Apply least-privilege IAM permissions.
  1. Monitor queue age, errors, and failed messages.
These principles apply beyond file uploads. The same architecture can support image processing, document extraction, data transformation, asynchronous notifications, and many other event-driven workloads.
Start with the simplest architecture that meets your requirements, then add complexity when your workload or reliability needs justify it.
# aws-lambda# amazon-api-gateway# serverless# architecture
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article