AWS Builder Center

Building a Serverless File Upload Pipeline with Amazon S3 and AWS Lambda

Learn how to build an automated serverless file-processing pipeline where files uploaded to Amazon S3 trigger AWS Lambda for validation, metadata extraction, and automated processing.

Building a Serverless File Upload Pipeline with Amazon S3 and AWS Lambda

File uploads are a common requirement in modern applications. Users may upload images, documents, videos, or other files that need to be validated and processed after the upload.
Instead of making the application server handle every processing operation, we can use an event-driven serverless architecture.
In this tutorial, we'll build a simple file-processing pipeline using Amazon S3 and AWS Lambda.

Architecture

The basic workflow looks like this:
┌─────────────────┐
│ User │
└────────┬────────┘
│
│ Upload
v
┌─────────────────┐
│ Amazon S3 │
│ Uploads │
└────────┬────────┘
│
│ S3 Event
v
┌─────────────────┐
│ AWS Lambda │
│ File Processor │
└────────┬────────┘
│
┌────────┴────────┐
v v
Metadata Store Processed File
The key idea is that uploading a file automatically starts the processing workflow.



1. Create an S3 Bucket

Create an S3 bucket to store uploaded files.
For example:
my-file-processing-bucket
Create a prefix for incoming files:
uploads/
Uploaded files might look like:
uploads/document-001.pdf
uploads/image-001.jpg
uploads/report-001.csv
For production applications, enable appropriate security controls and consider encryption, versioning, lifecycle policies, and access logging based on your requirements.



2. Create the Lambda Function

Create an AWS Lambda function using Python.
The function will receive information about the object uploaded to S3.
A simplified implementation looks like this:
import json
import urllib.parse

def lambda_handler(event, context):

for record in event["Records"]:

bucket = record["s3"]["bucket"]["name"]

key = urllib.parse.unquote_plus(
record["s3"]["object"]["key"]
)

size = record["s3"]["object"]["size"]

print(f"Bucket: {bucket}")
print(f"Object: {key}")
print(f"Size: {size} bytes")

return {
"statusCode": 200
}
The function receives the S3 event and extracts information about the uploaded object.



3. Configure the S3 Event Trigger

Configure the S3 bucket to invoke the Lambda function when an object is created.
The workflow becomes:
User
|
v
S3 Upload
|
v
Object Created Event
|
v
Lambda
|
v
Process File
This means the application doesn't need to continuously check whether new files have arrived.
S3 generates an event when the relevant object operation occurs.



4. Filter Which Files Trigger Lambda

You may not want every object in the bucket to trigger the function.
For example, you could configure the event to apply only to:
uploads/
You could also design the workflow around particular file types.
For example:
uploads/
|
+---- images
|
+---- documents
|
+---- reports
Different prefixes can be used to separate different processing workflows.



5. Process the Uploaded File

The Lambda function can download the object and perform application-specific processing.
For example, an image-processing workflow might look like:
Image Upload
|
v
S3
|
v
Lambda
|
+---- Validate image
|
+---- Extract metadata
|
+---- Resize image
|
v
Processed Image
The exact processing logic depends on the application's requirements.
For larger or longer-running workloads, Lambda may not be the best processing environment. In those cases, the workflow can be extended with additional services designed for the workload.



6. Store Metadata

The application may want to store information about each uploaded file.
For example:
{
"fileName": "report.pdf",
"bucket": "my-file-processing-bucket",
"size": 245760,
"status": "PROCESSED"
}
A database such as Amazon DynamoDB can be used to store this metadata.
The architecture then becomes:
Amazon S3
|
v
AWS Lambda
/ \
v v
DynamoDB Processing
This separates the actual file storage from the application's metadata.



7. Apply Least-Privilege IAM

The Lambda execution role should have only the permissions necessary for the processing workflow.
For example, if Lambda only needs to read objects from a particular S3 bucket, avoid granting unrestricted access to every S3 bucket in the AWS account.
A production IAM design should carefully define:
  • Which bucket Lambda can access.
  • Which S3 operations it can perform.
  • Which database resources it can modify.
  • Which logging permissions it requires.
Least-privilege permissions reduce unnecessary access.



8. Think About Duplicate Processing

Event-driven systems should account for the possibility that processing may occur more than once.
The application should therefore avoid assuming that every event will be processed exactly once.
For example, before updating metadata, the application could check whether the particular object has already been processed.
A simple workflow could be:
S3 Event
|
v
Lambda
|
v
Check Processing Status
|
+---- Already processed ---> Stop
|
+---- New file ------------> Process
This makes the system more resilient to retries and duplicate events.



9. Monitor the Pipeline

Once the pipeline is deployed, monitoring becomes important.
Useful things to monitor include:
  • Lambda invocation count
  • Lambda errors
  • Lambda duration
  • S3 object creation activity
  • Processing failures
  • Number of files processed
Amazon CloudWatch can be used to monitor Lambda execution and create alarms for important conditions.
For example, a sudden increase in Lambda errors could indicate a problem with a new file format or application deployment.



10. Add a Failure Workflow

A production file-processing system should also have a strategy for failed processing.
For example:
S3
|
v
Lambda
|
┌─────┴─────┐
v v
Success Failure
| |
v v
Processed Failed/
Files Quarantine
Failed files could be moved or copied into a dedicated location for investigation, depending on the application's requirements.
This prevents problematic files from repeatedly entering the normal processing workflow.



11. Extend the Architecture

Once the basic pipeline works, it can be extended considerably.
For example:
S3
|
v
Lambda
|
┌────────────┼────────────┐
v v v
Metadata Processing Events
| | |
v v v
DynamoDB S3 EventBridge
Possible extensions include:
  • Image resizing
  • File validation
  • Metadata extraction
  • Virus scanning
  • Document processing
  • Notifications
  • Audit logging
  • Automatic archival
  • Content classification
The architecture can evolve based on the application's requirements.



Conclusion

An S3-triggered Lambda workflow is a simple example of how AWS managed services can be combined to build an event-driven application.
The main concepts demonstrated in this project are:
  1. Amazon S3 for durable object storage.
  1. S3 events for reacting to object operations.
  1. AWS Lambda for serverless processing.
  1. IAM for controlling resource access.
  1. Amazon DynamoDB for optional metadata storage.
  1. Amazon CloudWatch for monitoring.
  1. Idempotent processing for handling retries safely.
The most important architectural lesson is that the file upload and file processing stages do not need to be tightly coupled.
Once the file is stored, the event can trigger the appropriate processing workflow automatically. This approach can make applications easier to scale and operate while reducing the amount of infrastructure developers need to manage.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article