Building an Event-Driven Image Processing Pipeline with Amazon S3 and AWS Lambda
Learn how to automatically process uploaded images using Amazon S3 and AWS Lambda. This guide explains event-driven architecture, S3 notifications, Lambda processing, and how to build a simple serverless workflow without managing servers.
Building an Event-Driven Image Processing Pipeline with Amazon S3 and AWS Lambda
Imagine uploading an image to an S3 bucket and having AWS automatically process it without running a server continuously.
That is the basic idea behind an event-driven serverless architecture.
In this article, we'll build a simple image-processing workflow using Amazon S3 and AWS Lambda.
The goal is to understand how an event in one AWS service can automatically trigger computation in another service.
What Are We Building?
Our workflow will look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
User
|
| Upload Image
v
Amazon S3
|
| Object Created Event
v
AWS Lambda
|
| Process Image
v
Processed Output
For example, when a user uploads:
1
photo.jpg
to an S3 bucket, the upload event can trigger a Lambda function.
The Lambda function can then perform an operation such as:
- Validate the file
- Extract metadata
- Resize the image
- Generate a thumbnail
- Move the processed file
- Store processing information
The important part is that no continuously running server is required.
Why Event-Driven Architecture?
Traditional applications often work using a request-response pattern.
A server is running and waiting for requests.
1
2
3
4
5
6
7
Client
|
v
Server
|
v
Process Request
An event-driven application works differently.
Something happens first.
That event triggers the required action.
1
2
3
4
5
6
7
Event
|
v
Trigger
|
v
Processing
This can make applications more efficient because computation happens when it is actually needed.
Architecture
Our architecture is intentionally simple:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
+-------------+
| User |
+------+------+
|
| Upload
v
+------------------+
| Amazon S3 |
| Input Bucket |
+--------+---------+
|
| ObjectCreated
v
+------------------+
| AWS Lambda |
| Image Processor |
+--------+---------+
|
| Process
v
+------------------+
| Amazon S3 |
| Output Bucket |
+------------------+
The S3 upload generates an event.
That event invokes the Lambda function.
Lambda processes the object and stores the result.
Step 1: Create an S3 Bucket
Start by creating an S3 bucket for uploaded images.
For example:
1
image-processing-input
The bucket could contain:
1
2
3
4
uploads/
image1.jpg
image2.png
image3.jpeg
For production applications, avoid making the bucket publicly accessible unless there is a specific requirement.
Use IAM policies and appropriate bucket permissions instead.
Step 2: Create the Lambda Function
Next, create an AWS Lambda function.
For a simple example, Python can be used.
1
2
3
4
5
6
7
8
9
10
11
12
import json
def lambda_handler(event, context):
print("S3 event received")
print(json.dumps(event))
return {
"statusCode": 200,
"body": "Image processing triggered successfully"
}
At this stage, the function doesn't actually process the image.
It simply receives and prints the event.
That lets us understand what S3 sends to Lambda.
Understanding the S3 Event
When an object is created, S3 can send an event containing information about the object.
A simplified event looks like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
"Records": [
{
"eventName": "ObjectCreated:Put",
"s3": {
"bucket": {
"name": "image-processing-input"
},
"object": {
"key": "uploads/image.jpg"
}
}
}
]
}
The Lambda function can read:
- Bucket name
- Object key
- Event type
- Object information
The function can then use this information to retrieve and process the uploaded file.
Step 3: Connect S3 to Lambda
Now configure an S3 event notification.
The important event is:
1
ObjectCreated
The workflow becomes:
1
2
3
4
5
6
7
Upload image
|
v
S3 ObjectCreated event
|
v
Lambda invocation
This is the key feature that makes the architecture event-driven.
The application doesn't need to constantly check S3:
1
2
3
"Is there a new image?"
"Is there a new image?"
"Is there a new image?"
Instead, S3 tells Lambda when something happens.
Step 4: Read the Uploaded Object
The Lambda function can use the AWS SDK for Python, Boto3, to access the object.
A simplified example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import boto3
s3 = boto3.client("s3")
def lambda_handler(event, context):
record = event["Records"][0]
bucket = record["s3"]["bucket"]["name"]
key = record["s3"]["object"]["key"]
print(f"Bucket: {bucket}")
print(f"Object: {key}")
response = s3.get_object(
Bucket=bucket,
Key=key
)
image_data = response["Body"].read()
print(f"Image size: {len(image_data)} bytes")
return {
"statusCode": 200,
"body": "Image processed"
}
This example retrieves the uploaded object from S3.
From here, an image-processing library or another AWS service could be used depending on the application's requirements.
Step 5: Process the Image
The actual processing depends on the use case.
For example:
1
2
3
4
5
6
7
8
9
10
11
Original Image
|
+----> Resize
|
+----> Generate Thumbnail
|
+----> Extract Metadata
|
+----> Validate Format
|
+----> Store Result
A thumbnail workflow might look like:
1
2
3
4
5
6
7
original.jpg
|
v
Lambda
|
v
thumbnail.jpg
The processed image can then be stored in another S3 location.
Avoiding Recursive Triggers
There is an important design problem here.
Suppose Lambda reads an image from S3 and writes the processed image back to the same bucket.
If the S3 event triggers Lambda again, you could create an infinite processing loop.
1
2
3
4
5
6
7
8
9
10
11
S3 Upload
↓
Lambda
↓
S3 Output
↓
Lambda
↓
S3 Output
↓
...
That's a bad architecture.
A simple solution is to separate input and output locations.
1
2
3
4
5
6
7
Input Bucket
|
v
Lambda
|
v
Output Bucket
Another option is to use different prefixes and configure event notifications carefully.
This is a small architectural decision that can prevent a serious production problem.
IAM Permissions
Lambda needs permission to access the S3 objects.
A typical architecture requires permissions such as:
1
2
3
4
5
Lambda
|
+--> Read objects from input S3 location
|
+--> Write objects to output S3 location
The Lambda execution role should follow the principle of least privilege.
Don't give the function broad administrator permissions simply because it makes configuration easier.
For example, if Lambda only needs to read from one bucket and write to another, its role should not have unrestricted access to every S3 bucket in the AWS account.
Error Handling
Real applications need to handle failures.
Possible problems include:
- Unsupported image format
- Corrupted file
- Permission failure
- Object not found
- Processing timeout
- Unexpected event structure
The Lambda function should validate the input before processing it.
For example:
1
2
3
4
5
6
7
allowed_types = [
"image/jpeg",
"image/png"
]
if content_type not in allowed_types:
raise ValueError("Unsupported image type")
Production applications should also use appropriate logging and monitoring.
Monitoring
AWS Lambda automatically integrates with Amazon CloudWatch for logging.
Useful information to monitor includes:
1
2
3
4
5
Invocations
Errors
Duration
Timeouts
Memory usage
For example:
1
2
3
4
5
100 invocations
|
+---- 97 successful
|
+---- 3 failed
This information helps identify problems before they become larger operational issues.
When Lambda Is a Good Choice
Lambda works well when processing can be triggered by events and doesn't require a continuously running server.
Good examples include:
- File processing
- Image processing
- Data transformation
- Scheduled automation
- API backends
- Event notifications
- Lightweight ETL tasks
However, Lambda isn't automatically the best solution for every workload.
For long-running or highly specialized workloads, other compute options may be more appropriate.
Architecture should follow workload requirements rather than forcing everything into serverless.
Extending the Architecture
The basic architecture can be expanded:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
+----------------+
| User |
+-------+--------+
|
v
+---------------+
| Amazon S3 |
+-------+-------+
|
S3 Event
|
v
+---------------+
| AWS Lambda |
+-------+-------+
|
+------------+------------+
| |
v v
Process Image Store Metadata
| |
v v
Amazon S3 Database
For more complex workflows, additional services can be introduced based on the requirements.
For example:
- Amazon SQS for buffering
- Amazon EventBridge for event routing
- AWS Step Functions for multi-step workflows
- Amazon DynamoDB for metadata
- Amazon CloudWatch for monitoring
The important principle remains the same:
Let events trigger work instead of keeping infrastructure running unnecessarily.
Cost Considerations
One major advantage of serverless architecture is that you don't need to maintain a continuously running server for this type of workflow.
However, serverless does not mean free.
Costs can come from:
- S3 storage
- S3 requests
- Lambda invocations
- Lambda execution duration
- Data transfer
- Other services added to the architecture
Always monitor usage and configure appropriate billing alerts, especially when experimenting with AWS services.
Final Takeaway
An event-driven architecture can be surprisingly simple.
The core workflow is:
1
2
3
4
5
6
7
8
9
S3 Upload
↓
Event
↓
Lambda
↓
Processing
↓
Output
The real value is not the number of AWS services used.
It's the architecture.
By connecting services through events, applications can react automatically to changes without requiring a server to constantly poll for new work.
This pattern can be used as the foundation for image processing, document processing, data pipelines, automation workflows, and many other serverless applications.
Start small, understand the event flow, apply least-privilege permissions, handle failures properly, and then scale the architecture when the workload actually requires it.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article