How I Built a Serverless Event-Driven Pipeline on AWS (And What I Learned Along the Way)
When you are learning cloud architecture, building a toy app is easy—figuring out how services talk to each other asynchronously without breaking the bank is where the real learning happens. In this article, I will walk you through how I built a simple, serverless event-driven file processing pipeline using Amazon S3, AWS Lambda, Amazon SNS, and Amazon DynamoDB.
The Architecture
Here is the data flow:
- User uploads a file to an Amazon S3 bucket.
- S3 emits an ObjectCreated event.
- The event triggers an AWS Lambda function written in Python/Node.js.
- Lambda processes the metadata, logs the record into an Amazon DynamoDB table, and pushes a notification to an Amazon SNS topic.
- SNS sends an alert email to the subscriber.
Step-by-Step Implementation
Step 1: Create the DynamoDB Table
• Go to the DynamoDB Console and create a table named FileMetadata.
• Set the partition key as FileId (String).
• Set the partition key as FileId (String).
Step 2: Write the Lambda Function
Here is the core logic used inside the Lambda function to catch the S3 event and write to DynamoDB:
python
import json
import boto3
import os
python
import json
import boto3
import os
dynamodb = boto3.resource('dynamodb')
table_name = os.environ['TABLE_NAME']
table = dynamodb.Table(table_name)
table_name = os.environ['TABLE_NAME']
table = dynamodb.Table(table_name)
def lambda_handler(event, context):
for record in event['Records']:
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
size = record['s3']['object']['size']
for record in event['Records']:
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
size = record['s3']['object']['size']
1
2
3
4
5
6
7
8
table.put_item(
Item={
'FileId': key,
'BucketName': bucket,
'FileSize': size
}
)
return {'statusCode': 200, 'body': json.dumps('Processed successfully!')}Use code with caution.
Step 3: Configure the S3 Event Trigger
• Navigate to your S3 bucket -> Properties -> Event notifications.
• Create a new event notification for All object create events.
• Select your Lambda function as the destination.
• Create a new event notification for All object create events.
• Select your Lambda function as the destination.
Key Takeaways & Lessons Learned
• IAM Permissions Matter Most: Spend 10 minutes checking your execution role policies before debugging your code. Most initial failures are AccessDenied errors.
• Keep Lambda Idempotent: S3 events can occasionally trigger more than once. Ensure your database writes handle duplicate keys safely.
• Cost is Zero (or Cents): Running this under the AWS Free Tier costs virtually nothing while giving you a clear picture of event-driven patterns.
• Keep Lambda Idempotent: S3 events can occasionally trigger more than once. Ensure your database writes handle duplicate keys safely.
• Cost is Zero (or Cents): Running this under the AWS Free Tier costs virtually nothing while giving you a clear picture of event-driven patterns.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article