Overview
Building real-time, scalable data ingestion pipelines can be complex when relying on traditional server-managed infrastructure. By shifting to an event-driven, serverless architecture on AWS, you can automate data processing, lower operational overhead, and ensure seamless scalability as incoming data volumes grow.
This guide demonstrates how to build an automated ingestion pipeline where files uploaded to an Amazon S3 bucket trigger an AWS Lambda function to parse, validate, and store records directly into an Amazon DynamoDB table.
Architecture Overview
Amazon S3: Acts as the entry point for raw object uploads.
AWS Lambda: Serves as the compute layer triggered automatically by s3:ObjectCreated:* events.
Amazon DynamoDB: Provides high-throughput, low-latency storage for processed records.
AWS IAM: Grants granular execution permissions between S3, Lambda, and DynamoDB.
Key Benefits
Automatic Scalability: Scales compute resources instantly without manual intervention.
Cost Efficiency: Pay-per-use model means you incur costs only during Lambda function execution.
Low Operational Overhead: No server provisioning, patching, or capacity management required.
Technical Demonstration & Code Snippets
1. Define Execution Permissions (AWS IAM Policy)
Your AWS Lambda function requires write access to Amazon DynamoDB and read/log access for S3 and CloudWatch.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::my-ingestion-bucket/*"
},
{
"Effect": "Allow",
"Action": [
"dynamodb:PutItem"
],
"Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/ProcessedEvents"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
2. Process Events in Python (AWS Lambda Handler)
This Lambda function parses the incoming S3 notification payload, retrieves the target file, and writes the processed data into DynamoDB.
import json
import urllib.parse
import boto3
import uuid
import datetime
s3_client = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('ProcessedEvents')
def lambda_handler(event, context):
# Extract Bucket and Key details from S3 event
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
try:
# Read object content from S3
response = s3_client.get_object(Bucket=bucket, Key=key)
content = response['Body'].read().decode('utf-8')
data = json.loads(content)
# Write record to Amazon DynamoDB
table.put_item(
Item={
'EventID': str(uuid.uuid4()),
'SourceBucket': bucket,
'FileName': key,
'Payload': data,
'ProcessedAt': datetime.datetime.now(datetime.timezone.utc).isoformat()
}
)
return {
'statusCode': 200,
'body': json.dumps(f'Successfully processed file: {key}')
}
except Exception as e:
print(f"Error processing key {key} from bucket {bucket}: {str(e)}")
raise e
3. Configure Event Notification Trigger (AWS CLI)
Attach an S3 event notification rule using the AWS Command Line Interface (CLI):
aws s3api put-bucket-notification-configuration \
--bucket my-ingestion-bucket \
--notification-configuration '{
"LambdaFunctionConfigurations": [
{
"LambdaFunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:ProcessS3Ingestion",
"Events": ["s3:ObjectCreated:*"]
}
]
}'
Important Disclaimer
Educational Notice: This content is for educational and learning purposes. Configurations, permissions, and code should be reviewed and adapted before deployment in production environments.
Loading article