AWS Builder Center
Event-Driven Data Processing: Automated S3 Ingestion with AWS Lambda and Amazon DynamoDB

Event-Driven Data Processing: Automated S3 Ingestion with AWS Lambda and Amazon DynamoDB

This technical tutorial demonstrates how to build a serverless, event-driven data processing pipeline using Amazon S3, AWS Lambda, and Amazon DynamoDB. It walks readers through automatically capturing file upload events, executing code without managing servers, and storing processed records in a scalable NoSQL database.

Overview Building real-time, scalable data ingestion pipelines can be complex when relying on traditional server-managed infrastructure. By shifting to an event-driven, serverless architecture on AWS, you can automate data processing, lower operational overhead, and ensure seamless scalability as incoming data volumes grow. This guide demonstrates how to build an automated ingestion pipeline where files uploaded to an Amazon S3 bucket trigger an AWS Lambda function to parse, validate, and store records directly into an Amazon DynamoDB table. Architecture Overview Amazon S3: Acts as the entry point for raw object uploads. AWS Lambda: Serves as the compute layer triggered automatically by s3:ObjectCreated:* events. Amazon DynamoDB: Provides high-throughput, low-latency storage for processed records. AWS IAM: Grants granular execution permissions between S3, Lambda, and DynamoDB. Key Benefits Automatic Scalability: Scales compute resources instantly without manual intervention. Cost Efficiency: Pay-per-use model means you incur costs only during Lambda function execution. Low Operational Overhead: No server provisioning, patching, or capacity management required. Technical Demonstration & Code Snippets 1. Define Execution Permissions (AWS IAM Policy) Your AWS Lambda function requires write access to Amazon DynamoDB and read/log access for S3 and CloudWatch.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": "arn:aws:s3:::my-ingestion-bucket/*" }, { "Effect": "Allow", "Action": [ "dynamodb:PutItem" ], "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/ProcessedEvents" }, { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "*" } ] }
2. Process Events in Python (AWS Lambda Handler) This Lambda function parses the incoming S3 notification payload, retrieves the target file, and writes the processed data into DynamoDB.
import json import urllib.parse import boto3 import uuid import datetime
s3_client = boto3.client('s3') dynamodb = boto3.resource('dynamodb') table = dynamodb.Table('ProcessedEvents')
def lambda_handler(event, context): # Extract Bucket and Key details from S3 event bucket = event['Records'][0]['s3']['bucket']['name'] key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8') try: # Read object content from S3 response = s3_client.get_object(Bucket=bucket, Key=key) content = response['Body'].read().decode('utf-8') data = json.loads(content) # Write record to Amazon DynamoDB table.put_item( Item={ 'EventID': str(uuid.uuid4()), 'SourceBucket': bucket, 'FileName': key, 'Payload': data, 'ProcessedAt': datetime.datetime.now(datetime.timezone.utc).isoformat() } ) return { 'statusCode': 200, 'body': json.dumps(f'Successfully processed file: {key}') } except Exception as e: print(f"Error processing key {key} from bucket {bucket}: {str(e)}") raise e
3. Configure Event Notification Trigger (AWS CLI) Attach an S3 event notification rule using the AWS Command Line Interface (CLI):
aws s3api put-bucket-notification-configuration \ --bucket my-ingestion-bucket \ --notification-configuration '{ "LambdaFunctionConfigurations": [ { "LambdaFunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:ProcessS3Ingestion", "Events": ["s3:ObjectCreated:*"] } ] }'
Important Disclaimer Educational Notice: This content is for educational and learning purposes. Configurations, permissions, and code should be reviewed and adapted before deployment in production environments.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article