AWS Builder Center

Event-Driven Automation: Processing Files with Amazon S3, AWS Lambda, and Amazon DynamoDB

Build a serverless data ingestion pipeline using AWS SAM to parse S3 uploads and log record metadata automatically to DynamoDB.

ML Research & Backend Dev | Java | Python.Building robust cloud architecture for security and scale.
Educational Disclaimer: This article is intended for educational and learning purposes. Code snippets and architecture patterns presented here should be tested, secured, and adapted to meet your organization's compliance standards before production deployment. When building serverless applications, reacting to state changes in real-time is a core operational goal. Instead of relying on polling mechanisms or batch processors, an event-driven design allows cloud components to communicate asynchronously. In this tutorial, we will walk through setting up a serverless workflow that automatically extracts metadata from newly uploaded JSON files in Amazon S3, processes the payload with AWS Lambda, and logs the result into Amazon DynamoDB. By offloading file ingestion to an event-driven architecture, developers achieve automatic horizontal scaling, near-zero idle compute costs, and improved operational efficiency. Technical Architecture Overview The end-to-end data pipeline flows through three core AWS services:
Amazon S3: Acts as the object store. Upon object creation (s3:ObjectCreated:*), S3 emits an event notification. AWS Lambda: Subscribes to the S3 event, downloads the payload, extracts required metadata, and formats the record. Amazon DynamoDB: Serves as a low-latency NoSQL database to store the structured metadata.
Technical Demonstration 1: AWS SAM Infrastructure Template To maintain declarative infrastructure-as-code (IaC), we can define our AWS Lambda function, S3 bucket, DynamoDB table, and IAM permissions using an AWS Serverless Application Model (AWS SAM) template. AWSTemplateFormatVersion: '2010-09-09'Transform: AWS::Serverless-2016-10-31Description: AWS SAM Template for S3 to DynamoDB Processing Pipeline Resources: # 1. DynamoDB Table to store file metadata MetadataTable: Type: AWS::Serverless::SimpleTable Properties: TableName: ObjectMetadataStore PrimaryKey: Name: FileId Type: String # 2. S3 Bucket for file ingestion IngestionBucket: Type: AWS::S3::Bucket Properties: BucketName: !Sub "${AWS::StackName}-ingestion-bucket-${AWS::AccountId}" # 3. AWS Lambda Processor Function MetadataProcessorFunction: Type: AWS::Serverless::Function Properties: FunctionName: S3MetadataProcessor Handler: index.handler Runtime: nodejs20.x CodeUri: ./src MemorySize: 256 Timeout: 10 Environment: Variables: TABLE_NAME: !Ref MetadataTable Policies: - DynamoDBCrudPolicy: TableName: !Ref MetadataTable - S3ReadPolicy: BucketName: !Sub "${AWS::StackName}-ingestion-bucket-${AWS::AccountId}" Events: FileUploadEvent: Type: S3 Properties: Bucket: !Ref IngestionBucket Events: s3:ObjectCreated:* Filter: S3Key: Rules: - Name: suffix Value: .json Technical Demonstration 2: Lambda Event Handler Code Below is a Node.js 20.x Lambda function handler using the AWS SDK v3. It handles the incoming S3 event payload, streams the object content, parses the data, and writes the item into DynamoDB. import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";import { DynamoDBClient } from "@aws-sdk/client-dynamodb";import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb"; const s3Client = new S3Client({});const ddbClient = new DynamoDBClient({});const docClient = DynamoDBDocumentClient.from(ddbClient); export const handler = async (event) => { const tableName = process.env.TABLE_NAME; for (const record of event.Records) { const bucketName = record.s3.bucket.name; const objectKey = decodeURIComponent(record.s3.object.key.replace(/\+/g, " ")); try { // Fetch object from Amazon S3 const getObjectParams = { Bucket: bucketName, Key: objectKey }; const s3Response = await s3Client.send(new GetObjectCommand(getObjectParams)); const objectBody = await s3Response.Body.transformToString(); const parsedData = JSON.parse(objectBody); // Construct metadata item const item = { FileId: `${bucketName}/${objectKey}`, ProcessedAt: new Date().toISOString(), FileSize: record.s3.object.size, UserPayload: parsedData, }; // Write metadata item to Amazon DynamoDB await docClient.send(new PutCommand({ TableName: tableName, Item: item, })); console.log(`Successfully processed object: ${objectKey}`); } catch (error) { console.error(`Error processing object ${objectKey}:`, error); throw error; // Re-throw to trigger Lambda retry mechanisms or DLQ } }}; Technical Demonstration 3: Testing with AWS CLI To verify the event workflow without manual UI interactions, execute the following commands using the AWS CLI: 1. Create a Sample JSON Payload cat <<EOF > test-payload.json{ "environment": "dev", "project": "builder-demo", "version": "1.0.0"}EOF 2. Upload Payload to S3 Bucket aws s3 cp test-payload.json s3://<YOUR-BUCKET-NAME>/test-payload.json 3. Verify Item Creation in DynamoDB aws dynamodb get-item \ --table-name ObjectMetadataStore \ --key '{"FileId": {"S": "<YOUR-BUCKET-NAME>/test-payload.json"}}' Key Learnings & Takeaways
Decoupled Architecture: Using native S3 event integrations removes the need to write custom polling loops or maintain persistent EC2 instances. Granular IAM Policies: Granting minimal access policies (e.g., S3ReadPolicy and DynamoDBCrudPolicy) via SAM templates adheres to the principle of least privilege. Resilience Considerations: For production environments, consider attaching a Dead-Letter Queue (DLQ) using Amazon SQS to catch failed Lambda executions and prevent data loss.
Happy building! Let me know in the comments if you have any questions or how you handle file ingest pipelines in your cloud architecture.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article