AWS Builder Center

Builder Center

Building a Serverless Streaming Data Pipeline on AWS

Building a Serverless Streaming Data Pipeline on AWS

Built a serverless streaming data pipeline using Firehose, Lambda, S3, Glue, Athena, and DynamoDB.

Trần Đồng

Cloud Engineer | AWS | Generative AI & Agentic AI
I recently completed an AWS SimuLearn lab focused on data ingestion and streaming analytics.
What I liked most about this lab was seeing several AWS services work together as one complete data pipeline instead of learning each service separately.
Before going deeper, this is how I now think about each component:
1
2
3
4
5
6
7
Producer = generates streaming data
Firehose = ingests, buffers, and delivers data
Lambda = processes and transforms records
S3 = stores data in the data lake
Glue = catalogs data and defines schema
Athena = queries data in S3 with SQL
DynamoDB = provides low-latency data access

Architecture

The overall flow looked like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Producer
|
v
Amazon Data Firehose
|
+----> AWS Lambda ----> Amazon S3
| |
| v
| AWS Glue
| |
| v
| Amazon Athena
|
+----> AWS Lambda ----> Amazon DynamoDB
This architecture gives the incoming data two useful paths:
1
2
3
4
5
Analytical path:
Firehose → Lambda → S3 → Glue → Athena

Operational path:
Firehose → Lambda → DynamoDB

1. Ingesting Data with Amazon Data Firehose

The producer generates clickstream data and sends records to Amazon Data Firehose.
One important thing I learned is that Firehose is not really the final storage location.
I think of it more like a managed delivery pipeline:
1
Producer → Firehose → Destination
Firehose can receive streaming records, buffer them, invoke Lambda for transformation, retry delivery when necessary, and deliver data to destinations such as Amazon S3.
The producer can also use APIs such as PutRecordBatch to send multiple records together instead of making one API call for every event.
This can reduce API overhead and improve throughput.

2. Buffering Before Delivery

Another concept that became clearer during the lab was buffering.
Instead of immediately creating an S3 object for every incoming event, Firehose can collect records until a configured size or time threshold is reached.
For example:
1
2
3
4
5
6
7
8
9
Incoming records
↓
Firehose
↓
Buffer records
↓
Deliver a larger batch
↓
S3
This helps avoid creating too many tiny files in S3, which can make downstream analytics less efficient.

3. Transforming Records with AWS Lambda

Firehose can invoke AWS Lambda before delivering records to S3.
In the lab, the Lambda function received records from Firehose and processed them in memory.
The flow was roughly:
1
2
3
4
5
6
7
8
9
10
11
12
13
Firehose record
↓
Base64 decode
↓
Parse incoming data
↓
Transform the record
↓
Convert to CSV
↓
Base64 encode
↓
Return to Firehose
This also helped me understand Lambda's event parameter better.
The structure of event depends on the service that invokes the function.
For example:
1
2
3
API Gateway → one event structure
S3 → another event structure
Firehose → another event structure
So when building a Lambda function, understanding the input event schema is very important.

4. Storing Data in Amazon S3

After transformation, Firehose delivers the processed records to Amazon S3.
S3 acts as the durable storage layer for the analytical pipeline.
A useful way to think about this is:
1
2
Storage → Amazon S3
Compute → Lambda / Athena / other services
The data can stay in S3 while different services independently analyze or process it.
This separation between compute and storage is one of the patterns that makes cloud-native data architectures flexible.

5. Cataloging the Data with AWS Glue

Once files are stored in S3, we still need a way to understand their structure.
This is where AWS Glue comes in.
A Glue Crawler can inspect data stored in S3, infer its schema, and store the metadata in the AWS Glue Data Catalog.
For example, a clickstream dataset might contain fields such as:
1
2
3
4
prev
curr
type
n
Glue allows other analytics services to understand those columns and their data types.
I now think of Glue in this architecture as:
1
2
S3 = stores the actual data
Glue = stores information about the data

6. Querying S3 with Amazon Athena

After the table metadata is available in the Glue Data Catalog, Amazon Athena can query the S3 data using SQL.
For example:
1
2
3
SELECT type, COUNT(*)
FROM clickstream
GROUP BY type;
Athena does not require me to provision or manage a traditional database server for this use case.
The relationship is simple:
1
2
3
S3 = data
Glue = schema / metadata
Athena = SQL query engine
This is also a good example of schema-on-read.
The data remains as files in S3, and its structure is interpreted when it is queried.

7. Adding DynamoDB for Fast Data Access

The DIY section added another interesting flow.
A Lambda function received records from Firehose, decoded the data, and stored the results in Amazon DynamoDB.
1
2
3
4
5
Firehose
↓
Lambda
↓
DynamoDB
While the S3 + Glue + Athena path is useful for analytical workloads, DynamoDB is designed for applications that require predictable, low-latency access.
This means the same streaming pipeline can support different use cases:
1
2
3
4
5
6
7
S3 + Glue + Athena
↓
Historical / analytical queries

DynamoDB
↓
Fast operational access
For example, DynamoDB could be used by a live dashboard or an application that needs to quickly retrieve the latest processed values.

What I Learned

The biggest takeaway from this lab was not memorizing individual AWS services.
It was understanding the responsibility of each component:
1
2
3
4
5
6
7
Producer = generates data
Firehose = receives and delivers the stream
Lambda = processes and transforms data
S3 = stores historical data
Glue = describes the data
Athena = analyzes the data with SQL
DynamoDB = serves data with low latency
Once those responsibilities became clear, the architecture became much easier to understand.
Instead of only asking:
Which AWS service should I use?
I am starting to ask:
What responsibility does this part of the architecture need to handle?
That way of thinking makes it easier to understand how AWS services fit together when designing real systems.
This lab was a useful hands-on introduction to building a serverless streaming data pipeline on AWS, covering the full flow from ingestion and transformation to storage, analytics, and low-latency access.
Lab: AWS SimuLearn – Data Ingestion Methods
https://skillbuilder.aws/learn/WT478HCMQE/aws-simulearn-data-ingestion-methods/MMQ3HG3UM1?parentId=67BBEU3C7N 
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article