AWS Builder Center

Builder Center

Building a Console-Only Real-Time Data Ingestion and Storage Pipeline on AWS

Building a Console-Only Real-Time Data Ingestion and Storage Pipeline on AWS

Data Ingestion and Data Storage

Data engineering often feels complicated when you first look at it. Streaming systems, processing frameworks, and lots of moving parts can make it hard to see how data actually flows.
For this demo, I wanted to keep things simple and focus on the basics. The goal was to show how data is ingested in real time and stored correctly on AWS using only the AWS Console. No CLI. No SDKs. No custom code.
This setup covers the first two stages of a data engineering pipeline. Data ingestion and data storage.

What this demo shows

The demo focuses on three ideas.
First, how real-time data can be ingested using a managed AWS service.
Second, how that data is stored as raw data.
Third, how processed data is organized in a data lake structure.
The services used are:
  • Amazon Kinesis Data Firehose
  • Amazon S3
The data itself is generated using the AWS Console test data feature.

Architecture overview

The flow is straightforward.
1
2
3
4
5
6
7
AWS Console Test Data
↓
Kinesis Data Firehose
↓
Amazon S3 (Raw Data)
↓
Amazon S3 (Processed Data)
This follows a common data lake pattern where data is stored first in its original form and later stored again after processing.

Ingesting data with Kinesis Data Firehose

For ingestion, the demo uses Amazon Kinesis Data Firehose.
Firehose is a good fit here because it is fully managed and works without any code. It handles scaling, buffering, and delivery automatically.
From the AWS Console:
  1. A Firehose delivery stream is created
  2. The source is set to Direct PUT
  3. The destination is set to Amazon S3
  4. Default settings are used
To send data, the console’s “Test with demo data” option is used. This sends JSON events into Firehose and simulates real-time ingestion.
At this point, data is already flowing through the pipeline.

Storing raw data in Amazon S3

Once Firehose receives the data, it delivers it directly into Amazon S3.
This data is considered raw data. It is stored exactly as it arrives, without any transformation.
A typical structure looks like this:
1
2
3
4
s3://data-lake-demo/
raw/
firehose-output/
*.json
Storing raw data is important. It allows you to reprocess data later, fix mistakes in transformations, and handle changes in schema without losing information.
This approach is commonly used in production data platforms.

Organizing processed data

In real systems, raw data is usually processed using tools like AWS Glue or Amazon EMR.
For this demo, the focus is on storage design rather than transformation logic. The processed layer is represented directly in Amazon S3.
Example structure:
1
2
3
4
5
6
s3://data-lake-demo/
raw/
json-events/
processed/
user-events/
csv or parquet files
The processed layer contains cleaned and structured data. This is the data that analytics tools and dashboards usually work with.
Keeping raw and processed data separate makes pipelines easier to maintain and extend.

Why this approach works

Even though this demo is simple, it shows several core data engineering concepts.
Data is ingested in real time.
Storage is durable and cost effective.
Raw and processed data are clearly separated.
Everything runs on fully managed AWS services.
Because it is console-only, it is also easy to explain and easy to reproduce.

How this can be extended

Once this foundation is in place, additional services can be added naturally.
AWS Glue can be used for transformations.
Amazon Athena can be used to query data in S3.
Amazon QuickSight can be used for visualization.
Those additions build on the same core pipeline shown here.

Final thoughts

You do not need a complex setup to understand data engineering on AWS. Using Amazon Kinesis Data Firehose and Amazon S3, it is possible to build a clean and practical ingestion and storage pipeline entirely from the AWS Console.
This demo provides a solid starting point for anyone learning how data flows through AWS.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article