From Physical Sensors to AWS: Building an End-to-End IoT Data Pipeline

Most data projects begin with a CSV file, an API, or a dataset that already exists. For this one, I wanted to start one step earlier: where the data is actually created.

The goal was to build a small environmental monitoring system that could measure real conditions, react locally to them, send telemetry securely to AWS, preserve the original data, process it for analysis, and eventually detect unusual behavior.

The final flow looked like this:

What interested me most was not any individual technology in that chain, but what happens when all of them have to work together.

The device is based on an ESP32 connected to a DHT11 sensor for temperature and humidity, an LDR for changes in ambient light, and an LED used as a simple actuator.

The ESP32 is not only collecting data. It also makes one decision locally: when the light reading falls below a defined threshold, the LED is turned on. When the environment becomes bright again, it is turned off.

I deliberately kept this logic on the device instead of sending every decision to the cloud. The ESP32 does not need AWS to decide whether the LED should be active, which makes this a simple example of edge processing: the response happens close to where the data is generated.

At the same time, each reading is enriched with information that later becomes useful for debugging and data quality, including the device ID, session, boot ID, sequence number, uptime, Wi-Fi signal strength, firmware version, sensor validity and the state of the actuator.

That turned out to be important later. Capturing only temperature and humidity would have been enough to draw a chart, but not enough to understand what happened when a device restarted, a sequence changed, or a particular run produced unexpected records.

Once a reading is created, the ESP32 serializes the telemetry as JSON and publishes it using MQTT over TLS to AWS IoT Core.

The device authenticates using an X.509 certificate, while an AWS IoT policy defines what the device is allowed to do. This means the firmware does not need an AWS access key and secret key embedded inside it.

On the AWS side, an IoT Rule receives the incoming telemetry and writes the messages into Amazon S3.

I chose S3 as the historical landing zone because I wanted to preserve the original messages exactly as they arrived. Instead of immediately transforming or aggregating the telemetry, the system first keeps an immutable raw layer that can be revisited later if something in the processing logic changes.

That gave me a simple but useful principle for the rest of the project:

A pipeline that appears to work is not necessarily a pipeline that is observable.

I used Amazon CloudWatch to monitor whether the ESP32 was connecting successfully, whether messages were reaching AWS IoT Core, whether the IoT Rule was executing, and whether the S3 action was succeeding or failing.

This made it possible to follow the path of a message rather than only checking whether a JSON file eventually appeared in S3.

For example, if a record did not reach storage, I could distinguish between a connection problem, a publishing problem, a rule execution problem, or an S3 action failure.

For a small project this may look like extra work, but it changed the way I thought about the system. The telemetry itself tells me what is happening in the physical environment; CloudWatch tells me whether the infrastructure carrying that telemetry is healthy.

Once the telemetry was stored in S3, I moved to the data engineering side of the project.

I used Boto3, the AWS SDK for Python, to connect to S3 and download the JSON objects into the local processing pipeline. Access from Python was controlled through IAM, keeping cloud permissions separate from the device authentication used by AWS IoT Core.

The data was then processed using Python and Polars.

The initial dataset contained 319 raw records. After validation, 316 records remained in the main curated dataset, while 3 records were placed in quarantine.

Those three records were not corrupt. They belonged to secondary device boots within sessions where another boot contained most of the observations. Instead of deleting them or silently mixing them with the primary run, I kept them in a separate quarantine dataset with the reason for their exclusion.

That distinction became one of the parts of the project I found most useful. Data quality is not only about deciding whether a record is valid or invalid. Sometimes a record is technically valid but does not belong in the analytical dataset being built.

Keeping the raw, curated and quarantined layers separate gave me a way to preserve that history.

Not all sessions were captured at the same frequency.

The normal baseline sessions produced approximately one reading per minute, while some controlled experiments generated readings much more frequently. Comparing those datasets directly would have made the high-frequency experiments disproportionately influential.

Before analysis, I therefore resampled the sessions to a common one-minute time interval.

That produced 135 normalized observations, of which 131 were used in the machine-learning-ready dataset after removing the first observation of each session where a previous value was required to calculate changes.

This step was less visible than connecting an ESP32 to AWS, but it was essential. Without normalizing time, many of the changes detected later could have reflected differences in sampling frequency rather than actual changes in the environment.

For historical analysis I used DuckDB and SQL over the processed Parquet files.

Instead of looking only at absolute temperature, humidity and light values, I also created features describing how those measurements changed over time.

For example, a temperature of 30°C may be perfectly normal for a particular environment. A rapid change from 27°C to 30°C in a short period may be much more interesting.

The same idea applies to light. During a controlled lighting experiment, the absolute values changed sharply compared with the relatively stable baseline sessions. Creating change-based features made those differences much easier to detect.

This is why the anomaly detection stage primarily used the magnitude of changes in temperature, humidity and light rather than variables such as the LED state. Using the actuator state itself would have made it too easy for the model to indirectly learn a decision that was already encoded in the firmware.

For anomaly detection I used two different approaches.

The first is a robust statistical detector based on the normal baseline session. It measures how far a new observation deviates from the typical behavior seen during that baseline.

The second is Isolation Forest, an unsupervised machine learning algorithm trained only on the normal baseline data. Instead of being told which observations are anomalies, it learns the structure of the normal data and assigns more unusual observations different anomaly scores.

I then added a consensus rule: an observation receives the strongest anomaly flag when both methods agree.

The results were useful because they matched the behavior I expected from the controlled experiments.

In simple terms, when the environment remained relatively stable, the system raised very few flags. When I deliberately introduced a strong change in lighting, the number of unusual observations increased substantially.

I intentionally describe these values as flag rates rather than accuracy, precision or recall. The dataset does not contain a minute-by-minute ground truth label telling the model exactly which individual observations must be considered anomalous, so presenting the results as classification accuracy would overstate what the experiment proves.

The final layer was a local dashboard built with Streamlit, using Polars for data handling and Plotly for visualization.

The dashboard makes it possible to explore temperature, humidity, light levels, LED activity, Wi-Fi signal strength and approximate AWS ingestion latency. It also includes historical session comparisons, data-quality information and the outputs of both anomaly detectors.

At that point, the original sensor readings had completed the entire path I wanted to build:

I also wanted to understand the operational side of the architecture, even at this small scale.

Using AWS Pricing Calculator, I estimated a scenario with one ESP32 publishing one message per minute, 24 hours a day for 30 days.

For the AWS services included in the architecture, the estimated list-price cost was approximately USD $3.29 per month.

The point of this estimate was not to claim an exact production cost, but to connect the technical architecture with something that is easy to overlook while building: every cloud design eventually has an operational cost.

The biggest takeaway was that the interesting problems often appear between technologies.

Connecting a sensor is one problem. Publishing to AWS is another. Storing JSON in S3 is straightforward on its own. But once the system has to preserve device history, distinguish different boots, compare experiments sampled at different frequencies, protect credentials, track failed cloud actions and avoid misleading machine-learning metrics, the project becomes much more than the sum of those pieces.

Building the complete path helped me understand why concepts such as observability, data lineage, temporal normalization and separation between raw and processed data matter even in a relatively small IoT system.

The complete implementation, architecture, electronic schematic and data pipeline are available in the GitHub repository:

GitHub: https://github.com/Jose3AM/OfficePulse