Why this problem
Every winter, air quality in Indian cities can climb past 300 on the AQI scale and sometimes beyond the top of it. The number people see is a city-wide average, published after the damage has begun. We wanted to forecast severe days and give people personal advice, and that starts with good data.
The data we needed
A useful forecast needs more than air-quality readings:
- Air quality: PM2.5 and other pollutants from OpenAQ and CPCB
- Weather: wind, humidity and temperature from Open-Meteo
- Fires: crop-burning hotspots from NASA FIRMS
Each source has a different format, update rate and failure pattern.
Architecture
Ingestion. An Amazon EventBridge schedule triggers an AWS Lambda function every hour. It calls the three sources and writes the responses to Amazon S3, partitioned by source and date, for example raw/openaq/2026/10/11/14.json . Partitioning keeps Athena queries fast and cheap.
Latest values. The same function writes the newest reading per location to Amazon DynamoDB, so the app can read current conditions in milliseconds without scanning the lake.
Querying history. An AWS Glue crawler catalogs the S3 data, and Amazon Athena lets us run SQL directly on it, with no database to manage:
sql
What we learned
Sources fail, so design for it. APIs time out or return gaps. We validated each response, kept a failed source from blocking the others, and stored a freshness timestamp so the app can show how recent the data is.
Keep raw data raw. We stored responses unchanged and cleaned them downstream. When we changed our cleaning rules, we could reprocess history without refetching anything.
Powertools helped. We used Powertools for AWS Lambda for structured logs and tracing, which made it easy to see which source failed and when.
Serverless fits a hackathon. The pipeline costs almost nothing at idle, and there were no servers to patch.
Infrastructure as code
The whole pipeline is defined with the AWS CDK, so a teammate can deploy an identical copy with one command.
What’s next
We are adding low-cost PM2.5 sensors through AWS IoT Core to reach neighborhoods that official stations miss.
Loading article