AWS Builder Center
VayuDrishti - From Satellite Data to Cyclone Detection

VayuDrishti - From Satellite Data to Cyclone Detection

VayuDrishti AI-powered cyclone detection system on AWS using six years of IBTrACS, TRMM, GRIDSAT, and CCMP data. The journey involved solving challenges - missing observations, spatial-temporal alignment, dateline crossings, negative-sample generation, and data leakage. We transformed scientific data into 80×80 multi-channel patches and built a PyTorch CNN pipeline using Amazon S3 and SageMaker. This article covers our architecture, challenges, ML workflow, evaluation, and interactive visualization system.

When the Model Is Not the Hardest Part

Cyclone detection sounds like a straightforward machine learning problem:
Given satellite and environmental data, can a model determine whether a cyclone is present in a geographic region?
Our first instinct was to focus on the neural network.
But once we started building the system, we realized something important:
The hardest part wasn't the CNN. It was the data.
We were working with multiple scientific datasets collected in different formats, on different grids, at different timestamps, with missing observations and geographic edge cases. Before a model could learn anything, we had to turn all of that complexity into a trustworthy machine-learning dataset.
That became the real challenge—and ultimately the most interesting part of our project.
In this article, we walk through how we built an end-to-end cyclone detection pipeline using IBTrACS, TRMM, GRIDSAT, CCMP, Amazon S3, Amazon SageMaker, Python, and PyTorch, from raw scientific data all the way to an interactive ML application.



1. The Goal

Our objective was to build a binary classifier capable of distinguishing between:
1 → Cyclone
0 → Non-Cyclone
Rather than relying on a single variable such as rainfall or wind speed, we wanted the model to learn from multiple environmental signals simultaneously.
Each sample represents a 20° × 20° geographic region sampled at 0.25° resolution.
That gives us:
80 × 80 spatial grid
with 10 input channels.

Our input channels

Channel
Description
precipitation
Precipitation information
uwnd
U-component of wind
vwnd
V-component of wind
ws
Wind speed
nobs
Observation information
irwin_cdr
Infrared window data
irwin_2
Additional infrared feature
irwvp
Infrared water-vapor information
irwvp_2
Additional water-vapor feature
vschn
Visible-channel information
The resulting model input can therefore be represented as:
10 × 80 × 80
Instead of treating satellite data as isolated images, we treat each geographic observation as a multi-channel environmental snapshot.



2. The Data Behind the Model

Our dataset combines four major sources:
IBTrACS
+
TRMM
+
GRIDSAT
+
CCMP
Each dataset contributes a different piece of information.

IBTrACS — Where Are the Cyclones?

We used IBTrACS as the reference source for historical tropical cyclone observations.
It provided information such as:
  • storm identifier
  • observation timestamp
  • latitude
  • longitude
  • historical track
Across 2012–2017, our master IBTrACS dataset contained:
37,156 observations.
This dataset became the reference for defining positive cyclone observations and validating the generation of negative samples.



TRMM — Precipitation

TRMM provided precipitation-related information.
One complication was that the HDF4 data did not directly provide coordinates in the exact structure required by our preprocessing pipeline.
We therefore had to derive the appropriate spatial grid before extracting geographic patches.



GRIDSAT — Satellite Information

GRIDSAT supplied important satellite-derived infrared and related features.
This required careful handling of:
  • scale factors
  • offsets
  • fill values
  • dense variables
  • sparse variables
  • coordinate conventions
  • missing observations
During our validation process, we also identified missing GRIDSAT observations and recovered them before continuing with dataset construction.
That validation step was important because silently dropping observations would have changed the composition of the dataset.



CCMP — Wind Information

CCMP provided wind-related environmental information.
However, CCMP observations do not necessarily occur at exactly the same timestamps as cyclone observations.
We therefore implemented explicit temporal matching rather than assuming all datasets were synchronized.



3. Our AWS Architecture

The overall system was designed as a pipeline rather than a collection of disconnected notebooks.
┌──────────────────┐
│ IBTrACS │
│ Cyclone Tracks │
└─────────┬────────┘
│
┌───────────────────┼───────────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌──────────┐ ┌─────────┐
│ TRMM │ │ GRIDSAT │ │ CCMP │
│Rainfall │ │Satellite │ │ Wind │
└────┬────┘ └────┬─────┘ └────┬────┘
│ │ │
└───────────────────┼───────────────────┘
▼
┌──────────────────┐
│ Amazon S3 │
│ Raw Data │
└────────┬─────────┘
▼
┌──────────────────┐
│ Preprocessing │
│ │
│ Spatial Align. │
│ Temporal Align. │
│ Quality Checks │
└────────┬─────────┘
▼
┌──────────────────┐
│ 80 × 80 × 10 │
│ ML Patches │
└────────┬─────────┘
▼
┌──────────────────┐
│ Train / Val / │
│ Test Datasets │
└────────┬─────────┘
▼
┌──────────────────┐
│ Amazon SageMaker │
└────────┬─────────┘
▼
┌──────────────────┐
│ PyTorch CNN │
└────────┬─────────┘
▼
┌──────────────────┐
│ Prediction + │
│ Evaluation │
└────────┬─────────┘
▼
┌──────────────────┐
│ Frontend & │
│ Visualization │
└──────────────────┘

AWS components

Our main cloud components included:
  • Amazon S3 — raw and processed dataset storage
  • Amazon SageMaker — ML development and training
  • EC2 GPU infrastructure — explored for accelerated training
  • AWS Region:  us-west-2 
The important architectural decision was to keep data preparation independent from model training.
That meant we could change the compute environment without rebuilding the entire dataset.



4. Designing the S3 Data Lake

We separated raw data from processed data.
Our S3 structure was organized approximately as:
s3://cyclone-12-09-2026/
│
├── raw/
│ ├── trmm/
│ ├── gridsat/
│ │ ├── 2012/
│ │ ├── 2013/
│ │ ├── ...
│ │ └── 2017/
│ └── ccmp/
│ ├── 2012/
│ ├── ...
│ └── 2017/
│
├── processed/
│ ├── patches/
│ │ ├── positive/
│ │ └── negative/
│ ├── final_nc/
│ ├── final_splits/
│ ├── normalization/
│ ├── quality/
│ └── metadata/
│
└── code/
This structure made it easier to:
  • trace data lineage
  • debug preprocessing
  • validate yearly datasets
  • regenerate processed data
  • separate source data from derived datasets



5. Turning Observations Into ML Patches

Once the source datasets were aligned, we converted observations into standardized geographic patches.
Our configuration was:
Patch size: 20° × 20°
Resolution: 0.25°
Grid size: 80 × 80
Input channels: 10
Years: 2012–2017
Every patch was stored with useful metadata:
label
center_lat
center_lon
year
sample_id
sid
iso_time
This was important because the ML tensor alone does not tell us where or when the observation occurred.
The metadata allowed us to later connect predictions back to actual cyclone tracks and geographic locations.



6. One of the Hardest Problems: The International Date Line

One of our most interesting data-engineering challenges involved the International Date Line.
Consider these two longitudes:
+179°
-179°
Geographically, they are only about two degrees apart.
But a naive numerical calculation can interpret them as being almost 360 degrees apart.
Our dataset contained 2,561 dateline-crossing cases across the six-year period.
We therefore had to make our longitude handling aware of geographic wrapping.
This prevented the patch extraction process from incorrectly treating nearby locations as being on opposite sides of the world.
It was a good reminder that:
Geospatial machine learning requires geographic reasoning, not just numerical processing.



7. Building the Negative Class

Positive samples were relatively straightforward because they came from known cyclone observations.
Negative samples were more challenging.
We could not simply pick random coordinates and call them non-cyclones.
A random location might be close to an actual storm.
We therefore applied an exclusion rule:
A negative sample had to be more than 15° away from any IBTrACS storm-track point within ±24 hours.
This helped us construct a cleaner background class.
Our final dataset contained:
Positive: 37,155
Negative: 37,155
-------------------
Total: 74,310
The approximately balanced dataset provided a useful starting point for binary classification.



8. Preventing Data Leakage

This was one of the most important decisions in our entire pipeline.
Cyclone observations are not independent samples.
A single storm can generate hundreds of observations:
Storm A
├── Observation 1
├── Observation 2
├── Observation 3
├── ...
└── Observation 200
If we randomly split individual observations, the same storm could appear in both training and testing.
The model could then benefit from seeing extremely similar observations during training.
That would make the final evaluation misleading.

Our solution

For positive observations, we grouped samples using:
Storm ID (SID)
For negative samples, we grouped using:
Observation timestamp
We then generated a global split assignment targeting:
70% Training
15% Validation
15% Testing
Our final split was:
Training: 52,016
Validation: 11,145
Testing: 11,149
-------------------
Total: 74,310
This was a major part of making the evaluation meaningful.



9. Validating the Dataset Before Training

Before training the CNN, we performed a series of validation checks.
We verified:
✓ Raw file counts
✓ Missing files
✓ Zero-byte files
✓ Coordinate ranges
✓ Patch dimensions
✓ Channel availability
✓ Fill values
✓ NaN values
✓ Infinite values
✓ Labels
✓ Class balance
✓ Split sizes
✓ Normalization statistics
✓ DataLoader output
We also ran a DataLoader smoke test.
The expected batch structure was:
X shape: [32, 10, 80, 80]
y shape: [32]
with:
X → float32
y → int64
We verified that feature values were finite and labels were valid binary values.
This kind of validation may look small, but it prevents a surprisingly large number of ML failures.



10. Why We Used NetCDF

Scientific datasets can become extremely large.
Our processed data was several gigabytes, so we needed a structured format that preserved scientific dimensions and metadata.
We consolidated the processed data into NetCDF datasets.
The training dataset has:
sample = 52,016
y = 80
x = 80
Instead of loading the complete dataset into memory, we use batch-wise loading during training.
This allows the model to process the data incrementally.



11. Normalization

The ten input channels naturally have different numerical ranges.
We therefore calculated channel-wise normalization statistics using the training dataset only.
The statistics were stored separately and reused for validation and inference.
This ensures that information from the validation or test distributions does not influence training preprocessing.



12. The CNN

Once the dataset was ready, we built a lightweight convolutional neural network.
The architecture was:
Input
10 × 80 × 80
│
▼
Conv Block
10 → 32
│
▼
Conv Block
32 → 64
│
▼
Conv Block
64 → 128
│
▼
Conv Block
128 → 192
│
▼
Adaptive Average Pooling
│
▼
Flatten
│
▼
Dropout 0.30
│
▼
Linear
192 → 1
│
▼
Cyclone Logit
The network contains approximately:
317,249 trainable parameters.
We deliberately kept the model relatively small.
Our goal was to create a model that could be trained and iterated on efficiently while still learning meaningful spatial patterns from the ten environmental channels.



13. Training Configuration

Our training setup included:
Framework: PyTorch
Input channels: 10
Batch size: 32

Optimizer: AdamW
Learning rate: 0.001
Weight decay: 0.0001

Loss: BCEWithLogitsLoss

Dropout: 0.30
Maximum epochs: 12
Early stopping: patience = 3
We also used validation-based learning-rate adjustment and retained the best validation checkpoint.



14. Why Batch-Wise Loading Matters

The training dataset is several gigabytes.
Loading the entire dataset into memory would be unnecessary.
Instead, our training loop processes batches:
for X, y in train_loader:

optimizer.zero_grad()

logits = model(X)

loss = criterion(logits, y)

loss.backward()

optimizer.step()
With a batch size of 32, each iteration works on a manageable portion of the dataset.
This approach also made our training workflow more portable across different compute environments.



15. Amazon SageMaker and the GPU Challenge

We used Amazon SageMaker as our ML development environment.
Our SageMaker workflow allowed us to:
  • access the prepared datasets
  • validate the DataLoader
  • run PyTorch experiments
  • train the CNN
  • save checkpoints
  • analyze metrics
We initially explored GPU-based training, but encountered AWS GPU quota limitations.
Rather than stopping development while infrastructure was being resolved, we continued using a CPU-based SageMaker notebook.
This became an important engineering lesson:
A robust ML pipeline should not be tightly coupled to one type of compute.
Because our data pipeline and training code were separated, we could continue development without rebuilding the dataset.



16. Evaluating the Model

We did not want to rely only on accuracy.
Our evaluation includes:
Accuracy
Precision
Recall
Specificity
F1 Score
ROC-AUC
PR-AUC
We also use the validation dataset to tune the classification threshold instead of automatically assuming:
Probability > 0.5 → Cyclone
The test dataset remains reserved for final evaluation.



17. Final Test Results

This section should be updated after the final test evaluation.
Accuracy: XX.XX%
Precision: XX.XX%
Recall: XX.XX%
Specificity: XX.XX%
F1 Score: XX.XX%
ROC-AUC: XX.XX
PR-AUC: XX.XX
We intentionally keep these numbers separate from development experiments so that the final test evaluation remains meaningful.



18. From Model to Application

A trained model is only part of the solution.
We wanted to make the output understandable through an interactive interface.
Our frontend is designed around several components.

Cyclone Detection

A user can select a sample and see:
Prediction: Cyclone / Non-Cyclone
Probability: XX.XX%
Latitude: XX°
Longitude: XX°
Timestamp: YYYY-MM-DD HH:MM



Satellite Data Viewer

The interface can display the ten model input channels as interactive heatmaps.
Users can explore:
  • precipitation
  • wind components
  • wind speed
  • infrared channels
  • water-vapor channels
  • visible-channel information
This helps connect the model's numerical input to something visually understandable.



Cyclone Map

The map can display:
  • cyclone center
  • geographic analysis region
  • latitude
  • longitude
  • historical track
  • timestamp
  • storm metadata



Explainability

We also designed an explainability layer to visualize spatial regions that contribute strongly to the model's prediction.
Instead of treating the model as a black box, the goal is to help users understand:
Where is the model finding evidence for its prediction?



Model Performance Dashboard

The application can expose:
Accuracy
Precision
Recall
Specificity
F1
ROC-AUC
PR-AUC
Confusion Matrix
ROC Curve
Precision-Recall Curve



19. Historical Cyclone Explorer

The frontend also provides a way to explore historical observations.
Users can filter by:
Year
Storm
Basin
Date
Location
and inspect:
  • historical cyclone track
  • observation location
  • environmental patch
  • actual label
  • model prediction
  • prediction probability
This creates a bridge between the machine learning system and the underlying scientific observations.



20. What We Actually Fought

This project taught us that real-world ML is rarely:
Dataset → CNN → Accuracy
Instead, our workflow looked more like:
Raw Data
↓
Missing Data
↓
Format Problems
↓
Coordinate Alignment
↓
Temporal Alignment
↓
Dateline Edge Cases
↓
Negative Sampling
↓
Leakage Prevention
↓
Quality Validation
↓
Large Dataset Management
↓
Cloud Infrastructure
↓
Training
↓
Evaluation
↓
Visualization
Some of the most important challenges were:

Missing observations

We had to identify and recover missing satellite data rather than silently dropping samples.

Scientific formats

We worked with NetCDF and HDF4 and had to correctly handle scale factors, offsets, fill values, and coordinate structures.

Spatial alignment

Different datasets use different grids.

Temporal alignment

Different sources do not necessarily provide observations at identical timestamps.

Dateline handling

Geographic longitude wrapping required dedicated logic.

Data leakage

Random sample-level splitting was not appropriate for observations belonging to the same storm.

Large datasets

Several gigabytes of training data required efficient batch-based processing.

Cloud constraints

GPU quota limitations required us to maintain a CPU-compatible path.
These challenges were not side problems.
They were a core part of building the machine-learning system.



21. Reproducibility

We wanted our experiments to be reproducible rather than dependent on a single notebook run.
We tracked:
Dataset version
Split seed
Training seed
Model architecture
Learning rate
Batch size
Optimizer
Loss function
Normalization statistics
Training configuration
Our split generation used:
Seed = 20260918
and training used:
Seed = 20260919
This makes experiments easier to reproduce and compare.



22. End-to-End Pipeline

Putting the complete system together:
┌──────────────┐
│ IBTrACS │
└──────┬───────┘
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
TRMM GRIDSAT CCMP
│ │ │
└────────────────┼────────────────┘
▼
┌───────────────┐
│ Amazon S3 │
│ Raw Data │
└───────┬───────┘
▼
┌───────────────┐
│ Preprocessing │
│ │
│ Spatial │
│ Temporal │
│ Dateline │
│ Quality │
└───────┬───────┘
▼
┌───────────────┐
│ Positive + │
│ Negative │
│ Samples │
└───────┬───────┘
▼
┌───────────────┐
│ 80 × 80 × 10 │
│ ML Patches │
└───────┬───────┘
▼
┌───────────────┐
│ Train / Val / │
│ Test │
└───────┬───────┘
▼
┌───────────────┐
│ SageMaker │
│ + PyTorch │
└───────┬───────┘
▼
┌───────────────┐
│ CNN │
└───────┬───────┘
▼
┌───────────────┐
│ Prediction │
│ + Evaluation │
└───────┬───────┘
▼
┌───────────────┐
│ Interactive │
│ Frontend │
└───────────────┘



23. What We Learned

The biggest lesson from this project was:
Machine learning starts with data engineering, not the model.
A CNN can be written in a relatively small amount of code.
Building a trustworthy dataset from multiple scientific sources is a much larger engineering problem.
We learned the importance of:
  • scientific data formats
  • spatial coordinates
  • temporal matching
  • missing-data handling
  • geographic edge cases
  • negative-sample quality
  • leakage prevention
  • normalization
  • reproducibility
  • memory management
  • cloud infrastructure
  • explainability
We also learned that infrastructure constraints should not stop experimentation.
A well-designed pipeline should allow the same processed dataset and training workflow to move between different compute environments.



24. What's Next?

The current pipeline gives us a foundation for several future improvements.

More advanced architectures

We can explore deeper CNNs, attention mechanisms, and other spatial architectures.

Temporal modeling

Instead of treating each observation independently, future versions could use sequences of satellite observations.

Better explainability

We can improve spatial explanations and provide more detailed visual feedback.

Larger datasets

The pipeline can be extended to additional years and geographic regions.

Production inference

A future version could expose the trained model through an inference API and connect it directly to the frontend.



Conclusion

Our project started with a simple question:
Can machine learning detect cyclones from satellite and environmental data?
Answering that question required much more than training a neural network.
We built an end-to-end pipeline combining IBTrACS, TRMM, GRIDSAT, and CCMP, processed six years of observations into standardized 80×80 multi-channel patches, stored the data using Amazon S3, developed the ML workflow using Amazon SageMaker and PyTorch, and designed an interactive interface for exploring predictions and environmental data.
The most important result of the project was not simply a trained classifier.
It was learning how to build a machine-learning system that is:
data-aware, leakage-conscious, reproducible, cloud-ready, and understandable.
The journey from raw satellite observations to a working ML system showed us that the model is only one piece of the puzzle.
The real work happens before, around, and after the model.
And that is what made this project a true end-to-end machine learning engineering challenge on AWS.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article