Building NeuroGait: What We Built for Someone, Not Something
A first-of-its-kind multimodal AI system combining gait video and wearable sensor data to detect Freezing of Gait episodes in Parkinson's patients, built on AWS.
Building NeuroGait: What We Built for Someone, Not Something
By Devika & Shriram — built for First Commit
They say you gotta build what you love. But this build... it wasn't for a what but for a who. Living with a grandmother who suffered from Parkinson's, I knew exactly what it was like — feeling so helpless with such an inevitable disease.
But today, thanks to the opportunity we were given and the free AWS resources, my friend and I were able to build something: NeuroGait, a multimodal AI project that analyzes gait recordings to help identify potential Freezing of Gait (FoG) episodes.
What is Freezing of Gait?
FoG is one of the most disruptive symptoms of Parkinson's — a moment where a person's movement suddenly becomes interrupted or difficult to initiate. It's sudden, it's unpredictable, and it's exactly the kind of moment I remember watching my grandmother go through. Detecting these episodes from movement data is genuinely hard, because the useful signal doesn't come from one clean source — it comes from several, and they don't always agree.
So we designed NeuroGait around two modalities: video and wearable IMU data.
From Raw Movement to FoG Episodes
Our pipeline runs walking video through MediaPipe Pose to extract body landmarks, then derives gait features from them — left and right ankle velocity, left and right knee angles, stride width. The wearable IMU adds accelerometer and gyroscope statistics on top of that.
Both modalities are temporally synchronized and fed into a Random Forest classifier, which outputs a FoG probability. That probability gets converted into temporal episodes, each with timestamps, confidence values, and interpretable movement cues — not just "an episode happened," but when, and what the model saw that made it think so.
Why AWS
Getting the model working locally was one problem. Turning it into something that could handle real data and real workloads was another. We leaned on:
- Amazon S3 for large input artifacts — the gait videos and IMU data
- Amazon ECS with AWS Fargate to run the containerized ML inference without managing servers
- Amazon DynamoDB for structured assessment results and metadata
- AWS IAM for controlled permissions across components
- Amazon CloudWatch for monitoring and logging
Instead of one service doing everything, the architecture keeps large binary data, ML computation, and structured application data cleanly separated.
The Hardest Problem: Synchronization
The least glamorous, most important problem we hit was lining up video-derived pose data with IMU measurements. Video and sensor streams don't share a clock — different sampling rates, different timestamps. Assume a fixed frame rate or sensor frequency and you'll quietly introduce timing errors that corrupt everything downstream. We had to derive the actual timing from the recordings themselves and do timestamp-based synchronization before building the final feature matrix. It was a real lesson: in multimodal ML, getting the alignment right matters as much as picking the model.
Testing Against Real Data
To validate the finished model, we used a public dataset of video, acceleration, and angular velocity from individuals with Parkinson's during a turning-in-place task . Rather than just counting detected episodes, we measured temporal overlap between our predicted episodes and the annotated ground truth, using intersection over union (IoU) alongside precision, recall, and F1. That let us check not just whether the model caught an episode, but how closely its timing matched the real one.
What This Build Taught Us
Across the full arc — dataset, feature extraction, multimodal synchronization, inference, evaluation, AWS deployment, web app — the biggest lesson was that deploying a model is a completely different job from getting one to work on your laptop. Data handling, containerization, permissions, service communication, observability — all of it becomes part of the actual engineering problem.
NeuroGait isn't a diagnostic tool, and we're not claiming clinical validation. What we wanted was something reproducible — a technical foundation that could genuinely be built on. We're still working on it, especially around real-time assessment.
But if I'm honest, the technical stack was never really the point for me. It was the closest I've come to building something that speaks to what I watched my grandmother go through — and getting to do that with AWS's resources, and my friend beside me, made it feel like more than a hackathon project.
Built with: Python · MediaPipe · NumPy · Pandas · scikit-learn · Amazon S3 · Amazon ECS · AWS Fargate · DynamoDB · IAM · CloudWatch
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article