AWS Builder Center
I taught an AI to label students' learning style. Here's why the honest number matters more.

I taught an AI to label students' learning style. Here's why the honest number matters more.

When my Random Forest model hit 100% accuracy, I realized it wasn’t a win, it was a red flag. A story about the critical importance of valid labels in a learning analytics project.

Series: Data (1 article)

  1. 1
    I taught an AI to label students' learning style. Here's why the honest number matters more. This article

The Day My Model Hit 100% Accuracy — and I Knew It Was a Lie

I recently built a learning-analytics system that classifies student "learning pace" using Random Forest — and the most important lesson from the entire project wasn't about trees, or kernels, or tuning. It was about labels.
Specifically, the moment I watched a model hit a perfect 100% on my test set and realized that number was a lie.



The problem: no one told me what "learning pace" means

I was working on a system called AI Learning Insight — a web platform that analyzes how students learn on a digital course platform I helped build during a capstone program with Accenture (Asah 2025). Students click through modules, submit assignments, take exams, and spend hours at different times of day. All of that behavior gets logged.
The goal was simple on paper: look at that behavior and figure out whether a student is a fast learner, a consistent learner, or a reflective learner — then serve them personalized study recommendations based on that profile.
The catch? There is no field in a database that says "this student is a fast learner." Learning pace is a construct. Nobody has ever labeled it for us.
So before I could train any classifier, I had to decide: what do these labels even mean, and who decides them?



The tempting shortcut: rules

The fastest path is rule-based labels. Something like:
  • If a student's  completion_speed  is above the median → they're a fast learner.
  • If their study consistency is high → they're a consistent learner.
  • Otherwise → reflective learner.
Simple, interpretable, and completely seductive. I actually ran this as a control experiment.
The Random Forest trained on those labels scored 100% test accuracy.
On the surface, that's the dream result every grad student wants to screenshot. But it's not a milestone — it's a red flag. The model wasn't learning anything about learning. It was replaying the rule I'd written to create the label. The tree just needed to re-find the threshold I used ( completion_speed > median → fast ) inside its own split logic. Perfect score, zero intelligence.
This is the quiet disaster of many ML projects: when the label is derived from the same features the model sees, accuracy is meaningless. It's like an exam where the answer key is printed in the margin of the question paper. Every student gets a 100, and it tells you nothing about who can actually do math.



The harder path: labels that weren't cheat-able

So I changed the approach. Instead of rule-based labels derived from the classification features, I built three independent binary scores —  fast_score ,  consistent_score ,  reflective_score  — computed from behavior criteria that did not overlap with the 10 features used for classification. Then I ran K-Means (K=3) on those binary scores to form the pace labels, validated with a silhouette score of 0.68 (above my 0.65 acceptance threshold).
Now the model couldn't cheat. It had to discover a real mapping between the classification features and the emergent clusters — something it couldn't get by re-deriving a threshold it was handed.
The result:
Metric
Score
Test Accuracy
80.92%
5-fold CV
82.91%
Lower. Deliberately lower. And dramatically more trustworthy.
I also verified the whole pipeline against data leakage:
  • The  StandardScaler  was fitted on the training set inside a single scikit-learn  Pipeline 
  • The test set was held out completely (never touched during training or pseudo-labeling)
  • The classifier features were kept independent from the variables that generated the labels



How Random Forest compared to baselines

To make sure the choice of algorithm was justified, I ran the whole thing on the exact same split (stratified 80:20,  random_state=42 ) against three baselines:
Model
Test Accuracy
Mean CV
Random Forest
80.92%
82.91%
SVM (RBF)
78.32%
74.73%
Logistic Regression
76.59%
75.96%
Decision Tree (single)
74.57%
77.70%
Random Forest won on test accuracy and was second on CV — solid, not spectacular, and honest. It's an ensemble of 200 trees ( max_depth=12 ,  class_weight='balanced' ) which keeps variance in check on this kind of small, tabular dataset.



The feature-importance plot taught me something I didn't expect

Honestly, before training, I would have guessed completion speed or average exam score would dominate. The top features turned out to be:
Feature
Importance
 completed_modules 
33.23%
 total_modules_viewed 
19.00%
 study_consistency_ratio 
13.09%
 study_consistency_std 
11.64%
 performance_score 
6.03%
The model learned that how much ground a student actually covers separates learning styles more than how quickly they cover it. That's the kind of counter-intuitive, data-driven insight a rule-based system would never surface on its own — and it's a nice reminder that trees pick whatever feature best separates the classes, not necessarily what a human would guess first.



It's not just a notebook — it's a system

Most ML projects stop at the confusion matrix. I wanted the model to actually do something.
So I deployed it end-to-end:
  • FastAPI service loads the trained  pace_classifier.pkl  and exposes  /api/v1/pace/analyze ,  /api/v1/persona/predict , and  /api/v1/advice/generate 
  • A Hapi.js / Prisma / PostgreSQL backend computes the behavioral features from user data and calls the ML service
  • A Next.js dashboard shows the student their pace profile, confidence score, study-hour distribution, and a personalized recommendation
The recommendation layer uses a Large Language Model (via OpenRouter) — but with a rule-based fallback so the feature keeps working if the external API goes down. Graceful degradation over brittle perfection.
So the pipeline is genuinely end-to-end: from 8 raw Excel files → cleaned datasets → engineered features → trained model → API → live dashboard.



What I won't hide: the honest limitations

I'm not going to pretend 80.92% is the end of the story, because the sharpest questions deserve honest answers:
  1. Only 31 unique users generated that dataset. The 1,727 samples come from (user × course) pairs, meaning rows are not fully independent. A group-based split (by user) would likely lower the accuracy — and that's the more truthful number for generalizing to new students. I couldn't fully explore that in this project, and I say so plainly.
  1. The labels are pseudo-labels. They're objectively derived, but no educational expert validated them. That's the natural next step.
  1. The LLM recommendation quality is not yet formally evaluated against user outcomes. It's a tool, not a promise.
I put all of that in the thesis as open limitations rather than hiding it, because a model that looks perfect but can't generalize is worse than a model that's honestly 80% and actually works.



Takeaway

This project made experiment validity concrete for me in a way "don't leak the target" never did on its own. Seeing a model cheat its way to 100% — then watching the honest 80.92% actually generalize — drove home that in ML, your biggest risk isn't a bad model. It's a good-looking one built on a broken question.
Before you ship that 99% accuracy, ask yourself:
Did the model learn the answer — or did you give it the answer?
The discipline is the same whether you're building heart-disease classifiers, spam filters, or systems that try to understand how humans learn: the label is the hypothesis. If the label is rigged, the accuracy is theater.

Series: Data (1 article)

  1. 1
    I taught an AI to label students' learning style. Here's why the honest number matters more. This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article