Good Accuracy, But Does It Work in Real Life?
A practical look at why model accuracy isn't everything, using pothole detection to explore real-world challenges, prediction errors, and application reliability.
A Model Can Be Accurate and Still Be Unhelpful
While working on machine learning projects, it's easy to focus on one thing first: accuracy. A good score feels like progress, but I think there's another question worth asking — does the model actually solve the problem it was built for?
Take pothole detection as an example.
A model might detect potholes correctly in a few test images. But what happens when we use it on an actual road? The lighting may be different, the road may be wet, or a shadow might look like a pothole. Sometimes, a small pothole may not be detected at all.
These situations make me realise that getting a prediction is only one part of building a useful application.
A high accuracy score can be misleading
Imagine a dataset containing 1,000 road images, of which only 50 have potholes. A model that predicts “no pothole” for every image would still be correct 950 times. That gives it 95% accuracy, even though it misses every pothole.
This is why I wouldn't judge a model using accuracy alone.
Precision helps us understand how many predicted potholes are actually potholes. Recall tells us how many of the real potholes the model manages to find. Both are important, and the right balance depends on how the system will be used.
The model isn't the whole application
There's another part that can easily be overlooked: everything that happens after detection.
For example, a pothole detection system might save GPS coordinates, store results in a database and display them on a map. Even if the model detects the pothole correctly, an incorrect location or duplicate database entry can still make the result less useful.
Improving the model won't automatically fix these issues. Sometimes, the problem is in the application workflow rather than the prediction itself.
What I would check
Before considering a computer vision project ready to use, I'd want to check a few things:
- Does it work with different lighting conditions and camera angles?
- What kinds of objects does it mistake for potholes?
- How often does it miss smaller potholes?
- Are the detected locations stored correctly?
- Can someone actually use the results to decide what to do next?
Testing with different images and reviewing incorrect predictions can often reveal problems that a single metric won't show.
One thing worth remembering
A model can perform well during evaluation and still struggle when it meets real-world conditions. That's why I think model metrics and application testing should go together.
For a pothole detection system, the goal isn't just to draw a box around a pothole. The result should help someone identify the problem and take action.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article