AWS Builder Center

Decision Trees vs SVMs on Heart Disease Data: What Overfitting Actually Looks Like

Built and compared Decision Trees and SVMs from scratch on the UCI Heart Disease dataset. Here's what I learned about overfitting, pruning, and kernel tricks along the way.

Machine Learning
I recently built and compared Decision Trees and SVMs (linear, RBF, polynomial kernels) from the ground up on the UCI Heart Disease dataset (303 patients, 13 clinical features), and the results taught me more about overfitting than any textbook chapter did.
Decision Tree, unconstrained: 100% training accuracy, 80.3% test accuracy. Classic overfitting — the tree memorized the training set instead of learning generalizable patterns.
Decision Tree, pruned (max_depth=5): 94.3% train, 80.3% test. Same test accuracy as the unconstrained tree, but with a huge drop in train accuracy — meaning the extra depth in the unconstrained tree was pure memorization, adding zero real predictive power. Pruning didn't cost me anything on unseen data.
Interestingly, the tree's root split was on the number of major vessels (a categorical feature), not a continuous measurement like cholesterol or blood pressure — a good reminder that trees pick whatever feature best separates classes, not necessarily what a human would guess first.
SVMs, three kernels:
  • Linear: 85.5% test
  • Polynomial: 86.8% test
  • RBF: 88.2% test — the best performer overall
The RBF kernel edging out linear and polynomial suggests the true decision boundary between healthy and at-risk patients isn't a straight line — there's meaningful non-linear structure in how these clinical features interact, and RBF's flexibility to model that paid off.
Takeaway: This project made overfitting concrete for me in a way "the tree fits noise" never did on its own — seeing that a much shallower tree achieved identical test performance with far better train accuracy really drove home why unconstrained trees are dangerous. It also builds real intuition for reaching for kernel methods when your problem might not be linearly separable.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article