Overfitting occurs when a machine learning model learns the training data too closely, including noise, so it performs poorly on unseen data. Underfitting occurs when the model is too simple to learn the important patterns, so it performs poorly on both training and unseen data.
The Reasoning
A model is trained to find patterns that allow it to make predictions. Its ability to apply those patterns accurately to new data is called generalization.
An overfitted model effectively memorizes details specific to its training data. It therefore achieves very low training error but higher error when evaluated using separate validation data or test data. This is associated with excessive model complexity and high variance.
An underfitted model has insufficient complexity or has not been trained effectively. It fails to represent the underlying relationship in the data, producing high error even on its training data. This is associated with high bias.
| Feature | Overfitting | Underfitting |
|---|---|---|
| Model complexity | Too complex | Too simple |
| Training performance | Very strong | Poor |
| Validation or test performance | Significantly worse than training performance | Poor, often similar to training performance |
| Main problem | Learns noise and training-specific details | Fails to learn important patterns |
| Possible solutions | Use more training data, regularization, cross-validation, pruning, or early stopping | Increase model complexity, add useful features, or train for longer |
For example, a decision tree that creates branches for almost every training example may overfit. A tree with only one split may underfit because it cannot represent more complex relationships.
A common misconception is that excellent training accuracy proves that a model is effective. The model must also perform well on unseen data; otherwise, its predictions do not generalize.