Overfitting
When a model memorises its training data — including noise and quirks — instead of learning general patterns, so it performs well on seen examples but poorly on new ones.
Overfitting is the classic failure mode of machine learning: the model fits the training set too closely and fails to generalise. It is the difference between a student who understands the subject and one who has memorised past exam answers — impressive on familiar questions, lost on anything new.
Signs include a widening gap between training accuracy (high) and validation accuracy (stagnant or falling), or suspiciously good results on data similar to training but poor results in the wild.
Standard defences:
- More and better data: diversity is the most reliable cure.
- Regularisation: weight decay, dropout, and label smoothing discourage memorisation.
- Early stopping: halt training when validation performance stops improving.
- Simpler models: fewer parameters or stronger constraints reduce capacity to memorise.
- Cross-validation: test on held-out folds to detect overfitting honestly.
Large foundation models complicate the picture. They are pre-trained on vast corpora, so “training data” overlaps with the world; apparent generalisation can be contaminated by test data leaking into training. Careful benchmark design and contamination checks are now essential when claiming a model performs well.