Day 16: Train/val/test splits; overfitting & underfitting
The cardinal rule: never test on training data
A model that has *memorised* its training examples can score 100% on them and be useless on anything new. To measure real learning you must hold data back. The standard split is three ways: a training set the model learns from, a validation set you use to tune choices (which model, which hyperparameters), and a test set touched exactly once at the very end to estimate real-world performance. A common split is 70/15/15.
Data leakage is the silent killer
If any information from the validation/test set sneaks into training — even something as subtle as computing a normalization mean over the *whole* dataset before splitting — your measured accuracy is a lie. Split first, then compute everything else using only the training portion. This bites experienced engineers constantly.
Overfitting vs underfitting
Overfitting: the model learns the training data's noise and quirks, not the general pattern — high training accuracy, low validation accuracy. Underfitting: the model is too simple (or under-trained) to capture the pattern at all — low accuracy on *both*. The gap between training and validation performance is your single most important diagnostic, and you'll watch it obsessively when training the Garment Classifier on Day 39.
Key terms
- Training set
- Data the model directly learns from by adjusting its parameters.
- Validation set
- Held-out data used to compare models and tune hyperparameters during development.
- Test set
- Data touched only once, at the end, to estimate true generalization performance.
- Overfitting
- Learning training-set noise instead of the general pattern; high train accuracy, low validation accuracy.
- Data leakage
- Information from validation/test data influencing training, producing falsely optimistic results.
Your model scores 99% on training data but 68% on validation data. What is happening?
