2 · Overfitting · How does a model learn?
Compare training and validation error on synthetic deliveries. Change model flexibility and regularization, then choose a model before opening the test result.
Compare training and validation error
A model can fit familiar examples while making poor predictions on held-out data. Compare both scores before selecting its flexibility.
The built-in synthetic split: 12 training deliveries. Regularization λ = 0. These are three selected degrees, not an exhaustive search.
This fixed worked example is separate from the interactive lab settings. Interactive controls require JavaScript.
| Degree | Training RMSE (min) | Validation RMSE (min) |
|---|---|---|
| 1 | 0.986 | 1.011 |
| 3 | 0.678 | 0.667 |
| 10 | 0.378 | 28.458 |
At degree 10, training RMSE is 0.378 minutes and validation RMSE is 28.458 minutes.
The smaller training score does not establish better predictions on new deliveries.
Validation error can be slightly lower than training error on a finite split. The gap need not always be positive.
Limits
- These results describe this synthetic dataset. They do not identify the best degree for other tasks.
- Use validation to select a model. Keep the final test separate until that selection is fixed.
Reproduce the calculation
- Use 12 training deliveries and set regularization to None.
- Compare degrees 1, 3, and 10 after revealing validation.
- Record both RMSE values. Do not select the model using its final test result.