Bonus · Learning from batches · How does a model learn?
Compare batch sizes at the same example budget in a synthetic delivery model. Inspect reproducible results and separate updates from runtime.
Compare batches at the same example budget
A smaller batch makes more parameter updates in one pass through the data. The extra updates do not guarantee a lower final loss.
Forty synthetic deliveries. Start: w = 0.3, b = 3.5. Learning rate: 0.03. Shuffle seed: 7. Decay off. Reset before each one-epoch run.
This fixed worked example is separate from the interactive lab settings. Interactive controls require JavaScript.
| Batch size | Examples used | Updates | Final MSE (min²) |
|---|---|---|---|
| 1 | 40 | 40 | 1.596419 |
| 4 | 40 | 10 | 1.449970 |
| 10 | 40 | 4 | 1.526880 |
| 40 | 40 | 1 | 2.628031 |
With these settings, batch 4 ends the epoch at lower loss than batch 1.
Every row uses forty examples for gradient updates. The number of updates differs.
Limits
- Examples used measures gradient work, not elapsed time or total computation.
- The lab also evaluates full-dataset loss after every update. That work is not included in this counter.
- This is one dataset and shuffle. Change the seed or learning rate before drawing a broader conclusion.
Reproduce the calculation
- Set learning rate to 0.03, turn decay off, and use shuffle seed 7.
- Reset and run one epoch for each batch size: 1, 4, 10, and 40.
- Compare final Loss on all 40 at forty examples seen, not the best loss reached along the way.