Bonus · Learning from batches · How does a model learn?

Compare batch sizes at the same example budget in a synthetic delivery model. Inspect reproducible results and separate updates from runtime.

Compare batches at the same example budget

A smaller batch makes more parameter updates in one pass through the data. The extra updates do not guarantee a lower final loss.

Forty synthetic deliveries. Start: w = 0.3, b = 3.5. Learning rate: 0.03. Shuffle seed: 7. Decay off. Reset before each one-epoch run.

This fixed worked example is separate from the interactive lab settings. Interactive controls require JavaScript.

Calculated with this lesson’s model
Batch sizeExamples usedUpdatesFinal MSE (min²)
140401.596419
440101.449970
104041.526880
404012.628031

With these settings, batch 4 ends the epoch at lower loss than batch 1.

Every row uses forty examples for gradient updates. The number of updates differs.

Limits

  • Examples used measures gradient work, not elapsed time or total computation.
  • The lab also evaluates full-dataset loss after every update. That work is not included in this counter.
  • This is one dataset and shuffle. Change the seed or learning rate before drawing a broader conclusion.

Reproduce the calculation

  1. Set learning rate to 0.03, turn decay off, and use shuffle seed 7.
  2. Reset and run one epoch for each batch size: 1, 4, 10, and 40.
  3. Compare final Loss on all 40 at forty examples seen, not the best loss reached along the way.
Download the example data and settings (JSON)

Sources for the method