3 · Attention · How does a model learn?
Inspect query-key scores, softmax weights, and weighted values in a constructed attention model. Follow the calculation and its limitations.
Follow one attention calculation
Queries and keys produce scores. Softmax normalizes those scores into weights, which mix the value vectors into an output.
Constructed example: “The bank approved the loan”. BANK query = (1.5, 1.5); key dimension = 2. The vectors are chosen for teaching.
This fixed worked example is separate from the interactive lab settings. Interactive controls require JavaScript.
| Token | Scaled score | Weight |
|---|---|---|
| The | 0.000000 | 6.9451% |
| bank | 0.212132 | 8.5864% |
| approved | 1.484924 | 30.6602% |
| the | 0.000000 | 6.9451% |
| loan | 1.909188 | 46.8631% |
The weighted value is approximately (0.713913, 0.000000).
The full-precision weights sum to 1. Displayed percentages can differ slightly because of rounding.
Limits
- These vectors were not extracted from a trained language model. Their financial and river dimensions are invented.
- An attention weight describes one routing operation. It does not measure that token’s share of the final decision.
Reproduce the calculation
- Multiply each key by the query using a dot product, then divide by √2.
- Apply softmax across the scores in this row.
- Multiply each value vector by its weight and add the resulting vectors.