The calculation that decides how much each token focuses on every other token.
Attention gives each word a score for how strongly it should look at every other word. The scores are shared out from a fixed total, so focusing hard on one word means less focus elsewhere. Where that focus lands helps the model work out things like what "it" refers to.
Attention became widely known through 2014 translation research, where it decided which part of the original sentence to look at. The 2017 Transformer built a whole model out of it, a version called self-attention.
Since every token looks at every other one, the work grows fast as text gets longer. That's one reason long context windows are hard.
Think of it like this
In a noisy cafeteria, you hear your name and your ears snap toward it while everything else fades a little. Attention is each word deciding whose voice to tune into right now.
For example
In "Jen packed her bag, and it was heavy," the word "it" pays more attention to "bag" than to "Jen".
Records about this
Source-checked moments from AI history.
Self-attention alone, without recurrence, made parallel training possible, establishing the architectural premise for large-scale pretraining
The paper that solved the fixed-length vector bottleneck with attention. The direct predecessor of the Transformer's self-attention