The words in a sentence started looking at each other — 2017: the Transformer
Once words could look at each other, AI started understanding whole sentences.
1 · Think first
In 2017, one paper from a Google research team set the basic design of the AI you use today. The design is called the Transformer, and the paper's title is 'Attention Is All You Need'.
Before that, AI read a sentence one word at a time, from the start. The Transformer let every word in a sentence look at all the others at the same time — that's called 'attention.'
Looking at everything at once means it can all be calculated in one go. That made it a perfect fit for graphics cards, and GPT, Claude and Gemini all grew up on top of this design.
2 · Experiment · Split it up
Try it — What does 'it' point to? Watch the words look at each other
Pick a word, and the colors show which words in the sentence it looks at, and how hard. Bright means a hard look; faded words barely get a glance.
Figure
Reading in order vs. looking all at once
When the way of reading changed, the way of calculating changed too.
3 · Quiz
Quick check
Three questions. Pick an answer and it's marked right away, with the reason why.
Today in one line
Today in one line
Once words could look at each other,
AI started understanding whole sentences.
Learn more
- The original 2017 paper. The title says it all. — Attention Is All You Need (2017)
- Google's official post, written by the paper's authors — it explains attention with pictures — Transformer: A Novel Neural Network Architecture (Google Research, 2017)