Why did just making it bigger give it new skills? — 2020: GPT-3
Same design, just more size and more reading, and it learned to copy from a few examples.
1 · Think first
They didn't change yesterday's Transformer design at all — they just made it bigger, and in 2020 out came a model with 175 billion numbers. Its name is GPT-3.
The surprise wasn't the size, it was how it learned. With no retraining, you could tuck just two or three examples into your question and it would pick up the task right away.
That's why the paper was titled 'Language Models are Few-Shot Learners' — 'few-shot' means from just a few examples. From then on, 'what you tell it,' the prompt, started to really matter.
2 · Experiment · Number sense
Try it — A feel for digits — how big did GPT get?
Move the slider to guess 'about 10 to the what?' first. Then reveal the answer to see the real number. The goal isn't to get it right — it's to feel how fast the digits jump.
Figure
How much GPT grew in 3 years
Measure them with the same ruler as bars, and the difference jumps out.
3 · Quiz
Quick check
Three questions. Pick an answer and it's marked right away, with the reason why.
Today in one line
Today in one line
Same design, just more size and more reading,
and it learned to copy from a few examples.
Learn more
- The original GPT-3 paper. Its title is today's key idea — a few examples are enough — Language Models are Few-Shot Learners (2020)
- The official announcement from June that year that opened GPT-3 to people: send in text, get text back — OpenAI API (OpenAI, 2020)