The observation that AI gets predictably better as models, data and compute grow.
Scaling laws describe a pattern: as you increase a model's parameters, its training data and the computing power used to train it, its prediction error falls in a fairly regular way. OpenAI researchers set this out in 2020, which meant people could estimate in advance how much better a bigger model would be.
In 2022 the Chinchilla study showed that the data should grow along with the model, not just the model itself. Despite the name, these aren't laws of nature; they're trends observed so far. Researchers disagree about how long the trend will hold.
Think of it like this
Think of training for a race. If every time you double your practice your time improves by a similar proportion, you can roughly predict your next result. But nobody can promise the pattern will last forever.
For example
Chinchilla, with 70 billion parameters and more training data, beat a model with 280 billion parameters.
Records about this
Source-checked moments from AI history.
Exposed that the large models of the time were undertrained; the 70B model outperformed the 280B Gopher
"Hitting a wall" became the stock phrase of scaling skepticism
Showed that loss can be predicted as a power law of model size, data and compute, becoming the rationale for large-scale investment