The first, giant training stage where a model learns broadly from huge amounts of data.
Pretraining is the first stage of building an AI model. The model is shown a huge amount of text, code and other data and asked to guess the next token again and again. Nobody has to label anything: the text itself supplies both the question and the answer.
Along the way the model picks up grammar, common sense and knowledge from many fields. It's expensive, though: thousands of graphics chips running for weeks or months. The result, a base model, is good at continuing text but clumsy at answering questions helpfully, so it's then polished with fine-tuning. The "pre" means it comes before that polishing.
Think of it like this
It's like an athlete spending years on fitness and basic skills before specializing in one sport. With strong basics, any sport comes faster. Pretraining is that basic training.
For example
Give a base model "Once upon a time" and it happily keeps writing a story, but it may respond oddly to "help me with my homework".
Records about this
Source-checked moments from AI history.
Scaling skepticism came not from a critic but from one of the founders of the scaling paradigm. Peak data became an industry-standard term
Showed once again that deep networks can be trained, sparking the revival of deep learning in the 2000s