Guessing the one token that comes next, based on all the text so far.
Large language models write in a surprisingly simple way. They look at all the text so far, give every possible next token a probability, and pick one. Then they add it to the text and do it again, hundreds of times, until the answer is done.
Training is the same game. The model reads huge amounts of text with the next token hidden, guesses it, and gets nudged a little whenever it's wrong. Repeated an enormous number of times, this simple drill teaches grammar, facts and style.
But the most likely next token isn't always the true one. That's one reason AIs sometimes make things up.
Think of it like this
Think of the word suggestions above your phone keyboard. If you kept tapping a suggestion, you'd build a whole sentence one word at a time. An LLM does the same thing, just with far better guesses.
For example
After "I ate a slice of …", "pizza" gets a high probability and "homework" a very low one.
Records about this
Source-checked moments from AI history.
Introduced in-context learning, which performs a task from examples in the prompt alone, without updating parameters
Proposed a neural language model that learns words as dense vectors. The direct predecessor of the embedding lineage that includes word2vec