The small piece of text an AI reads and writes, one at a time.
An AI doesn't read text letter by letter. It first cuts a sentence into pieces called tokens. Common words usually become one token; rare words get split into several.
How much an AI can read at once, and how much it costs to use, are both counted in tokens. Some languages, such as Korean, usually need more tokens than English to say the same thing. So when people say an AI 'picks the next word', it really picks the next token.
It's not only text. AIs that work with pictures or sound also cut them into small pieces and line them up, just like tokens.
Think of it like this
Think of LEGO. A common shape is one big brick; an unusual shape takes several small bricks snapped together. To an AI, a sentence is a row of these bricks.
For example
"unbelievable" might be split into "un" · "believ" · "able".
Records about this
Source-checked moments from AI history.
Popularized the idea that a mismatch between the tokenizer and the model's training corpus is a real failure mode