The tool that chops text into tokens and gives each piece an ID number.
A tokenizer is the first step before an AI reads anything. It cuts your text into tokens using a fixed list of pieces, called its vocabulary, and turns each one into an ID number. The model only ever sees those numbers.
The vocabulary is built from lots of text by keeping chunks that often appear together. Different tokenizers can split the same sentence differently, which changes the cost and how much fits in the context window.
In 2023, people found 'glitch tokens': pieces in the vocabulary that a model had barely seen during training, which made it act strangely. Some researchers are now testing models that skip the tokenizer.
Think of it like this
Imagine texting with only a sticker pack. Common words have their own sticker, but anything missing has to be spelled out with several small letter stickers. A tokenizer is that sticker pack plus the rules for which stickers to use.
For example
"Tokenization" might become "Token" · "ization": two pieces instead of twelve letters.
Records about this
Source-checked moments from AI history.
Processes bytes directly, without a fixed vocabulary, using entropy-based dynamic patches; the first large-scale scaling of the tokenizer-free approach
Popularized the idea that a mismatch between the tokenizer and the model's training corpus is a real failure mode