한국어EN
60-day mapFive-Minute AI LessonsPart 1 · What AI Is

What's a token, the 'piece' AI talks in?

To AI, a sentence isn't letters or words — it's a line of pieces called tokens.

  • 5 min
  • Split it up
  • Quiz · 3

1 · Think first

The “next piece” from yesterday has a name in the AI world: a token.

A token is not a letter and not a word — it's a piece cut to be easy for AI to compute with, and its size changes from case to case.

So the same sentence gets a different count in tokens, and time and cost go up with that count.

2 · Experiment · Split it up

Try it — the sentence-splitting machine

Pick a sentence and cut it up three ways. Each way gives a different number of pieces.

Figure

In one picture

Same sentence, but the unit people count (words) and the unit AI counts (tokens) are different.

Sentence   My bestie lowkey devoured nachos tbh ↓ Split into words (the unit people read) My bestie lowkey devoured nachos tbh ↓ Split again into tokens (the unit AI counts) My best ie low key dev oured nach os tb h 6 words → 11 tokens People and AI count in different units
The orange boxes are pieces that got split off. The common word ('My') stays whole; slang and less common words ('lowkey', 'devoured') get cut finer. (Splits from the tokenizer GPT-4 uses.)

3 · Quiz

Quick check

Three questions. Pick an answer and it's marked right away, with the reason why.

Today in one line

Today in one line

To AI, a sentence isn't letters or words —
it's a line of pieces called tokens.

Learn more