Tokenization is the process of breaking text into tokens — the small chunks that an AI language model actually reads and generates. It’s the first step in how a model handles any piece of text.
What it means in plain English
Computers can’t work with raw words directly, so text must be converted into a form a model can process. Tokenization is that conversion: a tokenizer splits your text into tokens (whole words, parts of words, or punctuation) and maps each to a number. The model works with those numbers, and when it generates a response, the tokens are turned back into readable text.
Different models use different tokenizers, which is why the same sentence can be a slightly different number of tokens depending on the system. Common words tend to be single tokens; rare words, names, and other languages often split into several.
A simple example
The word “unbelievable” might be tokenized as “un” + “believ” + “able.” Splitting words into reusable pieces lets a model handle a huge vocabulary — including words it has never seen — by combining familiar sub-parts.
Why it matters
Tokenization is the invisible first step behind every AI text interaction. It explains quirks people notice — why models occasionally misspell unusual words, why some languages cost more tokens than English, and why counting “words” isn’t quite how AI measures text.
Related terms
- Token — the unit tokenization produces.
- Embedding — how tokens are turned into meaningful numbers.
- Natural Language Processing — the field tokenization supports.
Frequently asked questions
What is tokenization?
Tokenization is the process of breaking text into tokens (words or sub-word pieces) that a model can process numerically.
Why is tokenization necessary?
Models work with numbers, not raw text, so text must first be split into tokens and converted to IDs before it can be fed into the model.
