A Token is the basic unit of text that an AI language model processes. Rather than reading whole words or individual letters, models break text into tokens — chunks that are, on average, about three-quarters of a word.
What it means in plain English
Before a model can work with text, it splits it into tokens. A common word like “the” is one token; a longer or unusual word might be broken into several (“tokenization” could become “token” + “ization”). The model then works entirely in terms of these tokens, predicting the next token over and over to generate text.
Tokens matter for two practical reasons. First, AI services usually charge by the token, so cost scales with how much text you send and receive. Second, a model’s context window — the maximum text it can consider at once — is measured in tokens, which sets a hard limit on how much it can “remember” in one go.
A simple example
The sentence “AI is useful” is roughly four tokens. As a rough rule of thumb, 1,000 tokens is about 750 words — handy for estimating both cost and how much text will fit in a model’s context window.
Why it matters
Understanding tokens demystifies two things people find confusing about AI: why they’re billed the way they are, and why there’s a limit to how much text a model can handle at once. Both come down to tokens.
Related terms
- Tokenization — the process of splitting text into tokens.
- Context Window — the token limit a model can process at once.
- Large Language Model — the systems that operate on tokens.
Frequently asked questions
What is a token in AI?
A token is a chunk of text — often a word or part of a word — that a language model processes as a unit. Models read and generate text token by token.
Why do tokens matter?
Model limits (context windows) and usually pricing are measured in tokens, so understanding tokens helps you manage cost and how much text a model can handle.
