Glossary · 1 · Foundations: how AI works
Token (AI)
Also known as: Tokenization
A token is the unit of text a language model reads and writes — a word, part of a word, a punctuation mark or a space. Models measure input length, output length, context limits and usage-based prices in tokens.
- Beginner
- Technical writers
- Technical marketers
- Technical project managers
- Developers
In one sentence
Tokens explained: the word pieces a language model reads and writes, and why they set context limits and AI costs.
Example
The word “interoperability” may be split into several tokens, while “the” is one. In English, 1,000 tokens are roughly 750 words; other languages often need more tokens per word.
Why it matters on your learning path
- Technical writers: Long, repetitive documents consume tokens. Modular, deduplicated content is cheaper and easier for AI to process.
- Technical marketers: Token-based pricing explains why “unlimited AI” offers often come with fair-use limits.
- Technical project managers: Cost estimates for AI features start with token volume: requests per day × tokens per request × price per token.
- Developers: Count tokens with the vendor’s tokenizer before sending requests; limits and prices differ by model.