Tokens and Tokenization
Learn what tokens are, why they're not the same as words, and why some words cost more than others.
Open any AI provider's pricing page and you'll see something like "$3 per million tokens". Not per word, not per character, not per request. Your entire bill, and every context limit you'll ever hit, is measured in this unit. So what exactly is a token?
A token is a chunk of text, and it is not the same as a word. Modern LLMs use subword tokenization (a common algorithm is Byte Pair Encoding, or BPE): text is split into frequently occurring substrings, so common words get one token and rare words get assembled from pieces.
"hello" → ["hello"] (1 token)
"tokenization" → ["token", "ization"] (2 tokens)
"2024" → ["202", "4"] (2 tokens)Even numbers get chopped into chunks (which is one reason models are shaky at digit-by-digit arithmetic). The model never sees your letters. It sees a sequence of these chunks. Exact splits vary between model families, since each has its own tokenizer.
Which of these is most likely to be split into the MOST tokens?
- Asupercalifragilisticexpialidocious
- Bthe dog ran to the park
- Chello hello hello
- Dimport json
How many tokens is a given amount of text? For ordinary English, the standard rule of thumb:
> 1 token ≈ 0.75 words, or roughly 750 words ≈ 1,000 tokens
Some calibration points:
- "Hello, world!" is about 4 tokens
- A 100-word paragraph is about 130 tokens
- A 500-word essay is about 650 tokens
- A 10-page document is roughly 3,500 to 7,000 tokens, depending on how dense the pages are
Details that shift the count: punctuation and spaces are tokens too, capitalization changes how words split, code and non-English text often tokenize less efficiently, and every model family has its own tokenizer, so exact counts differ between models.
Using the rule of thumb, approximately how many tokens is a 750-word article?
- AAbout 1,000 tokens
- BAbout 750 tokens (1 token per word)
- CAbout 500 tokens
- DAbout 7,500 tokens
You don't have to guess token counts. Tokenizer libraries count them exactly before you ever call an API. Here's OpenAI's tiktoken:
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Hello, how are you today?")
print(len(tokens)) # 7Counting tokens up front lets you:
- Estimate API costs before running a large workload
- Check that a prompt fits within a context limit
- Decide when a long document needs to be split up (a big theme in the RAG module)
Fill in the method that converts text into its list of tokens using tiktoken. Write just the missing method name.
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.____("Hello, how are you today?")
print(len(tokens)) # 7A useful side effect of understanding tokens: it explains a famous class of LLM failures.
Ask a model to count the letters in a word, reverse a string exactly, or do arithmetic on long digit strings, and it often fails, even though it writes working code. Why? Those tasks live at the character level, but the model lives at the token level. To the model, "tokenization" is two opaque chunks, and how many letters are inside a chunk is something it can only guess from patterns.
When you actually need character-level precision, the reliable move is to have the model write and run code (a Python one-liner counts letters perfectly), not to trust its token-level guess.
- A token is a chunk of text: whole common words, pieces of rarer words, punctuation, spaces
- Subword tokenization (like BPE) splits text into frequent substrings; common = few tokens, rare = many
- Rule of thumb: 1 token ≈ 0.75 words, so 750 words ≈ 1,000 tokens
- Character-level tasks (letter counting, exact reversal) trip models up because they see tokens, not letters
- Libraries like tiktoken count tokens exactly, so you can budget cost and context before calling the API
Next: turning token counts into dollar amounts, and why chat conversations quietly get expensive.