What Is a Token in AI? Tokens Explained

How Tokens Work

Before an AI model can read or write anything, it breaks text into small pieces called tokens. Think of it like how a spell-checker processes one word at a time — except tokenizers often split words into smaller chunks. With OpenAI's current tokenizer, "understanding" becomes two tokens: "under" and "standing" (see thereal examples below).

This process is called tokenization. Every piece of text you send to an AI — your prompt, instructions, and any context — gets split into tokens before the model processes it. The model's response gets generated one token at a time, too.

Different AI models use different tokenizers, so the exact same text can produce slightly different token counts depending on the model. GPT, Claude, and Gemini each have their own tokenizer. However, the ratio of roughly 0.75 words per token (or 1.33 tokens per word) holds well across all of them for typical English text.

Numbers, code, and non-English text tend to use more tokens per word. A line of code averages about 10 tokens, and the same sentence can take noticeably more tokens in Hindi or Japanese than in English.

Real Examples: How Text Splits Into Tokens

These splits come from running OpenAI's o200k_base tokenizer (used by GPT-4o through GPT-5.x) on each example. Common words stay whole; rare words, numbers and code break into pieces.

A · marks a space — tokenizers usually attach the space to the start of the next word.

The same question in five languages

Tokenizers split some languages into more pieces than others, so the same meaning can cost more to process in one language than another. One question, counted with the same tokenizer:

One sentence is only an illustration — ratios vary with the text. Count your own with thetoken counter.

What Does 1 Million Tokens Look Like?

A million tokens is the standard billing unit for AI APIs. Here's what that actually looks like in terms you can picture:

1 spine = 1 novel

📚≈ 8 novels

average novels

A typical novel runs about 90,000 words (~120,000 tokens), so a million tokens holds roughly eight of them.

basis: 90,000 words per novel

1 block = 100 lines of code

💻≈ 100,000 lines of code

lines of source code

Code averages roughly 10 tokens per line, so a million tokens is about 100,000 lines — nearly twice the original Doom engine (~57,000 lines).

basis: ~10 tokens per line

1 block = 25 messages

💬≈ 75,000 chat messages

chat messages

A typical text message is about 10 words (~13 tokens). A million tokens is over six years of texting at 30 messages a day.

basis: ~13 tokens per message

1 block = 1 page

📃≈ 1,500 pages

single-spaced pages

At about 500 words per single-spaced page, 750,000 words fills roughly 1,500 pages — a stack of paper about 15 cm (6 in) tall.

basis: 500 words per page

the 7 books — filled part = 1M tokens

🧙≈ 69% of Harry Potter

of the complete 7-book series

The full Harry Potter series is 1,084,170 words — about 1.45 million tokens. A million tokens gets you past Order of the Phoenix and into the early chapters of Half-Blood Prince.

basis: Series total 1,084,170 words

Every estimate shows its assumption. See all 16 comparisons on the main page →

How Much Do Tokens Cost?

AI companies charge separately for input tokens(what you send) and output tokens (what the model writes back). Output always costs more — typically 3–6× more — because generating text requires far more computation than reading it.

Here's what a few popular models charge per million tokens:

Prices range from $0.10 to $50 per million tokens — a huge spread. See all model prices →

Token Calculator

Want to estimate how much your specific use case will cost? Use our calculator to pick a model, set a budget, and see exactly how many words, pages, or lines of code that buys.

Open the token calculator →

To see how many tokens a specific piece of text uses, paste it into the token counter. It counts GPT tokens exactly and estimates Claude and Gemini, with the cost on every model.