Tokens are pieces, not words
After this you can say how your text becomes token IDs, count tokens three ways, and predict why a language or a format costs more.
Strawberry
Guess first Spell strawberry with spaces: s t r a w b e r r y. How many tokens does our model see?
A famous puzzle: count the r's in strawberry. A model said two r's.
Our script tokens.mjs cuts "How many r's are in strawberry?" into 8 tokens. The model gets one number for strawberry: token ID 72600.
That token holds the space before the word too. At the very start of text, strawberry is 3 tokens: str, aw, berry.
So a token is not a word and not a letter. It is a piece from a fixed list, the vocabulary. Qwen3's has 151,669 entries.
Spell it out with spaces and you get 10 tokens, one per letter. Each space joins the letter after it.
Common mistake: tokens hide letters, so models can never count them. But each ID always means the same text, so its letters can be learned.
Studies find the hard part is counting, worst for letters that repeat, like those r's. Tokens are one cause, not the only one.
Your turn: run tokens.mjs on three forms of your name.
Quick check After a space, strawberry is 1 token. At the very start of text, it is 3. What makes the difference?
Text to IDs and back
Guess first Where does the model's fixed list of tokens come from?
A tokenizer is the program that turns text into token IDs and back. Nothing is lost: the IDs turn back into the same text.
First, one standard form: é typed as one character or as e plus an accent becomes the same. Then the split rule cuts word-like pieces.
Each piece becomes bytes: 1 per English letter, 3 per Hindi character. All 256 byte values are in the list, so no text is unknown.
Then merge rules join neighbor pieces. The rules came from counting the most common pair in lots of text, again and again: byte pair encoding.
Climb the ladder. A rule's rank is its place in the list; each press applies the lowest-ranked rule that fits, in both rows.
The rows climb: str, err, aw, erry. The row without a space makes berry, then stops: no rule joins str and aw.
With the space, merging ends in one token. Its ID, 72600, is its rank plus 256: the 256 single bytes come first in the list.
Going back, IDs become bytes, and bytes become text. A token can end mid-character, so the server holds that character until it is whole.
Hindi shows it. नमस्ते is 6 characters, 18 bytes and 6 tokens. Token 30484 ends with part of त, and token 97 finishes it.
Quick check Put the steps from text to token IDs in order.
Quick check Our script prints token 30484 as ्<E0><A4>. What do the <E0><A4> mean?
Quick check You count the lines of a stream and get 41. The last line says eval_count 44 (example numbers). How many tokens did the model write?
Numbers, code, languages
Guess first The same text in Hindi needs how many times the tokens of English, on our model?
The rule of thumb: about 4 characters per token, for English. On Qwen3 our samples ran from 4.78 (prose) to 1.00 (numbers).
Digits follow rules too. Qwen3 cuts 2026 into 4 tokens. GPT-4's split rule cuts it into 202 and 6; GPT-2's leaves it whole.
Languages differ more. On the UN human rights declaration, our rough measure: Spanish needs about 1.4 times English's tokens, Chinese 0.9, Hindi 5.2.
Other tokenizers give other gaps. One study found up to 15 times, for the Shan language on GPT-4's tokenizer.
Why? Qwen3's list has 94,712 tokens with Latin letters, 26,079 with Chinese characters, only 65 with Hindi letters. Hindi stays in small pieces.
The split rule adds to it: it treats Hindi's marks, like vowel signs, as non-letters and cuts there. मनुष्यों becomes 4 pieces.
Real case, April 2024: Llama 3 in llama.cpp got 3333+777 wrong the same way, in 16-bit and 4-bit files alike. Cloud copies got it right.
Cause: llama.cpp cut every model's text with GPT-2's split rule, so Llama 3 got 3333 whole, not 333 and 3. The fix: per-model rules.
Quick check Your English prompt is 2,000 Qwen3 tokens (example numbers). On our measure, Spanish needs about 1.4 times as many. About how many tokens in Spanish?
Quick check A local copy of a model gets a sum wrong the same way every time. The cloud copy gets it right. What do you check first?
Quick check Per character, which of these costs the most tokens on Qwen3?
Tokens at work
Guess first A paid API charges $1 per million input tokens and $5 per million output tokens. 200 drafts a day, 1,000 tokens in and 150 out each. Guess the daily bill, in dollars.
Ollama counts tokens too. Raw mode sends your text as is, and our run matched our script. Chat adds hidden tokens; lesson 4 shows them.
Tokens are also the bill. Example: the Draft reply button on a paid API, 200 drafts a day, 1,000 input and 150 output tokens each.
At Claude Haiku 4.5 prices on 2026-10-09 ($1 per million input tokens, $5 per million output): $0.20 + $0.15 = $0.35 a day.
What if every ticket came in Hindi, split like Qwen3 splits it? Input alone: about $1.04 a day. Hindi replies grow the output too.
Recount for each model: tokenizers differ between model families. Anthropic says its newer one makes about 30% more tokens for the same text.
Tokens also set capacity. Anthropic limits input and output tokens per minute; go over and the API answers HTTP 429, with a time to wait.
At work, every reply carries its token counts. Claude's API calls them input_tokens and output_tokens; teams bill and rate-limit by them.
Quick check Say raw mode shows 6 prompt tokens for a question, and the chat API shows 22 for the same question. Where do the 16 extra tokens come from?
Quick check The same bot drafts 200 replies a day, now with 2,000 input tokens each (example numbers). At $1 per million input tokens, what is the input cost per day, in dollars?
Quick check A provider switches to a tokenizer that makes 30% more tokens for the same text, at the same price per token. What changes for you?