Practice: four labs on your laptop
Count tokens three ways, run the loop by hand, time both phases, and fill a fit table. Part of chapter 2: What just happened: tokens, the loop, the two phases, and "will it fit?"
Count tokens three ways
Before you start
You need Ollama with qwen3:0.6b, Node 18 or newer, and Python 3. The labs follow Ollama 0.35.0. On Windows, use WSL or Git Bash.
Run it
Run the line below. It counts one question three ways: our script, Ollama in raw mode, and Ollama's chat route.
Three routes
Our script splits the text and nothing else. Raw mode sends it as it is. Chat wraps it in the chat template first.
Script and raw agree
Script and raw agreed in our run: same text, same token list, and raw mode adds nothing to your text.
Tokens nobody typed
Chat added tokens nobody typed: role marks, the switch /no_think, and an empty think block: 16 tokens, by our count and in our run.
See them yourself
Add --show to print the exact string Ollama builds, without running the model. Find the role marks and the empty think block.
Add a system prompt
Change one thing: add the system prompt with --system. Then compare the chat count with your first one.
What our run showed
Our run gave 33, as our count predicted: 22, plus 6 text tokens and 5 marks: <|im_start|>, system, a blank line, <|im_end|>, a line break.
Cost, as a what-if
At $1 per million input tokens, a million calls cost $1 per prompt token. A 22-token prompt: $22 for a million calls.
Go further: your language
Count one sentence of your own with node tokens.mjs, in English and in another language you know. How many times the English count is it?
Be the loop
Run it
Run the script. Twelve calls in raw mode, one token each, at temperature 0.
Read the rows
Each row: the token picked, then the five best candidates. A logprob is the log of a probability, so percent = exp(logprob) x 100.
Why so few
The server kept the earlier text in its prompt cache. Each call read only the token you added: 1 new token in our run.
Your script is the loop
The model never saw a loop. Each call was one pass: text in, one score per possible token out. Your script did the looping.
Turn up the temperature
Change one setting: --temperature 1.5. The script always fixes the seed, so runs repeat.
Compare the percentages
Our run: call 1's percentages matched within 0.1 at both temperatures, so Ollama shows scores before temperature. At 1.5 the draw still picked a space.
One normal call
Now one ordinary call that writes 12 tokens. Is its text the same as your loop's?
Why they match
At temperature 0, both take the top score each step: our texts matched. If yours differ, two almost equal scores swapped, as lesson 2 showed.
See both phases
Run it
Run the script: three prompt sizes times three answer caps, three runs each, thinking off. It writes phases-table.md.
A new tag every time
Each prompt opens with a new random tag, so the cache barely helps: Cached stays small, the template's start and the word Ticket.
Two speeds
Prefill speed is new prompt tokens divided by prompt_eval_duration. Decode speed is eval_count divided by eval_duration. Both times are in nanoseconds.
Your clock
TTFT and total come from your own clock: send to first answer piece, send to end. So TTFT also holds waits outside prefill.
The whole wait
So two phases explain the whole wait. Reading the prompt sets TTFT. Writing the answer, one token at a time, sets the rest.
Send one ticket twice
Change one thing: send the same ticket twice, with no new tag. --verbose prints a summary after each run.
Only prefill got shorter
In our run, prompt eval duration fell to under 1% of the first; the eval rate moved much less. The cache skips prefill only.
Read the second summary
The second prompt eval rate looks tiny: it divides only the new tokens by the time. The replies differ: ollama run draws at temperature 0.6.
The fit table
The method
Lesson 5's method: weights = parameters x bytes per parameter. Cache = tokens x bytes per token. Compare the sum with the budget, 14.23 GiB.
A worked row
Qwen3-0.6B in 16-bit: 751,632,384 x 2 bytes = 1,503,264,768 bytes. Divide by 1,073,741,824 to get 1.40 GiB.
Its file and its cache
Its 4-bit download: 522,653,767 bytes, or 0.49 GiB. Its cache for one 4,096-token request: 4,096 x 114,688 bytes, or 0.44 GiB.
Add and compare
In 16-bit: 1.40 + 0.44 = 1.84 GiB. In 4-bit, the exact bytes give 0.92 GiB. Both are far under 14.23.
Check with fit.py
Check every cell with fit.py. It works from the exact bytes, so a gap of 0.01 from rounding is fine.
One cell fails
Qwen3-8B in 16-bit. Its weights alone, 15.26 GiB, are more than the 14.23 GiB vLLM allows itself.
The format decided
The same model as Ollama's 4-bit file needs 5.43 GiB. The file format, not the model, decided the fit.
Leave working room
Leave room for the engine's working memory: 0.5 to 1.3 GiB in two vLLM logs. Above 12.9 GiB, 14.23 minus 1.3, is tight.
Your own machine
Now swap the card for your own machine. The first line loads the model with one short call. The script prints the memory numbers.
Check the KV line
Check the KV line: CONTEXT x 114,688 bytes, so exactly 448 MiB at 4,096 tokens. The GPU's model buffer is about the file size.
What SIZE adds
SIZE adds up the three GPU lines: model, KV and compute. Ours: 990.95 MiB, SIZE's 1,039,086,387 bytes. Prompts saved in RAM are not in it.
Where CONTEXT comes from
The last line sets CONTEXT by GPU memory: 4,096 under 23 GiB, 32,768 from 23 GiB, 262,144 from 47 GiB, cut to the file's 40,960.