Known errors (chapter 2)

Counting tokens

What you see
tokens.mjs cannot download tokenizer.json
Cause
No internet, or Hugging Face is blocked on your network.
Fix
Download the file once from the link the script prints, and save it as qwen3-tokenizer.json next to tokens.mjs.
Status
Our script.
What you see
Error: Cannot find module '.../tokens.mjs' from count3.sh
Cause
count3.sh runs tokens.mjs from its own folder, and the file is not there.
Fix
Get tokens.mjs and count3.sh from Practice and put them in one folder.
Status
Seen on our Mac (Node v22.23.2).
What you see
The script count and the raw count differ
Cause
Different text reached each one, for example a shell changed a quote or added a newline, or the model is not qwen3:0.6b.
Fix
Put the text in a file, one text per line, and run sh count3.sh --file on it, so both counts read the same line. Check the model name.
Status
Our advice. Qwen3 adds no start token, so the two counts should match.
What you see
Chat is raw + 10, not raw + 16
Cause
think was left out. Ollama then turns thinking on: the template adds /think and no empty think block (16 tokens, not 22).
Fix
Send "think": false.
Status
In Ollama's code, and seen on our Mac: with think left out, our 6-token question counted 16.
What you see
Chat is much more than raw + 16
Cause
The model has a system prompt set in its Modelfile, or you sent one.
Fix
ollama show --system qwen3:0.6b.
Status
In Ollama's code.
What you see
--show prints nothing or an error
Cause
_debug_render_only is a debug option and may change between versions.
Fix
Check ollama -v says 0.35.0. Use ollama show --template qwen3:0.6b to read the template instead.
Status
In Ollama's code. Not seen on our Mac: on 0.35.0, --show printed the string.
What you see
The first streamed piece starts with a space
Cause
Many tokens carry their leading space: " strawberry" is one token, ID 72600.
Fix
Nothing is wrong. Trim the space when you show the text, if you need to.
Status
Seen in our runs of tokens.mjs.
What you see
A raw prompt full of control tokens counts many more tokens than predicted
Cause
The engine read <|im_start|> as plain text, not as one control token.
Fix
Use /api/chat, which builds the template for you. Note the raw count in your README: it shows how the engine reads typed control tokens.
Status
Not seen on our Mac: Ollama 0.35.0 read each typed role mark as one control token, 18 and 14 tokens, as predicted.
What you see
The script count is higher than raw when your text spells <|im_end|>
Cause
tokens.mjs counts the name as plain text; Ollama's engine reads it as one control token, in raw mode and in chat.
Fix
Add --special to tokens.mjs. Never pass user text through unchecked: it can fake a turn.
Status
Seen on our Mac: A<|im_end|>B counted 8 in our script, 3 raw, 19 through chat.

The loop, stops and seeds

What you see
No logprobs in the reply
Cause
You used the OpenAI-compatible route, which does not support them.
Fix
Use /api/generate or /api/chat with "logprobs": true.
Status
In Ollama's docs.
What you see
The loop's text sits in thinking, not response
Cause
think was left out. Ollama turned thinking on and parsed the raw output for think tags.
Fix
Send "think": false.
Status
In Ollama's code. Not seen on our Mac: with think left out, our loop's text held no think tags and stayed in response.
What you see
message.content is empty, done_reason is length
Cause
Thinking used the whole num_predict budget.
Fix
Raise num_predict, or send "think": false.
Status
In Ollama's code, and seen on our Mac: a cap of 64 left the content empty.
What you see
The answer stops mid-sentence, done_reason is length
Cause
The num_predict cap. The model never sees it.
Fix
Raise num_predict; ask for a short answer in the prompt instead.
Status
In Ollama's code.
What you see
The same words over and over at temperature 0 with thinking on
Cause
Greedy picking can repeat itself, and Qwen3's makers advise against it for thinking.
Fix
Use the model's defaults (temperature 0.6 for thinking), or cap num_predict.
Status
In Qwen3's model card. Not seen in our runs: two answers at temperature 0 with thinking on repeated no sentence.
What you see
Seed 42 still gives different words on call 1
Cause
Call 1 read the whole prompt; later calls reused the cache, and the scores moved slightly.
Fix
Expected. Compare calls 2 onward, or send a fresh prompt each time.
Status
Seen in a report (still open). Not seen on our Mac: in 3 runs, all 5 calls matched each time.
What you see
HTTP 400 "<model>" does not support thinking
Cause
Not this model: another model got think: true.
Fix
Use qwen3:0.6b.
Status
See chapter 1's known errors: Thinking.

See both phases (Practice)

What you see
Cached is large in phases-table.md
Cause
The prompt started the same way as an earlier one: an old copy of phases.py without the random tag, or another program sent the same ticket.
Fix
Use the phases.py from this page, then rerun the table.
Status
In Ollama's docs.
What you see
TTFT jumps on one row only
Cause
That call loaded the model (look at load_ms in phases-runs.json).
Fix
Rerun the table.
Status
See chapter 1's known errors: Practice table.
What you see
Formula and total differ by more than 10%
Cause
A pause outside the counters, for example another app, or a long first token.
Fix
Close other apps and rerun.
Status
The formula is NVIDIA's. Not seen on our Mac: at most 1.8% in 108 calls. A 1-token call read eval_duration 1000 ns, so it leaves out the first token.
What you see
Output tokens is below the cap
Cause
The model ended its reply first: done_reason is stop.
Fix
Expected. The formula uses the real eval_count.
Status
Seen on our Mac.
What you see
TTFT of the long prompt is about 0.1 s more than its prefill time
Cause
Before it reads a prompt, the engine saves the last one to its prompt cache in RAM. Your clock counts that wait; prompt_eval_duration does not.
Fix
Expected: TTFT is what the user waits for.
Status
Seen on our Mac: the server log timed that save at 71 to 282 ms.
What you see
prompt eval cached is missing from the second ollama run --verbose summary
Cause
The two runs did not send the same text: ticket.txt was made again, or the prompt words changed. The line shows only when the cached count is above 0.
Fix
Make ticket.txt once, then run the same ollama run line twice.
Status
In Ollama's code.
What you see
The second --verbose summary shows a tiny prompt eval rate
Cause
The rate divides only the new, uncached tokens by the time, and the cache left 1 new token.
Fix
Read prompt eval duration instead.
Status
In Ollama's code, and seen on our Mac.

When the prompt does not fit

What you see
HTTP 200, the reply ignores the system prompt, prompt_eval_count is num_ctx / 2 + 2 (258, 1,026, 2,050)
Cause
The prompt was longer than the context. Ollama kept the first 4 tokens and the end.
Fix
Raise num_ctx, or shorten the history yourself. Compare prompt_eval_count with your own count.
Status
In Ollama's code, seen in reports, and seen on our Mac: 258 at num_ctx 512.
What you see
Log: truncating input prompt (Warn)
Cause
Same cut, as the log shows it, with limit, prompt, keep and new.
Fix
Same fix.
Status
In Ollama's code, and seen on our Mac.
What you see
Old turns of a long chat are missing, no Warn line
Cause
Stage 1 dropped whole old messages. It logs only at Debug level.
Fix
Set OLLAMA_DEBUG=1 the way chapter 1 set OLLAMA_NUM_PARALLEL to see it, then undo it the same way (on macOS launchctl unsetenv OLLAMA_DEBUG, then restart the app). Keep the chat shorter than num_ctx.
Status
In Ollama's code, and seen in reports (still open).
What you see
HTTP 400 the prompt is longer than the context length currently available to the model; ...
Cause
You sent "shift": false, so Ollama refuses instead of cutting.
Fix
Raise num_ctx or shorten the prompt.
Status
In Ollama's code, and seen on our Mac.
What you see
Log: level=ERROR ... msg="llama-server terminated" error="signal: killed"
Cause
Ollama stopped its engine to start one with new settings, such as "shift": false.
Fix
Nothing failed: the next lines start the new engine.
Status
Seen on our Mac.
What you see
request (7851 tokens) exceeds the available context size (4096 tokens), try increasing it
Cause
llama-server's own check, seen with imported GGUF models that use their built-in template.
Fix
Raise num_ctx.
Status
Seen in a report, and in llama.cpp's code.
What you see
"truncate": false gives an error
Cause
No cut, so the engine gets a prompt longer than its context.
Fix
Raise num_ctx.
Status
Seen on our Mac: HTTP 400, request (871 tokens) exceeds the available context size (512 tokens), try increasing it.
What you see
No cut and no error, on a newer Ollama
Cause
In v0.35.0's code, models that run on MLX skip this cut (truncate is forced off for them), and v0.40 runs supported models on MLX by default on Apple Silicon.
Fix
The labs follow v0.35.0: check ollama -v, and install 0.35.0 with the commands on Practice's "Before you start" screen.
Status
In Ollama's code and release notes. Not verified: v0.40's code.
What you see
CONTEXT is not what the lab expects
Cause
The default depends on GPU memory and on the app's context slider.
Fix
Our scripts set num_ctx on every call.
Status
In Ollama's code and docs. Our 24 GB Mac: total_vram 17.8 GiB, so 4096.
What you see
vLLM: This model's maximum context length is ... tokens. However, you requested ...
Cause
vLLM counts prompt plus requested output and refuses.
Fix
Shorten the prompt or lower max_tokens.
Status
In vLLM's code.

Memory and fit (Practice and Break it)

What you see
ps_budget.sh finds no buffer lines
Cause
The lines appear only when a model loads, and the log may have rotated.
Fix
ollama stop qwen3:0.6b, one call, then run it again.
Status
Not seen on our Mac: v0.35.0's log holds the model, KV and compute lines at every load.
What you see
No recommendedMaxWorkingSetSize line in the log
Cause
Ollama 0.35.0's log does not print it. llama.cpp prints it when it starts on a Mac.
Fix
Read total_vram in the vram-based default context line instead.
Status
Seen on our Mac.
What you see
Two model buffer lines, and ps_budget.sh adds them
Cause
On a Mac the log gives the GPU part (MTL0) and a small CPU part.
Fix
Use the MTL0 line: it is on the GPU, and SIZE counts it.
Status
Seen on our Mac.
What you see
ps_budget.sh cannot open the log file
Cause
The script reads the macOS log. Linux keeps the log in the system journal, and Windows under %LOCALAPPDATA%\Ollama.
Fix
Linux: journalctl -u ollama --no-pager > ollama.log, then LOG=ollama.log sh ps_budget.sh. Windows: set LOG to that folder's server.log.
Status
In Ollama's docs.
What you see
ollama ps SIZE does not match your GiB math
Cause
ollama ps prints decimal GB with one decimal, the log prints MiB, and your table uses GiB.
Fix
Read the exact bytes from /api/ps and convert once.
Status
In Ollama's code. On our Mac, the three GPU lines summed to 990.95 MiB and /api/ps said 1,039,086,387 bytes: the same, within the log's rounding.
What you see
ollama list says 522 MB, the web page 523 MB
Cause
ollama list cuts the decimals: 522,653,767 bytes is 522.65 MB.
Fix
Nothing is wrong.
Status
In Ollama's code.
What you see
ollama ps PROCESSOR shows a split such as 48%/52% CPU/GPU
Cause
Weights plus cache did not fit the GPU's share, so some layers run on the CPU, slower.
Fix
Lower num_ctx, use a smaller file, or fewer slots.
Status
In Ollama's docs.
What you see
Log: llama-server load failed; reducing automatic context and retrying once
Cause
The load ran out of memory, so Ollama tried a smaller default context.
Fix
Expected. Set num_ctx yourself to choose the trade.
Status
In Ollama's code.
What you see
Log: model is too large for system memory
Cause
Weights plus cache do not fit at all.
Fix
A smaller file or a smaller num_ctx.
Status
In Ollama's code.
What you see
SIZE did not change after a bigger num_ctx
Cause
The call did not reload the model, or you read ollama ps before the second call.
Fix
Check CONTEXT in ollama ps.
Status
In Ollama's code. Not seen on our Mac: 16,384 reloaded the model, and SIZE grew.
What you see
SIZE grew more than the KV cache
Cause
Above 4,096 tokens the engine reads the prompt in bigger groups, so its working memory grows too.
Fix
Expected. Count working memory in your budget.
Status
Seen on our Mac: the GPU compute buffer grew from 50.20 to 146.30 MiB.
What you see
Activity Monitor shows the engine using far more memory than ollama ps
Cause
The engine keeps old prompts in RAM, up to 8,192 MiB, and SIZE leaves them out.
Fix
Expected. Plan for that RAM when you run many long prompts.
Status
Seen on our Mac: 3,537 MiB of RAM after the full phases.py table, with SIZE 1.0 GB.

vLLM on a GPU (lesson 5)

What you see
Failed to load model - not enough GPU memory. Try lowering --gpu-memory-utilization ...
Cause
The weights alone do not fit the GPU. No cache line comes before it.
Fix
A 4-bit or FP8 file, a smaller model, or more GPUs.
Status
In vLLM's code, and seen in a report.
What you see
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 96.00 MiB ...
Cause
The same failure, as PyTorch reports it.
Fix
A 4-bit or FP8 file, a smaller model, or more GPUs.
Status
Seen in a report.
What you see
To serve at least one request with the model's max seq len ...
Cause
The weights fit, but the cache for one request at full length does not.
Fix
--max-model-len lower, or auto. The message also gives an estimated maximum.
Status
In vLLM's code.
What you see
No available memory for the cache blocks.
Cause
Nothing is left for the cache after the weights.
Fix
Raise --gpu-memory-utilization, or a smaller model.
Status
In vLLM's code.
What you see
Free memory on device ... on startup is less than desired GPU memory utilization ...
Cause
Another process holds GPU memory, for example a second engine.
Fix
Stop the other process, or lower --gpu-memory-utilization.
Status
In vLLM's code.

Old model files and templates

What you see
missing pre-tokenizer type, using: 'default' then GENERATION QUALITY WILL BE DEGRADED!
Cause
An old GGUF without its model's split rule. Numbers and text split wrongly.
Fix
Download a fresh file of the model.
Status
In llama.cpp's code, and seen in a report.
What you see
... So now the final prompt starts with 2 BOS tokens.
Cause
Your text already starts with the start token, and the engine adds another. Happens in raw mode with models that use one (not Qwen3).
Fix
Leave the start token out of your raw text.
Status
In llama.cpp's code.

From chapter 1

What you see
Failed to connect, Connection refused, fetch failed, model not found (HTTP 404), a timeout, or a broken stream
Cause
Ollama is not running, the model is not pulled, or the stream was read the wrong way: chapter 1 covers each one.
Fix
Open chapter 1's known errors page: Start and connect, Model, Timing and counters.
Status
See chapter 1's known errors: Start and connect, Model, Timing and counters.

The check page

What you see
"My answer is close but marked wrong"
Cause
The band is ±5%, or the band the item states, or the unit is wrong (GB against GiB).
Fix
Read the unit and the band in the question.
Status
What you see
"I used my own run's numbers"
Cause
The check uses fixed numbers from sources, not your run.
Fix
Use the numbers in the question.
Status