Where this fits in real jobs
After this you can say which jobs use TTFT and tokens/s, and check one tool against a speed budget.
Your two numbers at work
Guess first A developer's Ollama said their model loaded in 24 ms. Can their user still wait more than 3 seconds for the first word?
You now have two numbers: TTFT and tokens/s. Ours: 56 ms and 278.0 tokens/s. Real inference teams watch the same two, on much bigger machines.
The tool changes, the skill stays. Of 44 inference job posts we read on 2026-10-01, from 20 companies, 0 named Ollama, llama.cpp or GGUF.
Platform engineers and site reliability engineers (SREs) keep the service fast and up. OpenAI's runtime team checks every engine build for TTFT and TBT.
TBT means time between tokens, like ITL. One engineer who runs models on 48 GPUs calls the job "80% SRE and 20% software engineering".
A forward-deployed engineer (FDE) works hands-on with key customers. Together AI's post says they tune engines "to hit throughput and latency targets".
Engine and performance engineers find the slow step and fix it. Anthropic: "Measure, model, then change." Here "model" means: estimate how much a change helps.
GPU kernel engineers write small programs that run on the GPU, the deepest layer. Perplexity's post asks for deep GPU skill and 3+ years.
Entry doors exist, but they are few. CoreWeave's first-level inference post lists "Add basic metrics and dashboards". That is measuring work, like yours.
All four roles start from your two numbers. This course does not promise a job. It shows where the skill is used.
Quick check A team checks TTFT and time between tokens for every engine build before it ships. Which role on the map is this closest to?
Quick check Which part of your chapter 1 work do inference job posts ask for?
The case: loaded, but still slow
A real case from Ollama's GitHub: reported 2026-10-02, fixed four days later. A developer ran a model on a Mac mini with an M4 Pro.
They kept the model loaded and sent the same small /api/chat request, with think: false: once after about 4 idle hours, then warm.
After idle: total 3,273 ms; load 24, prompt eval 3,044 and eval 39 ms add to 3,107. Warm: total 111 to 116 ms.
Load was 24 ms, so the model looked loaded. Yet the user waited over 3 s. The wait hid in step 3, reading the prompt.
The cause: about 2 s after each request, macOS stopped holding the model's memory in place. The next request waited for it to return.
The issue blames MLX, Apple's library. Our GGUF file runs on llama.cpp, which the issue says holds weights until 3 minutes after the last work.
Which engine runs yours? Ollama's server log says. macOS: cat ~/.ollama/logs/server.log. Linux: journalctl -u ollama --no-pager --follow --pager-end. Windows: server.log in %LOCALAPPDATA%\Ollama.
Search the log for "using llama-server". Ours has the line "using llama-server for model": llama.cpp's server runs our GGUF file.
Common mistake: "Loaded means the user is not waiting." One counter can look fine while the user waits. Measure from the user's side.
Their model is 35B, ours 0.6B. Their workaround changes a whole-machine setting: "not a general recommendation". The real fix came in Ollama 0.40.0.
From Ollama 0.40.0, models that MLX supports run on MLX on Apple chips. Our qwen3:0.6b file is still GGUF, so it runs on llama.cpp.
Which version do you have? ollama --version prints it. Ours was 0.35.0; the newest on 2026-10-23 was 0.40.8.
Quick check First request after idle: total 3,273 ms, load 24 ms, prompt eval 3,044 ms, eval 39 ms. Where did most of the wait sit?
Quick check About how many times longer did the first request take than a warm one? Use total 3,273 ms (after idle) and 116 ms (warm).
Is it fast enough for a Draft reply button?
Guess first Example numbers, not a measurement: TTFT 0.5 s. How many tokens per second do you need, at least, to finish a 150-token draft within 10 seconds?
Back to the Draft reply button from lesson 2. Rule 1: first words within 1 s. Rule 2: a 150-token draft within 10 s.
These limits come from Jakob Nielsen: about 1 s keeps the user's flow of thought; about 10 s keeps their attention on the task.
A speed budget is a time limit set before you build. Zed's: half its suggestions within 200 ms, 9 in 10 within 500 ms.
A 32B model "wasn't really compatible with our strict latency requirements", Zed wrote. Their model, Zeta, is built on a 7B one.
Flip the lesson 2 formula: tokens/s needed = 150 / (10 - TTFT). Example: TTFT 0.5 s needs at least 15.8 tokens/s.
Paste the part after the colon of lesson 2's for lesson 4: line after python3 budget_check.py. Pasting the label too gives "unrecognized arguments".
Our run: TTFT 0.06 s, 278.0 tokens/s, draft 0.60 s: both pass. Rule 1 FAIL? Maybe a cold call: run speed.mjs again.
Twist 1: 5 people click at once; Ollama runs 1 at a time by default. Example: 4 s each, the fifth ends near 20 s.
Twist 2: after 5 idle minutes, Ollama unloads the model by default. The next click is a cold request: lesson 2's took 839 ms.
Twist 3: someone turns thinking on. In lesson 3, our first answer word moved from 0.03 s to 3.61 s. Rule 1 breaks.
You test all three twists in Break it. There you also answer the support lead: yes or no for each rule.
Quick check Example numbers: TTFT 0.8 s and 30 tokens/s. How long does a 150-token draft take, in seconds?
Quick check Example numbers: TTFT 2 s and 60 tokens/s. A 150-token draft. Which rules pass? (Rule 1: first words within 1 s. Rule 2: full draft within 10 s.)