Break it
Change one thing, guess first, look, and explain. Part of chapter 3: Python for JavaScript developers (optional bridge).
One process or four: a control
Set 4 slots
Parts 1 and 2 need 4 slots: set them, then restart Ollama. You undo it at the end of part 2.
On Linux, or with ollama serve
Not on the Mac app? Set the 4 slots one of these ways.
Windows app
Quit Ollama, add the user variable OLLAMA_NUM_PARALLEL with the value 4 in Settings, then start Ollama again.
Check it took
Find the server config line in Ollama's log. It should show OLLAMA_NUM_PARALLEL at 4.
What changes
Client processes: 1, then 4. Fixed: 32 requests at the same moment, 4 slots, the Draft reply call.
A control
A control is a run where you expect no change. It tells you when a bigger effect is real.
Run it
Run 32 requests from one process. With fewer than 100 requests, read the median and the slowest, not a p99.
Then from four
Run the same 32 requests from four processes.
The staircase you made
What changes
How each call is made: chapter 1's urllib inside async def, then the same call through asyncio.to_thread. Fixed: 32 requests, 4 slots.
Run it
Run the naive port. Columns: request, sent, first word, in seconds.
Then the fix
Same 32 requests and columns, each through to_thread. Show all lines.
Then debug mode
Debug mode on 5 requests: one long line for each call that held the loop.
Undo it
Put the setting back, restart Ollama, and check the server config line again.
On Linux, or with ollama serve
Not on the Mac app? Undo it one of these ways.
Windows app
In the same Settings dialog, delete the OLLAMA_NUM_PARALLEL variable, then quit Ollama and start it again.
Where the lie starts (a simulation)
A simulation
Simulation, made-up timings: drip_server.py streams Ollama-shaped lines at a set rate with no model. It shows where a client lies, not model speed.
What changes
Client work per chunk: 0, then 40 µs (millionths of a second). Then processes: 1, then 4. Fixed: 512 streams at 50 tokens a second.
Chunk lag
Here a chunk is one stream line. Chunk lag is when your client read it, minus when the server stamped it.
Run it
Two terminals. In the second, run ulimit -n 2048 to lift the open-file limit, then start the simulator.
No work
In the first terminal, lift the limit too, then run the client with no extra work.
40 µs of work
Now 40 µs of work on each chunk, in one process.
Four processes
The same work, spread over four processes.
On a small laptop
With 4 cores or fewer: --n 256 in all three runs, and --work-us 80 in place of 40. Simulator behind? Close other programs.