Streams, async and the clock
After this you can write a load-test client in Python that reports what the user waited for, not what your code happened to notice.
A stream is a generator
Guess first In JavaScript and in Python, you make a generator but never read from it. Does the code inside it run?
A Python function with yield is a generator, like function*. Each read resumes where it stopped. Python's for reads it like for...of.
Both are lazy and single-use: nothing runs until you read, and a second pass is empty. At the end, next() raises StopIteration in Python.
Chapter 1's loop becomes stream_chat(url, body), a generator of (seconds, message) pairs. Code that reads pairs can now be tested with a saved log.
Ollama streams NDJSON: one JSON object per line. A byte chunk can stop mid-line, so both versions keep the half line for the next one.
Splitting raw bytes at the newline byte, b"\n", is safe: it never sits inside a UTF-8 character, and JSON escapes newlines inside text.
A chunk can end mid-character, and then decoding it alone fails. TextDecoder with stream: true waits for the rest; so does Python's incremental decoder.
Run it: café in two chunks. Chunk 1 alone fails to decode; the incremental decoder waits for byte two and gives café.
On break, JavaScript's for...of closes the generator and runs its cleanup. Python's for does not: a generator you still hold stays open.
Run both: Node runs the cleanup at break. Python runs it only when you call stream.close().
With a stream, the HTTP response stays open, so the server may keep writing an answer nobody reads. Our demo reads 2 of 5 lines.
Run it: after break, the response is still open. close(), or a with closing(...) block, closes it.
So wrap the loop: with closing(stream_chat(url, body)) as pieces:. When the block ends, by break or by an error, the stream closes.
Quick check pieces = stream_chat(url, body). You read 5 lines with for seconds, message in pieces:, then break. What is true right after the loop?
Quick check Which way of splitting Ollama's stream into lines is safe?
Calling is not running
Guess first say() is an async function whose first line prints. You call say() without await, in JavaScript and in Python. Which one prints?
Common mistake: calling an async def function starts it, as in JavaScript. In Python it only makes a coroutine, which waits for await.
Python's only sign is a warning: RuntimeWarning: coroutine 'say' was never awaited. There is no error, and the code inside never ran.
So the JavaScript habit of calling two async functions, then awaiting both, runs them one after the other in Python.
Run it: both files call two 1-second waits, then await both.
Node runs all your JavaScript on an event loop from the start. Python starts one only when you ask, with its asyncio library: asyncio.run(main()).
Top-level await fails in a script: SyntaxError: 'await' outside function. Notebooks already run a loop, so asyncio.run() fails there: write await main() in the cell.
Run it: both errors print. The two warnings after them come from work(), which the failed asyncio.run() never awaited.
To run coroutines at the same time, make them tasks with asyncio.create_task(). A task is a coroutine the event loop has scheduled.
In code: make both tasks first, then await them. The two 1-second waits overlap, so the total is about one wait, not two.
asyncio.gather() is Promise.all: results come back in order, and the first error is raised to you. In both, the other calls keep running.
For a load test, use gather(..., return_exceptions=True), the twin of Promise.allSettled. Failures come back as values, so you can count them.
asyncio.TaskGroup (Python 3.11 and newer) cancels the other tasks when one fails. Prefer it for new code. JavaScript promises cannot be cancelled.
Run both files: one line per way. gather and Promise.all raise at once, yet the rest finish; the TaskGroup cancels the rest.
A task you do not store can vanish before it ends: the event loop does not keep it alive. Store it, or use a TaskGroup.
Quick check wait_one_second() is an async def function that sleeps 1 s. In Python: a = wait_one_second(), b = wait_one_second(), then await a and await b. About how long does it take?
Quick check You gather five requests and one raises an error. What happens to the other four?
Quick check A script with await main() at the top level stops with SyntaxError: 'await' outside function. What is the smallest fix?
One loop, many streams
Guess first Five async def tasks each call chapter 1's chat(), which uses urllib. gather runs them, and the server has five free slots. When do the first words arrive?
An event loop is one thread. Tasks take turns: each runs until it waits, then the next runs. Node's event loop works the same way.
JavaScript's await lets queued promise callbacks run first. Python's await on a coroutine runs it straight through: other tasks wait for a real wait.
Run both: Node lets the promise callback in after the first await. Python runs busy straight through; the other task waits.
While fetch waits for the network, Node's loop runs other code. Python's urllib, even inside async def, holds the loop's one thread while it waits.
Compare: speed.mjs awaits fetch. stairs_demo.py calls chapter 1's blocking chat() inside async def, with no await on it.
Five such calls hit a server with five free slots. Their first words came one after another, a staircase, and nothing warned you.
To see it, run_fake.py starts chapter 1's fake server with five slots. It loads fake_ollama.py, so save the two files in one folder.
stairs_demo.py sends five calls through gather. It imports chapter 1's timing.py, whose chat() blocks: save it in the same folder.
Run it: start the fake server, then make the five blocking calls. Read the sent column: each call left after the one before it.
It looks like chapter 1's queue, when Ollama served one request at a time. Here the server was free. The queue was in your client.
await asyncio.to_thread(func, ...) runs a blocking function in a worker thread, so the loop stays free. All five calls now leave at once.
Run it: the same five calls, through asyncio.to_thread.
to_thread uses at most min(32, your CPU count + 4) threads: 16 on a 12-core Mac. Start 32 calls, and 16 wait in your client.
Run it: 32 calls of 1 s each. The last two lines give the total time and how many calls waited for a free thread.
A semaphore, asyncio.Semaphore(n), caps how many requests run at once: n go through, the rest wait.
async with is with for objects that wait on the way in or out: a semaphore for a free place, a TaskGroup for its tasks.
async def with yield is an async generator, like async function*. async for reads it, like for await. Async HTTP clients stream this way.
Run both: the same three lines. A plain for on an async generator fails in both languages: it needs async for or for await.
blocking_demo.py runs five 1-second waits three ways: polite awaits asyncio.sleep, rude calls time.sleep, threaded uses to_thread.
Debug mode names the blocker: with PYTHONASYNCIODEBUG=1 or -X dev, asyncio logs each step slower than 100 ms. Turn it off before you measure.
Quick check Five requests through gather arrive one after another, because chat() uses urllib. What is the smallest fix?
Quick check Your client starts 32 requests with to_thread on a laptop with 12 CPU cores. What happens?
Quick check lines() is an async def function with yield. Inside another async def, which loop reads it?
Timeouts and errors
Guess first urlopen(req, timeout=2). Call A streams 5 tokens, one per second. Call B sends nothing for 3 seconds, like a cold start. Which fails?
urllib's timeout limits each wait for data, not the whole call. With timeout=2, a token every second worked; a 3-second silence failed.
timeout_demo.py asks the drip simulator for both calls, A and B, and reads each with timeout=2.
Run it: call A finishes, with a token every second. Call B stops after 2 s of silence.
The error is a bare TimeoutError: timed out, not a URLError. Chapter 1's speed.py catches only HTTPError and URLError, so a read timeout crashes it.
Clients mean two things by timeout: the longest silence allowed, or a limit on the whole call. Check which before you stream a long answer.
A 200 status does not mean the stream worked. If Ollama fails after it starts sending, the error arrives as one more line: {"error": ...}.
fetch returns a 404 quietly, so chapter 1 checks res.ok. urllib raises HTTPError for you; catch it before URLError.
Quick check urllib with timeout=5. The server takes 40 s to stream 200 tokens and never pauses more than 1 s. What happens?
Quick check Your stream returned HTTP 200, but its last line is {"error": "..."}. Did the call work?
Which clock, and where it starts
Guess first During a request, your laptop sets its clock back 2 seconds to match an internet time server. You timed TTFT with time.time(). What do you see?
Each language has two clocks. Date.now() and time.time() read the wall clock. performance.now() and time.perf_counter() are stopwatches for durations.
The computer can reset the wall clock, even backwards. A monotonic clock never goes back.
Chapter 1's twins differ here: speed.mjs times with Date.now(), speed.py with perf_counter(). Your port uses perf_counter(), the stopwatch.
Where does your clock start? Follow one request: it is scheduled, may wait in your client, is sent, may wait on the server, then streams.
The wait inside your client is client queue time. A user feels everything from scheduled to the first answer token, client queue included.
Real case: vLLM's own benchmark started its timer after its semaphore, so it never counted the client queue. It was fixed on 2026-09-01.
Stamp three moments per request: scheduled, sent, first answer. Report client queue time, sent minus scheduled, beside TTFT. Build it, part 2, does this.
p95 is the time that 95% of requests stay under. Python's statistics.quantiles can put it above your slowest sample.
The default method guesses beyond your data, so pass method='inclusive'. With few samples, report the count and the slowest time, not a p99.
Quick check Which clock do you use to time TTFT in Python?
Quick check A semaphore caps your client at 4 requests, and your TTFT timer starts after acquire(). Requests arrive faster than the server can serve them. The reported p99 is:
Quick check Ten latencies; the slowest is 1,000 ms. statistics.quantiles(data, n=20)[18] says the p95 is 1,369 ms. What do you report?