Threads, the GIL and processes
After this you can say why serious load-test tools run several Python processes, and run one that does.
One thread at a time
Guess first Say a Python function that only computes takes 1 s. You run 4 copies on 4 threads at once, on a laptop with 8 or more cores. Total time?
In the default Python build, one thread runs Python code at a time. A lock called the GIL makes sure of it. Threads take turns.
Same code in both languages: run a job that only computes (CPU work) once, then four copies on four threads.
Run it, and compare each language's two lines. Node's worker threads run in parallel; Python's threads take turns on the GIL.
Common mistake: the GIL is gone in 2026. It is not: a separate free-threaded build (python3.14t) runs threads in parallel, supported but optional.
Check yours: True means the GIL is on in your Python.
Quick check Default Python build. Which is true of Python threads?
Quick check It is October 2026. Which is true?
Quick check sys._is_gil_enabled() prints False. What do you know?
Threads for waiting, processes for work
Guess first One thread in your client stamps each message the moment it arrives, one every 10 ms. You add one CPU-busy thread to the same process. How late are the stamps now?
Rule: work that waits goes to threads or asyncio. Python CPU work goes to processes: a ProcessPoolExecutor starts one per CPU.
gil_demo.py times four CPU jobs three ways: one after another, on four threads, and on four processes. Then it times four waits.
Run it: compare the threads row and the processes row with the one-job row. Threads take turns on the GIL; processes side-step it.
Waiting is different: a waiting thread lets go of the GIL, so four 1 s waits on four threads overlap. See the run's last row.
Chapter 1's five_at_once.py starts five threads that each wait on the network, so for that job threads were right.
A stamp taken in a thread waits while another thread holds the GIL: a turn lasts 0.005 s or longer.
busy_thread.py sends a time stamp every 10 ms from another process, while 0, 1 or 3 threads beside the reader burn CPU.
Run it: compare the rows with 0, 1 and 3 busy threads. While a busy thread holds the GIL, the stamps wait.
Quick check Which job gets faster on 4 Python threads?
Quick check Your client receives tokens in one thread. Another thread of the same process runs slow Python code on each token. What happens to the arrival stamps?
Starting a child: spawn and the main guard
Guess first macOS, Python 3.13. Your script starts a child process at the top level of the file, with no if __name__ == "__main__": guard. What happens?
Python has three ways to start a child process: fork copies the parent, spawn starts a fresh Python, forkserver copies a clean helper process.
Run it: start_cost.py starts and joins four children three times with each method. On macOS it skips fork.
Careful with the name: Node's child_process.fork() does not copy the parent. It starts a fresh Node, like Python's spawn, not Python's fork.
With spawn, the child imports your main file first. Start code outside the guard runs again there and raises RuntimeError. The main guard stops that.
no_guard.py starts a spawned child at the top of the file, outside any main guard, then prints the child's exit code.
Run it: the child fails, yet the shell says 0.
The parent carries on: the child's exitcode is 1, yet the script exits 0. Check exitcode, or use ProcessPoolExecutor: it raises when a worker dies.
A child gets only what you pass it. Globals you change under the guard, and a random.seed(42) there, do not reach it.
Run it: spawn_globals.py changes MODEL under the guard. The spawned children still see the old value.
seeds.py runs three spawned workers twice each way: with random.seed(42) only in the parent, then with a seed in each worker.
Run it: seeded only in the parent, two runs give different numbers. Seeded with base + worker id, they repeat.
Arguments travel to the child as bytes (pickling). Functions travel by name only, so with spawn a lambda or a notebook function fails.
pickle_errors.py sends three things to a spawned worker: a lambda, a function typed into python -c, and a top-level function.
Run it: the lambda and the python -c function fail in the worker. The top-level function works.
Quick check Traceback: RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. Smallest fix?
Quick check Which Python start method works like Node's child_process.fork()?
A client that tells the truth
Guess first One Python process reads 25,600 chunks a second and spends 45 µs on each. Does it keep up?
A load-test client is a queue too. One event loop takes one chunk at a time and stamps it at its turn, not on arrival.
Utilization: chunks per second × seconds of work each. Below 1, the loop keeps up. Near 1, waits get long. Above 1, they keep growing.
Worked example: 512 streams × 50 tokens a second = 25,600 chunks a second. At 45 µs each: 1.15 s of work per second.
On a laptop the loop stays nearly idle. Example: 4 slots at 250 tokens/s send about 1,000 chunks a second: utilization 0.045.
Single-process tools fall behind at hundreds to thousands of requests a second: asked for 1,000, vLLM's tool managed 146.
Would the free-threaded build fix this? Not by itself: one event loop runs on one thread. More loops spread the load; tools use processes.
Make every client report on itself: achieved against asked rate, its own CPU share, and its queue time, apart from TTFT.
The checklist for a multi-process client: five habits you have met. load_client.py follows them all, and Break it runs it.
Quick check Your turn. 128 streams at 80 tokens per second each. Your client spends 60 µs of work on each chunk. Utilization of one client process?
Quick check Two example tables. A: sent climbs 0.00, 0.32, 0.64, 0.96 s, while each request's total_duration stays about 0.32 s. B: every sent is 0.00, while total_duration climbs 0.33, 0.65, 0.98 s. Which shows a client queue?
Quick check Your single-process client reports 146 requests per second when you asked for 1,000, and an example TTFT p99 of 4 s. What do you know?