masterinference.com

The course

Every part and chapter, in order, and what each one teaches. Chapters with a link are open now.

Part 1

Start here: the whole game

After it you can run and measure a model; do a first "will it fit?"

  1. Chapter 1: Your first model, measuredRun a small model on your laptop, call it over HTTP, and measure its speed.Free
  2. Chapter 2: What just happened: tokens, the loop, the two phases, and "will it fit?"Explain how a model writes one token at a time, why the first token takes longest, and whether a model file fits in GPU memory.Free
  3. Chapter 3: Python for JavaScript developers (optional bridge)Read and write the Python the labs use, starting from the JavaScript you already know.Free

Part 2

Just enough ML for inference

Planned. After it you can use PyTorch for inference; read model files, tokens, layers.

  1. Chapter 4: PyTorch and Transformers for inference (no training)Load a model with PyTorch and write your own decode loop on a laptop, with no training.Not built yet
  2. Chapter 5: A model is a file of numbersOpen a model's files and explain what its weights, config and tokenizer are and how big they are.Not built yet
  3. Chapter 6: Tokens in depth: tokenizers, templates, sampling, thinking and tool-call textPredict token counts, format prompts correctly, control sampling, and find the thinking and tool-call text in a raw output.Not built yet
  4. Chapter 7: Inside a transformer, for engineersFollow one token through a transformer and point to where the weights and the KV cache come from.Not built yet

Part 3

Memory, bandwidth and speed

Planned. After it you can open a free GPU and time it right; do memory math; explain slow decode.

  1. Chapter 8: Your first GPU, free: open it, check it, time it rightOpen a free GPU, check its driver and versions, time its work correctly, and read its memory.Not built yet
  2. Chapter 9: Will it fit? Weights and number formatsWork out how much GPU memory a model's weights need in any number format, and pick a GPU that fits.Not built yet
  3. Chapter 10: The KV cache: memory that grows with every tokenSize the KV cache for any model, context length and user count, and predict when memory runs out.Not built yet
  4. Chapter 11: What a GPU is, for software developersRead a GPU spec sheet and explain why memory bandwidth, not compute, limits most inference.Not built yet
  5. Chapter 12: Why decode is slow: the roofline in plain wordsPredict a model's fastest possible tokens per second on a GPU, and explain why batching raises throughput.Not built yet

Part 4

Serve it and measure it

Planned. After it you can use Linux, Docker and Kubernetes; run vLLM; load test with noise; work an overload incident.

  1. Chapter 13: Linux, Docker and Kubernetes in one weekendRun, inspect, break and fix an inference-shaped service in Docker and on a local Kubernetes cluster, with no GPU.Not built yet
  2. Chapter 14: vLLM on a free GPU without pain or a surprise billInstall vLLM on a free GPU within an hour, with pinned versions and a shutdown rule, and fix the setup errors you meet.Not built yet
  3. Chapter 15: Your first vLLM serverServe a model with vLLM's OpenAI-style API and read its startup log and main settings.Not built yet
  4. Chapter 16: Load testing that tells the truthRun a load test that is repeatable, measure its noise, and read a real latency and throughput curve.Not built yet
  5. Chapter 17: Queues, overload and SLOsPredict when a server falls over, read the signs of an overload incident, and protect the server with limits and SLOs.Not built yet

Part 5

Inside the engine

Planned. After it you can build a scheduler with a KV allocator; tune batching, paged KV, scheduling, caching; compare engines.

  1. Chapter 18: Batching: from one request to hundredsBuild a continuous batcher with tests, and measure what batching gives one user and what it gives everyone.Not built yet
  2. Chapter 19: Paged KV memory and preemptionBuild a KV block allocator with preemption, and tune an engine's memory so requests stop getting preempted.Not built yet
  3. Chapter 20: Scheduling: chunked prefill, CPU overhead, CUDA graphs, the TTFT-ITL tradeStop long prompts from freezing other users, and explain where CPU overhead, CUDA graphs and compilation fit.Not built yet
  4. Chapter 21: Prefix caching: reuse instead of recomputeMake repeated prompt prefixes nearly free, design prompts that hit the cache, and measure the hit rate.Not built yet
  5. Chapter 22: Read a real engine, then choose oneTrace a decode step through a small engine's code, then choose a production engine with evidence.Not built yet

Part 6

Faster and cheaper

Planned. After it you can quantize, speculate, constrain output, serve reasoning and tool calls.

  1. Chapter 23: Quantization: smaller models, same answers?Serve a quantized model, measure its gains, and prove that its answers hold up.Not built yet
  2. Chapter 24: Speculative decoding: faster when it works, slower when notTurn on speculative decoding, measure how often the draft is accepted, and find where it stops paying.Not built yet
  3. Chapter 25: Structured output and multi-LoRA servingReturn JSON and tool calls that always parse, and serve many adapters on one base model.Not built yet
  4. Chapter 26: Serving reasoning and tool-calling modelsServe reasoning and tool-calling models so the right fields come back, cap thinking cost, and prove tool-call accuracy with a fixed test set.Not built yet

Part 7

Production: ship it, keep it up, keep it correct

Planned. After it you can API, Kubernetes, operators, routing, scaling, observability, incidents, cost, capacity.

  1. Chapter 27: Go for platform engineers (optional bridge)Read and write enough Go to build a gateway or a Kubernetes controller in it.Not built yet
  2. Chapter 28: The API layer: streaming, cancellation, limits, auth, TLSBuild a production API in front of an engine, with streaming, cancellation, rate limits, keys and TLS.Not built yet
  3. Chapter 29: Containers, Kubernetes and operators for GPU inferenceRun an engine on Kubernetes with GPUs, honest health checks and safe rollouts, and write a small operator.Not built yet
  4. Chapter 30: Routing and load balancing for LLMsRoute requests so caches stay warm and no replica melts, behind a gateway that can fail over between providers.Not built yet
  5. Chapter 31: Autoscaling, cold starts and the OS under the serverScale on the right signal, cut cold starts from minutes to seconds, and explain the operating-system work inside a cold start.Not built yet
  6. Chapter 32: Observability: see performance, quality and hardwareBuild dashboards and alerts that catch latency, capacity, quality and GPU problems early.Not built yet
  7. Chapter 33: Correctness, reliability and incidentsKeep answers right and the service up with deploy gates and load shedding, and work real incidents from the first alert to the postmortem.Not built yet
  8. Chapter 34: The serving plan: requirements, capacity and costTurn a vague request into a serving plan with SLOs, a model choice, a fleet size and a cost per million tokens.Not built yet
  9. Chapter 35: Fleet and capacity: regions, quotas and failing GPUsKeep enough healthy GPUs in the right regions, and pull a failing node out of service safely.Not built yet

Part 8

Scale out

Planned. After it you can tensor parallelism (core), expert parallelism and MoE, disaggregation, KV tiers.

  1. Chapter 36: Multi-GPU basics: tensor parallelismSplit a model across GPUs and predict when more GPUs stop helping.Not built yet
  2. Chapter 37: Big models: pipeline and expert parallelism, MoEExplain how the largest mixture-of-experts models run across many GPUs, and read a real production design.Not built yet
  3. Chapter 38: Disaggregation, KV tiers and long contextDecide when to split prefill from decode, and when to move the KV cache to CPU memory or disk.Not built yet

Part 9

GPU performance and kernels

Planned. After it you can profile (core), test numerics, write kernels.

  1. Chapter 39: Profiling: find where the time goesProfile a request from Python down to GPU kernels and name the bottleneck with evidence.Not built yet
  2. Chapter 40: Numerics and determinismExplain why one prompt can give different outputs, and test numerics across batch sizes and versions.Not built yet
  3. Chapter 41: GPU programming for inference engineersWrite and time your first GPU kernels, and explain how threads, warps and memory set their speed.Not built yet
  4. Chapter 42: Attention kernels and backendsExplain FlashAttention and paged-attention kernels, and fix unsupported-backend errors.Not built yet

Part 10

Beyond chat: speech, vision and customer work (electives)

Planned. After it you can serve speech, vision and embedding models; run a customer proof of concept.

  1. Chapter 43: Speech: speech-to-text and streaming text-to-speechServe speech-to-text and text-to-speech models to real-time targets, and measure the time to first audio.Not built yet
  2. Chapter 44: Vision, image generation and embeddingsServe vision-language, image generation and embedding models, and explain how their bottlenecks differ from chat.Not built yet
  3. Chapter 45: The proof of concept: from a customer brief to a memoTurn a customer's brief into a working proof of concept, a benchmark and a one-page memo.Not built yet

Part 11

Show your work: proof, interviews and the capstone

Planned. After it you can build proof, drill interviews, finish a take-home, ship the capstone.

  1. Chapter 46: The job market and your trackRead a job post's real level and skills, and name the track and the portfolio you will build.Not built yet
  2. Chapter 47: Proof of work: open source and write-upsPublish work a hiring manager can check in minutes, including a pull request a maintainer reviewed.Not built yet
  3. Chapter 48: Interview practicePractice system-design, coding and debugging interviews under time limits, and finish one take-home.Not built yet
  4. Chapter 49: Capstone: ship a production inference serviceShip a production inference service for one real use case, proven by numbers, a live drill and a design review.Not built yet