About this course
This course explains how language models such as the ones behind ChatGPT and Claude run on GPUs, from a single prompt to services that answer millions of users. It’s for anyone who uses an AI assistant and wants to know what happens between pressing send and reading the reply. Everything is explained in plain language, each lesson has diagrams you can run in the page, and if you’d rather listen, the narration reads each one aloud.
Why this course
I’ve spent much of this year working on inference, first measuring how prefill and decode compete for GPUs under load, then building Narwhal, an open-source framework for disaggregated inference. I made this course to share what I’ve learned in a form anyone can follow, and to show you how to run an inference server of your own, starting in Chapter 2.
In this chapter
Chapter 1 covers the foundations in four lessons.
- From prompt to reply follows a prompt through a GPU, from tokens to a reply written one token at a time, and covers what decides whether a model fits in its memory.
- Serving multiple requests shows how one GPU serves many requests at once as a batch, and how that affects the speed of each.
- Measuring inference introduces the measures of a reply’s speed, such as time to first token and time per output token.
- Inference at scale covers the GPUs and data centers behind a large service, how a load balancer spreads requests across replicas, and what it all costs.
Athrael.net