Athrael.netWhat this lesson covers
When you ask an AI assistant such as ChatGPT or Claude a question, a GPU on a server somewhere does the work of answering it. This lesson follows one question through that machine, from the moment you press send to the last word of the reply.
You don’t need to write code or do any math. Each part has a diagram you can run yourself. If you’d rather listen, the player at the bottom of the page reads the lesson aloud and moves the page along with it.
What inference means
A language model is built in two stages. First it is trained: it works through a huge amount of data, most of it text, over weeks or months and learns from it. Then it is put to work, and each reply it writes is called inference. Training is done once, but inference runs every time someone sends a prompt.
- 1Your prompt travels to a server that has a GPU.
- 2The model runs on the GPU and works out a reply.
- 3The reply comes back a piece at a time.
Text becomes tokens
A model works with numbers, so your text is first cut into small pieces called tokens. A short word is usually one token and a longer word is often several. Each token is then swapped for a number.
The diagram below cuts a sentence into tokens and can show them as numbers, which is what the model receives. It works on any sentence you type into it.
Tokens are also the unit everything else is counted in: how much a model can read at once, how fast it writes, and what a reply costs.
One token at a time
A model writes its reply one token at a time. It reads everything so far, picks the next token, adds it to the text and reads again, which is why answers appear on your screen word by word.
The bar at the bottom shows two waits. The first is the pause while the model reads your prompt, before anything appears. The second is the steady pace of new tokens after that. People who run these systems measure both and call them time to first token and tokens per second.
Why a GPU
Picking one token takes billions of multiplications. Each one is easy, and most of them don’t depend on each other, so they can be done at the same moment.
A CPU, the main chip in a laptop, has a few workers that are each very fast. A GPU has thousands of simple ones. When the job is a pile of small independent sums, the many simple workers finish first.
The model has to fit in memory
A model is a long list of numbers. An “8 billion parameter” model is a list of 8 billion of them, and the whole list has to sit in the GPU’s own memory before the GPU can use it.
How much room the list takes depends on how precisely each number is stored. Storing them more roughly makes the model smaller and costs a little quality.
A 70 billion model does not fit on a 48 GB card until its precision drops to 4-bit. In practice you choose the model, the precision and the GPU together.
Recap
To recap, here is the whole lesson in five points.
- Inference is a trained model answering a prompt.
- Text is cut into tokens, and each token becomes a number.
- A reply is written one token at a time, after a short wait while the model reads the prompt.
- A GPU is fast at this because it does a great many small sums at once.
- The model has to fit in the GPU’s memory, and precision decides how big it is.