Athrael.netWhat this lesson covers
Previously we looked at inference on a single GPU. An assistant such as ChatGPT or Claude serves millions of users, many of them at the same time, which takes far more than one GPU. This lesson covers the hardware involved, how requests are distributed across it, how demand varies through the day and what it costs.
Consumer and data center GPUs
A consumer GPU and a data center GPU are similar chips built for different uses. A consumer GPU goes into a personal computer for games and creative work, and has its own fans and display outputs. A data center GPU runs models around the clock alongside other GPUs, and is cooled by the server it sits in.
Built for games and creative work on a personal computer.
Fast with small models, but short of memory for large models or concurrent requests.
Built to run AI models around the clock, alongside other GPUs.
Has the memory for large models and concurrent requests.
The two cards differ most in memory. A consumer GPU has 8 to 32 gigabytes, and a data center GPU often has 80 or more. Memory limits how large a model fits and how many concurrent requests a GPU can serve, so a high-end consumer GPU can be nearly as fast with a small model and a single request, while large models and many concurrent requests need a data center GPU. Data center GPUs are also linked by fast connections, so a large model can run across several of them and requests can be spread across many nodes.
Data centers
The GPUs that process your prompts sit in data centers. They are installed in servers called nodes, each with 4 to 8 GPUs, and the nodes are stacked in racks. A large data center can hold tens to hundreds of thousands of GPUs.
GPUs turn nearly all of their electricity into heat, so a large data center draws tens to hundreds of megawatts. Older servers are cooled by fans, but the densest racks produce more heat than air can carry away, so they pump liquid through a cold plate on each GPU.
This capacity is spread across several regions of the world, so that a request can be served close to where it was sent and the service continues if one data center goes offline.
Load balancing
One copy of a model runs on one GPU, or on several when the model is too large for one GPU’s memory. Each copy is called a replica, and a large assistant runs thousands of them, each processing its own batch of requests. Every request in progress keeps its conversation in the replica’s memory, so the memory left after the model is loaded limits how many slots it has.
When you send a prompt, the request first reaches a load balancer, which selects the replica to process it, usually one with a free slot and a short queue. With this many machines replicas fail regularly, and the load balancer sends requests to the others until the failed one is repaired.
Peak demand
Use of an assistant rises through the working day in each part of the world and falls at night, and it rises sharply when a new model or feature is released.
GPUs have to be bought, installed and powered months in advance, so a company decides early how many to have. If it has enough for the busiest hour, many are idle the rest of the day. If it has fewer, requests queue for a slot at the peak, which raises the time to first token, and replies slow down as batches grow.
- Served
- Queued
- Idle GPUs
- Capacity
Usage limits and notices that a service is at capacity both keep demand within what the GPUs can serve. Many providers also offer a lower price for requests that can wait up to a day, which uses the idle capacity at quiet times.
Costs
A data center GPU costs tens of thousands of dollars, and to that a company adds the building, the electricity, the cooling and the staff who operate it. Most of it is paid for whether the GPUs are busy or idle, so the cost of one reply depends on how many replies each GPU produces. Batching lowers it, because a GPU serving many concurrent requests spreads its cost across all of them.
Most services charge per token, since the computation a GPU does grows with the number of tokens. Output tokens usually cost more than input tokens, since a prompt is processed in one pass while a reply is generated one token at a time. A smaller model costs less because each token takes fewer calculations and more requests fit on one GPU.
Recap
A data center GPU is similar to a consumer GPU, with more memory and fast connections to other GPUs. These GPUs are installed in nodes and racks, and a large data center holds tens to hundreds of thousands of them. A model runs on them as many replicas, and a load balancer sends each request to one with a free slot. Demand changes through the day, so capacity is planned for the busiest hours and is partly idle the rest of the time. The cost of a reply depends on how many replies each GPU produces, so batching lowers it.
This lesson concludes Chapter 1, Foundations. Chapter 2, Serving inference, starts with inference serving strategies and then introduces Narwhal, which you can use to set up a live inference server of your own on a consumer GPU.