What this lesson covers
So far, each request has been one piece of work for a GPU, from the prompt to the last token of the reply. A request goes through two phases that use the GPU in different ways: the model first processes the prompt, then generates the reply. This lesson explains these two phases, how they affect each other when they share a GPU, and how a service can run them on separate GPUs instead.
Prefill and decode
In the first phase of a request, called prefill, the model processes the whole prompt in one pass. In the second, called decode, it generates the reply one token at a time. Prefill ends when the first token is produced, so it accounts for most of the time to first token once a request has a slot. Decode sets the time per output token.
During prefill the model stores intermediate results for every token of the prompt in the GPU’s memory, in what is called the key-value cache, or KV cache. Each decode step reads the cache and adds one token’s results to it, which saves the model from processing the whole text again for every new token. The KV cache is the memory that each reply in progress occupies, and it grows with the length of the prompt and the reply.
Prefill computes all of a prompt’s tokens in parallel and occupies most of the GPU for a short time, while decode computes one token per step for each request in the batch, over the hundreds of steps a reply takes.
Aggregated serving
In aggregated serving every replica runs both phases. This is the setup the course has described so far, and the default for most inference servers. A request stays on one replica from its prompt to its last token, and its KV cache remains in that GPU’s memory.
+ Each request stays on one GPU− A long prompt slows the replies in progress
The two phases then compete for the same GPU. When a long prompt arrives, its prefill takes GPU time that the replies in progress would have used for decode steps, so their tokens arrive more slowly until it finishes. Servers usually reduce this by splitting the prompt into chunks and processing them between decode steps, which slows the prefill itself.
Disaggregated serving
Disaggregated serving gives the two phases separate replicas. A prefill replica processes the prompt and sends the resulting KV cache over the network to a decode replica, which generates the reply. The replicas of each kind form a pool, and the load balancer selects one from each for every request.
+ Replies keep their rate during a long prompt− Each KV cache crosses the network
A long prompt then occupies only a prefill replica, and the replies on the decode replicas keep their rate. Time to first token and time per output token can also be managed separately, by adding replicas to the pool that is falling behind.
The KV cache for a long prompt can take several gigabytes, so the pools need the fast connections found in data centers, and the transfer adds to the time to first token. With short prompts or light load, aggregated serving often performs as well and is simpler to operate.
Pool allocation
With two pools, an operator has to decide how many replicas go in each. Long prompts with short replies, such as summarizing a document, need more prefill capacity, short prompts with long replies need more decode capacity, and this mix changes through the day.
Static allocation
With static allocation the split is set when the service is deployed, and an operator changes it by hand. It is the simplest to run and works well while the requests stay close to the mix it was planned for. When the mix moves away from it, requests queue for one pool while the other has free slots.
+ The simplest to run− A fixed split can idle one pool
Adaptive allocation
Adaptive allocation moves replicas between the pools while the service runs. A planner measures the load on each pool and moves a replica to the one falling behind, with a cold or hot swap.
In a cold swap the replica finishes its requests in progress, stops, and restarts in the other role, loading the model into memory again, which can take several minutes for a large model. That suits shifts lasting hours, such as the difference between day and night.
+ The split follows shifts lasting hours− A restart can take several minutes
In a hot swap the replica can run either phase and changes role while its model stays loaded. A change takes a small fraction of the time a restart does, so the split can follow demand as it is measured.
+ Roles change in place, model stays loaded− Load is measured continuously
Comparing the architectures
The diagram below runs eight replicas behind a load balancer. The menus at its top choose the architecture, the pool allocation for disaggregated serving, and how the requests arrive.
- Aggregated
- Disaggregated
- Static
- Cold swap
- Hot swap
- Constant
- Poisson
- Bursty
- Ramp
- Spike
- Diurnal
Aggregated serving is the simplest to run and works well with short prompts or light load. Disaggregated serving keeps replies steady under long prompts, at the cost of moving each KV cache across the network. Static allocation suits requests that stay close to one mix. Adaptive allocation follows a changing mix: a cold swap suits changes over hours, and a hot swap follows changes within seconds but needs replicas that can run either phase.
Recap
A request has two phases, where prefill processes the prompt in one pass and fills the KV cache, and decode generates the reply from it one token at a time. Aggregated serving runs both phases on every replica, where a long prefill slows the replies in progress, and disaggregated serving places them on separate pools and transfers the KV cache between them. The split between the pools can be static or adaptive, where a cold swap restarts a replica in the other role and a hot swap keeps its model loaded.
The next lesson covers a few widely used serving frameworks and introduces Narwhal, which we will use in the lesson after it to set up our first live inference server.
Athrael.net