Skip to content
Athrael.net logoAthrael.net

Introduction to GPU InferenceChapter 2: Serving inference

Serving architectures

Lesson 1 of 3 · about 11 min

What this lesson covers

So far, each request has been one piece of work for a GPU, from the prompt to the last token of the reply. A request goes through two phases that use the GPU in different ways: the model first processes the prompt, then generates the reply. This lesson explains these two phases, how they affect each other when they share a GPU, and how a service can run them on separate GPUs instead.

Prefill and decode

In the first phase of a request, called prefill, the model processes the whole prompt in one pass. In the second, called decode, it generates the reply one token at a time. Prefill ends when the first token is produced, so it accounts for most of the time to first token once a request has a slot. Decode sets the time per output token.

During prefill the model stores intermediate results for every token of the prompt in the GPU’s memory, in what is called the key-value cache, or KV cache. Each decode step reads the cache and adds one token’s results to it, which saves the model from processing the whole text again for every new token. The KV cache is the memory that each reply in progress occupies, and it grows with the length of the prompt and the reply.

Prefill computes all of a prompt’s tokens in parallel and occupies most of the GPU for a short time, while decode computes one token per step for each request in the batch, over the hundreds of steps a reply takes.

Aggregated serving

In aggregated serving every replica runs both phases. This is the setup the course has described so far, and the default for most inference servers. A request stays on one replica from its prompt to its last token, and its KV cache remains in that GPU’s memory.

Aggregated serving
Load balancer0 waiting
Replicas · prefill and decode
Replica 1
Replica 2
Replica 3
Replica 4

+ Each request stays on one GPU− A long prompt slows the replies in progress

Each square is a slot on a replica, purple while it runs prefill and teal while it runs decode.

The two phases then compete for the same GPU. When a long prompt arrives, its prefill takes GPU time that the replies in progress would have used for decode steps, so their tokens arrive more slowly until it finishes. Servers usually reduce this by splitting the prompt into chunks and processing them between decode steps, which slows the prefill itself.

Disaggregated serving

Disaggregated serving gives the two phases separate replicas. A prefill replica processes the prompt and sends the resulting KV cache over the network to a decode replica, which generates the reply. The replicas of each kind form a pool, and the load balancer selects one from each for every request.

Disaggregated serving
Load balancer0 waiting
Prefill · 2 replicas
Replica 1
Replica 2
KV cache
Decode · 2 replicas
Replica 3
Replica 4

+ Replies keep their rate during a long prompt− Each KV cache crosses the network

Each square is a slot on a replica, purple while it runs prefill and teal while it runs decode.

A long prompt then occupies only a prefill replica, and the replies on the decode replicas keep their rate. Time to first token and time per output token can also be managed separately, by adding replicas to the pool that is falling behind.

The KV cache for a long prompt can take several gigabytes, so the pools need the fast connections found in data centers, and the transfer adds to the time to first token. With short prompts or light load, aggregated serving often performs as well and is simpler to operate.

Pool allocation

With two pools, an operator has to decide how many replicas go in each. Long prompts with short replies, such as summarizing a document, need more prefill capacity, short prompts with long replies need more decode capacity, and this mix changes through the day.

Static allocation

With static allocation the split is set when the service is deployed, and an operator changes it by hand. It is the simplest to run and works well while the requests stay close to the mix it was planned for. When the mix moves away from it, requests queue for one pool while the other has free slots.

Static allocation
OperatorSplit set at deployment
Load balancer3 waiting
Prefill · 2 replicas
Replica 1
Replica 2
KV cache
Decode · 2 replicas
Replica 3
Replica 4

+ The simplest to run− A fixed split can idle one pool

Each square is a slot on a replica, purple while it runs prefill and teal while it runs decode.

Adaptive allocation

Adaptive allocation moves replicas between the pools while the service runs. A planner measures the load on each pool and moves a replica to the one falling behind, with a cold or hot swap.

In a cold swap the replica finishes its requests in progress, stops, and restarts in the other role, loading the model into memory again, which can take several minutes for a large model. That suits shifts lasting hours, such as the difference between day and night.

Cold swap
PlannerWatching the load
Load balancer2 waiting
Prefill · 2 replicas
Replica 1
Replica 2
KV cache
Decode · 2 replicas
Replica 3
Replica 4

+ The split follows shifts lasting hours− A restart can take several minutes

Each square is a slot on a replica, purple while it runs prefill and teal while it runs decode.

In a hot swap the replica can run either phase and changes role while its model stays loaded. A change takes a small fraction of the time a restart does, so the split can follow demand as it is measured.

Hot swap
PlannerWatching the load
Load balancer2 waiting
Prefill · 2 replicas
Replica 1
Replica 2
KV cache
Decode · 2 replicas
Replica 3
Replica 4

+ Roles change in place, model stays loaded− Load is measured continuously

Each square is a slot on a replica, purple while it runs prefill and teal while it runs decode.

Comparing the architectures

The diagram below runs eight replicas behind a load balancer. The menus at its top choose the architecture, the pool allocation for disaggregated serving, and how the requests arrive.

Serving architectures
Architecture
Pool allocation
Arrivals
UsersSent 00 per second
Load balancerPrefill queue 0Decode queue 0
UsersServed 00 per secondFirst token in 0.0 s0 tokens per second
Every replica runs both prefill and decode.Disaggregated serving with a fixed split between the pools.Adaptive allocation where a replica changes pool by restarting.Adaptive allocation where a replica changes role while its model stays loaded.Model in memoryKV cachePrefillReplies in progressThe figures are illustrative.

Aggregated serving is the simplest to run and works well with short prompts or light load. Disaggregated serving keeps replies steady under long prompts, at the cost of moving each KV cache across the network. Static allocation suits requests that stay close to one mix. Adaptive allocation follows a changing mix: a cold swap suits changes over hours, and a hot swap follows changes within seconds but needs replicas that can run either phase.

Recap

A request has two phases, where prefill processes the prompt in one pass and fills the KV cache, and decode generates the reply from it one token at a time. Aggregated serving runs both phases on every replica, where a long prefill slows the replies in progress, and disaggregated serving places them on separate pools and transfers the KV cache between them. The split between the pools can be static or adaptive, where a cold swap restarts a replica in the other role and a hot swap keeps its model loaded.

The next lesson covers a few widely used serving frameworks and introduces Narwhal, which we will use in the lesson after it to set up our first live inference server.