Serving multiple requests | Introduction to GPU Inference

What this lesson covers

Previously we looked at how a GPU answers a single prompt. A real server handles many prompts at once, and each one is called a request. This lesson explains how one GPU serves several requests together and how that affects the speed of each.

Batching

The simplest way to serve several requests is one at a time, where the GPU generates the first reply in full, then the second, and so on.

Each request gets the GPU’s full speed during its turn, but the last one waits for every reply ahead of it, so the wait grows with the queue.

Four requests, one GPU

One GPUClock

GPU cores in use

Request 1

Request 2

Request 3

Request 4

Each square beside a request is one token of its reply. The times and the share of cores in use are illustrative, with a step shared by four requests taking a little longer than a step for one.

One request usually uses only part of a GPU’s capacity. A server uses the rest by computing the next token for every waiting request in the same step, and the requests it processes together are called a batch.

A step for four requests takes slightly longer than a step for one, since there is more to compute, but it advances four replies. In the diagram the first request finishes slightly later than it did when served alone, and the other three finish much sooner.

Adding more requests

Each request in a batch gets fewer tokens per second than it would with the GPU to itself, and the figure drops as the batch grows. The GPU’s total output across all requests, known as throughput, rises at the same time.

What a bigger batch does

Requests in the batch

One GPU

Each request's speed

Total output (throughput)

Total output at every batch size

1 request16 requests

The numbers are illustrative, but real servers show the same shape: total output rises as requests are added, then levels off once the GPU is fully used.

Throughput stops rising once the GPU is fully used, because the same output is then divided among more requests and every reply slows down.

Continuous batching

A server sets how many requests a GPU processes at once, and each one occupies a slot. Replies differ in length, so a server that started a new batch only when the previous one had finished would leave slots empty while the longest reply was completed, with new requests waiting behind it.

Continuous batching avoids this by giving a free slot to the next waiting request as soon as any reply finishes.

Three slots, six replies

Waiting

Slot 1

Slot 2

Slot 3

Each numbered column is one step and each letter is a reply, while a hatched square is an empty slot.

The time a request spends waiting for a slot is part of its time to first token, and it grows when more requests arrive than the GPU has slots for.

The number of slots depends mostly on the GPU’s memory, because every reply in progress needs room for its text so far. A later lesson returns to this.

Recap

A GPU that serves requests one at a time makes each request wait for all those ahead of it, so a server computes the next token for several requests in one step, which is called a batch. A bigger batch raises throughput and slows each request slightly, until the GPU is fully used and throughput stops rising. With continuous batching, a waiting request takes a slot as soon as one becomes free.

An assistant can feel slower at busy times because more requests share each GPU, so tokens arrive more slowly and requests wait longer for a slot. Batching also keeps these services affordable, since one GPU serving many concurrent requests costs far less per reply than a GPU for each request.