Athrael.netWhat this lesson covers
Previously we came across terms like time to first token, tokens per second and throughput. This lesson expands on them and introduces a few more important ones.
Time to first token
Time to first token, or TTFT, is the time between sending a prompt and receiving the first token of the reply.
Part of it is queue time, while the request waits for a free slot on the GPU. The rest is the model processing the whole prompt before it can generate anything, which takes longer for a longer prompt.
So when the first token is slow, the cause is usually a busy server or a long prompt.
Time per output token
Once the reply starts, tokens arrive one after another, and time per output token, or TPOT, is the average gap between them.
Tokens per second counts how many of those gaps fit in one second, so a TPOT of 20 milliseconds equals 50 tokens per second.
Most people read about four words a second, which is five or six tokens, so a reply generated faster than that is ready before you reach it, and a slower one makes you wait for each word.
End-to-end latency
End-to-end latency is the time from sending the prompt to the last token of the reply, which comes to the time to first token plus one TPOT for every token after the first.
With a TTFT of 300 milliseconds and a TPOT of 20, a 10-token reply is done in about half a second, most of which is the time to first token. A 500-token reply takes about 10 seconds, nearly all of it spent generating tokens.
That is why TTFT matters most for short answers and TPOT for long ones.
Throughput and goodput
Everything so far describes a single request, whereas throughput describes the whole server: the tokens it produces each second across all the requests it is serving.
A bigger batch raises throughput but slows each request down, so a server run at its highest throughput can have a TTFT and TPOT that are too slow to be useful. Goodput accounts for that by counting only the requests that met a target, such as a first token within half a second.
As more requests arrive, throughput rises and then levels off, while goodput rises with it at first and then falls as more requests miss the target.
Percentiles
A server handles thousands of requests and each has its own TTFT, so the results are usually summarized in a single number. The average is the common choice, but it can look acceptable even when some requests took several times longer.
A percentile states the share of requests that were faster than a given time. The median, or 50th percentile, is the time that half of the requests were faster than, and the 95th percentile, written p95, is the time that 95 requests in 100 were faster than, which makes it a measure of the slowest requests.
A service is usually judged on its p95 or p99, because those describe its slowest requests.
Recap
The speed of one reply is described by time to first token, which covers queue time and prompt processing, and time per output token, which is the gap between tokens after that, and the two together give end-to-end latency. Throughput describes the whole server, goodput counts only the requests that met a target, and percentiles such as p95 report the slowest requests.
You will find these numbers on provider pages, benchmark tables and model comparisons, and teams use them to decide whether a setup is good enough. When someone says a model is fast, it is worth asking which number they mean and at what load.
Servers are also measured on things like how often requests fail, how long a model takes to load and how much energy it uses, but those are beyond this lesson. Memory and cost come up in the next one.