Skip to content
Athrael.net logoAthrael.net

Introduction to GPU InferenceChapter overview

Chapter 2: Serving inference

Where we left off

In Chapter 1 we followed inference from a single prompt on one GPU to serving a model to millions of users at once. We learned about batching and measures such as time to first token (TTFT), time per output token (TPOT) and goodput, and then looked at inference at scale.

This chapter looks at the architectures used to serve a model and the frameworks that run them, and then has you set up a live inference server of your own.

In this chapter

  • Serving architectures covers the two phases of a request, called prefill and decode, and the ways of running them on the same GPUs or on separate ones.
  • Serving frameworks covers the software that runs these architectures across many GPUs, and introduces Narwhal, an open-source framework for disaggregated inference.
  • Our first live inference server sets up a small server on your own computer and sends it requests.