vLLM and SGLang: LLM serving and batching explained

This lecture introduces vLLM and SGLang as inference engines for serving models to many users. Its running example is Llama 3.1 8B with 16-bit weights on a single 80 GB H100. The speaker calculates about 16 GB for the weights alone, then considers the memory needed for each user's KV cache.

The explanation separates prefill, which processes available prompt tokens in parallel, from decode, which generates tokens sequentially. It covers why attention reuses earlier keys and values without retaining earlier queries, and how longer context windows reduce the number of concurrent requests that fit in GPU memory.

For self-hosted deployments, the lecture explains PagedAttention's allocation of noncontiguous cache blocks and continuous batching's reuse of slots as requests finish. It also introduces cache-aware routing across replicas and SGLang's RadixAttention for reusing shared prompt prefixes. The speaker contrasts these serving frameworks with llama.cpp for individual desktop or phone use.

The final cost comparison uses rough workload assumptions rather than measured benchmarks. The speaker revises some concurrency assumptions during the calculation, so the projected GPU savings are illustrative. The animations and serving simulator do not run actual inference in the background.