vLLM vs SGLang: GPU inference and KV cache tutorial

Harshul Jain and Tanmay Sah explain LLM inference for beginner and intermediate audiences, starting with GPU memory growth, time to first token and throughput. The workshop connects these symptoms to attention, KV cache storage and the different demands of prefill and decode. A capacity calculator explores how context length, concurrent users and latency targets affect GPU selection for self-hosted inference.

Mistral 7B provides the main teaching example. The quantization demo reports roughly 15 GB of model memory at FP16 and 7.5 GB at INT8. The speakers caution that compression needs quality testing and does not automatically improve throughput. They also cover attention variants and FlashAttention, then explain paged attention, continuous batching, prefix caching and KV quantization on the serving side.

The engine comparison uses the speakers' H100 benchmarks, which were run before the session. They report no statistically significant difference between vLLM and SGLang on their standard workload, but a three to four times advantage for SGLang in their agentic branching test. They explicitly warn that other setups may produce different results. Slides and Jupyter notebooks accompany the workshop, though network delays and GPU detection problems interrupt some demonstrations. The session also corrects its MLA savings example to 14 times relative to multi-head attention.