SGLang and vLLM: how inference engines work

Charles Frye explains the machinery behind a model-serving API, using SGLang and vLLM as reference points. The workshop focuses on cloud deployments, with lessons relevant to operating self-hosted inference rather than a walkthrough for a desktop local LLM app.

He follows a request through server IO, tokenization, scheduling, GPU execution and detokenization. Prefill processes the input; decode produces the output tokens. Frye connects those phases to time to first token and inter-token latency, then explains why chatbots, background agents and document processors need different deployment choices.

The performance discussion covers batch construction, KV cache capacity and shared prefixes. Frye describes the scheduler as a potential bottleneck even though the GPU performs most of the computation. CUDA graphs reduce repeated CPU launch work, while speculative decoding uses a separate model to propose tokens for the target model to check together.

A museum placard generator provides the production example. Frye reads rising latency and queues from a Modal dashboard and explains how additional replicas relieve congestion. He also recommends evaluating the actual deployment, logging token IDs to investigate tokenizer bugs, and collecting metrics and traces for performance regressions. The closing resources include small reference engines for studying the architecture in code.