llama.cpp vs vLLM, SGLang and TensorRT-LLM explained

This overview compares inference engines through the problems they were designed to solve. The speaker starts with PyTorch as a general tensor library that also supports inference, then explains why autoregressive models prompted more specialized approaches.

For people who want to run models locally, the llama.cpp discussion focuses on consumer hardware, quantization and memory mapping. The speaker describes how it uses the memory hierarchy rather than assuming a model fits entirely in GPU memory. Mapping weights into virtual memory lets the operating system load pages on demand.

The comparison then moves to serving multiple users. vLLM addresses growing KV caches with PagedAttention and uses continuous batching to admit requests as others finish. SGLang's RadixAttention reuses cached computation for shared prompt prefixes. The speaker cites a reported throughput improvement of up to 6.4 times, while stressing that the result depends on the workload.

TensorRT-LLM appears as Nvidia's approach to optimizing inference around its hardware and software stack. A sponsored Zapier interlude demonstrates a news scanner built through its CLI with Claude Code. The closing discussion notes that these engines have developed overlapping features, so their original design goals provide context rather than an exhaustive account of their current capabilities.