vLLM, TGI, SGLang and TensorRT-LLM comparison

This session compares vLLM, Text Generation Inference (TGI), SGLang and Triton TensorRT-LLM through Wildberries' production experience. Recorded on October 6, 2025 and uploaded on October 20, 2025, it describes the team's self-hosted infrastructure on in-house GPUs, Kubernetes and HPC clusters. Runtime assessments belong to that recording; the speaker mentions TensorRT-LLM reaching version 1.0.0.

Aleksandr Shirokov explains a custom benchmark that measures time to first token, throughput, GPU memory consumption and replica startup time. It also checks answer quality and how many requests meet business latency targets. His central recommendation is to test each workload rather than assume one runtime wins everywhere.

The comparison covers attention backends, speculative decoding, KV cache reuse and weight loading. Shirokov describes vLLM's speculative decoding support as experimental at the time. He also reports that placing several models on one GPU tied their scaling together, while adding GPU cache usage to autoscaling metrics produced a more stable signal.

The gateway section explains how the team uses Kong for model access, token-based rate limits and monitoring. A custom plugin limits tokens per model, and an approval workflow provisions access through version-controlled configuration. The Q&A discusses library compatibility and the time available to optimize before deployment.