TensorRT-LLM vs vLLM: RTX 6000 PRO benchmarks and setup

This comparison examines inference engines for a self-hosted model server, with Docker deployment examples and benchmark results. The benchmark setup uses Qwen3-32B-FP8 on an RTX 6000 PRO Blackwell with 96GB VRAM, CUDA 13.1 and driver 590.48.01. The listed environment is Ubuntu 24.04 with Docker, NVIDIA Container Toolkit and a Hugging Face access token.

The speaker compares standalone TensorRT-LLM and vLLM with Dynamo deployments. He explains how NATS carries messages between services and etcd tracks engine availability. Examples cover separate containers, commands inside one container, and a local mode without those two services. He also describes a CUDA compatibility workaround for vLLM and file permission issues around KV cache storage.

The local LLM tests use 1,000 prompts, ShareGPT conversations and synthetic random inputs, with a 10,000-token maximum. Most engine settings remain at their defaults. The speaker reduces concurrency after Dynamo crashes during burst testing and uses ten warm-up requests.

He reports that TensorRT-LLM achieves the best overall results and end-to-end latency in these tests. That finding applies to this hardware, model and workload. The practical guidance is to check model compatibility first, then compare latency and resource needs using prompts that resemble the intended application.