TensorRT-LLM

A self-hosted LLM inference library runs on NVIDIA GPUs, with OpenAI-compatible serving, a Python API and support for deployments across machines.

TensorRT-LLM is an inference library for developers serving LLMs on their own NVIDIA GPUs. It supports application-level inference and server deployments, with performance optimizations for workloads spread across one or several machines.

API serving: Serve models to OpenAI-compatible clients or use the Python LLM API within applications, with support for multimodal requests and embeddings.
Inference optimization: Quantization, in-flight batching, cache reuse and speculative decoding help reduce memory use or improve throughput, depending on the model and hardware.
Distributed workloads: Run across multiple GPUs or machines, including deployments that separate prompt processing from token generation; integrations include NVIDIA Dynamo and Triton Inference Server.
Adapted models and structured output: LoRA support handles model adaptations, while guided decoding supports output constrained by JSON schemas.

The PyTorch-based architecture lets developers extend the runtime and customize models in Python. Supported models include DeepSeek-R1, Llama and GPT-OSS-120B. Benchmarking and evaluation tools help assess serving performance.

Inference runs on your hardware. Anonymous usage telemetry is enabled by default and can be disabled. It sends deployment and hardware metadata, but excludes prompts, generated outputs, model weights and local model paths.

GPUs, Clusters and Edge Inference Libraries and Bindings Inference Servers and Clusters Docker Batch processing Distributed execution LoRA Multimodal input OpenAI-compatible API Prompt caching Quantization Speculative decoding Structured output Works offline

Nobody has claimed TensorRT-LLM yet

Work on TensorRT-LLM? Claim this page and we'll verify you by hand. TensorRT-LLM gets the verified checkmark, and you can upgrade the listing.

Want more people to find it? Promote it

Something wrong or outdated on this page?

Logotyper

Got an idea for an AI tool? I'm Eugene, the designer and developer behind Localhosted. I'll help you build it, from the first sketch to launch.

Start your project

Similar to TensorRT-LLM

View all tools

vLLM

58 videos

An open source LLM serving engine runs models on your hardware, with an OpenAI-compatible API and support for NVIDIA, AMD and CPU inference.

96.9KUpdated 20 hours agoApache-2.0

macOS · Docker#Batch processing#Distributed execution#GGUF

vLLM serves language models on hardware you control. It's for developers and teams building model APIs that need to handle concurrent requests while using memory and compute efficiently.

GPUs, Clusters and EdgeInference Servers and Clusters

llama.cpp

92 videos

An MIT-licensed local LLM engine runs language and vision-language models on CPUs or GPUs, with quantization, hybrid execution and Docker support.

134.9KUpdated 21 hours agoMIT

Web#Code execution#GGUF#Hugging Face integration

llama.cpp is a local LLM inference engine for developers building AI applications or running models on hardware they control. It supports language and vision-language models, with command-line tools and a server API for using the engine directly or connecting other software.

Inference Libraries and BindingsInference Servers and ClustersLocal Model RunnersSelf-Hosted Chat Web UIs

SGLang

13 videos

A self-hosted AI inference framework serves language, vision-language and diffusion models, with OpenAI API compatibility and broad hardware support.

38.8KUpdated 21 hours agoApache-2.0

#Batch processing#Distributed execution#LoRA

SGLang is a self-hosted inference framework for teams serving AI models to applications. It supports production workloads on a single GPU or across distributed clusters, including AI agents and reinforcement learning rollouts.

GPUs, Clusters and EdgeInference Servers and Clusters

TensorRT-LLM

A self-hosted LLM inference library runs on NVIDIA GPUs, with OpenAI-compatible serving, a Python API and support for deployments across machines.

Nobody has claimed TensorRT-LLM yet

Videos about TensorRT-LLM

llama.cpp vs vLLM, SGLang and TensorRT-LLM explained

Ollama, llama.cpp, LM Studio and vLLM compared

NVIDIA Dynamo explained: distributed LLM serving

TensorRT-LLM vs vLLM: RTX 6000 PRO benchmarks and setup

Similar to TensorRT-LLM

vLLM

llama.cpp

SGLang

Similar to TensorRT-LLM

Nobody has claimed TensorRT-LLM yet

Videos about TensorRT-LLM

llama.cpp vs vLLM, SGLang and TensorRT-LLM explained

Ollama, llama.cpp, LM Studio and vLLM compared

NVIDIA Dynamo explained: distributed LLM serving

TensorRT-LLM vs vLLM: RTX 6000 PRO benchmarks and setup

Similar to TensorRT-LLM

vLLM

llama.cpp

SGLang

NVIDIA Dynamo

Sonar (formerly Aphrodite Engine)

FunASR