Favicon of TensorRT

TensorRT

An AI inference SDK optimizes models for NVIDIA GPUs on PCs, servers and edge devices, with ONNX, PyTorch and Hugging Face integration.

Screenshot of TensorRT website

TensorRT is NVIDIA's inference toolkit for developers deploying AI applications on NVIDIA GPUs. Its compilers, runtimes and model optimizations target lower response latency and higher throughput on local machines, self-hosted servers and edge devices.

  • Model integration: Import ONNX models or use PyTorch and Hugging Face integrations for language, vision, audio and diffusion workloads.
  • LLM inference: TensorRT-LLM accelerates language model inference on NVIDIA data center and workstation GPUs.
  • Model compression: Model Optimizer supports quantization, pruning and distillation, with deployment to TensorRT, TensorRT-LLM, vLLM and SGLang.
  • Self-hosted serving: NVIDIA Dynamo Triton serves optimized models with dynamic batching and concurrent model execution.
  • RTX applications: TensorRT for RTX targets GeForce RTX and RTX Pro laptops and desktops, with inference engines portable across operating systems and GPUs.

Inference runs on the target NVIDIA GPU. Linux and Docker build environments are supported, and deployment targets include Jetson and NVIDIA DRIVE. The separate TensorRT Cloud service builds optimized engines on NVIDIA's cloud infrastructure and requires approved access.

The Apache 2.0 components include plugins, the ONNX parser and sample applications. They cover only part of the full TensorRT SDK.

Similar to TensorRT