Torch-TensorRT
An open-source PyTorch inference compiler that uses TensorRT on NVIDIA GPUs, with Linux and Windows support and Python or C++ deployment.
Torch-TensorRT compiles PyTorch models with TensorRT to reduce inference latency on NVIDIA hardware. It's for developers running models on their own machines or servers who want faster execution while keeping their PyTorch workflow. The project is open source under the BSD 3-Clause license.
Its main distinction is its integration with PyTorch's compilation and export APIs. Through torch.compile, it can optimize a model when it first runs, with few changes to existing code. The torch.export path lets developers compile ahead of deployment and save an optimized module for reuse.
Those two approaches suit different deployment needs. Just-in-time compilation fits applications that already run models in Python. Ahead-of-time compilation produces saved modules that can run in PyTorch or in a C++ application through libtorch. C++ deployment uses TorchScript and doesn't require a Python runtime; ExportedProgram deployment uses Python.
Hardware support centers on NVIDIA GPUs. Torch-TensorRT supports Linux on AMD64 and SBSA systems, plus Windows through its Dynamo compilation path. Jetson devices support source-built deployments on GPUs and NVIDIA's DLA inference hardware. NVIDIA also distributes it in the NGC PyTorch container, which includes compatible dependencies and example notebooks.
