TensorRT Model Optimizer
A Python model optimization library that compresses Hugging Face, PyTorch and ONNX models for TensorRT-LLM, TensorRT, vLLM and SGLang.
TensorRT Model Optimizer (ModelOpt) is a Python library for developers who need smaller models and faster inference. It accepts Hugging Face, PyTorch and ONNX models and lets developers combine optimization techniques before deployment.
- Model compression: Post-training quantization reduces model size, while pruning removes unnecessary weights to cut memory use and accelerate inference.
- Accuracy and smaller models: Quantization-aware training and distillation can refine quantized model accuracy; distillation also trains smaller models to reproduce the behavior of larger ones.
- Deployment exports: Export optimized checkpoints for TensorRT-LLM, TensorRT, vLLM and SGLang, with Hugging Face export supporting both transformers and diffusers models.
- Further inference optimization: Combine compression with neural architecture search, sparsity and speculative decoding through the library's Python APIs.
ModelOpt is open source under Apache 2.0. Its quantization support includes language, vision-language, diffusion and ONNX models, plus Windows workflows. Model and technique compatibility follows separate support matrices.
Training-based optimization integrates with NVIDIA Megatron-Bridge, Megatron-LM and Hugging Face Accelerate. NVIDIA also publishes pre-quantized checkpoints on Hugging Face for deployment with TensorRT-LLM, vLLM and SGLang.
Logotyper
