Optimum Quanto
An open-source PyTorch quantization library that reduces model memory use, supports CUDA and Apple MPS, and integrates with Hugging Face models.
Optimum Quanto is an open-source PyTorch library for developers who want to reduce the memory needed to run models on their own hardware. It handles language models and image-generation models, with integrations for Hugging Face Transformers and Diffusers. The project is in maintenance mode, accepting minor fixes and documentation work rather than major features.
Its main distinction is that quantization works in PyTorch's eager mode, including with models that can't be traced. Quantized models can run on PyTorch devices including NVIDIA GPUs through CUDA and Apple GPUs through MPS. Selected CUDA matrix operations have accelerated implementations, though speed depends on the model and the available kernels.
Quanto supports int2, int4, int8 and float8 weights, plus int8 and float8 activations. Lower precision reduces model memory use. Developers can quantize weights alone or include activations, and use calibration and quantization-aware training to address accuracy loss.
Hugging Face helpers cover quantizing, saving and reloading language models such as Llama. Diffusers support lets developers quantize individual components, such as a PixArt transformer, and reuse them in image-generation pipelines. A lower-level API also works with ordinary PyTorch models.
The library uses the Apache 2.0 license. It supports saving quantized weights with safetensors and PyTorch serialization, so converted models can be stored and loaded again.
