Intel Neural Compressor

Intel Neural Compressor is a Python library for developers who want to compress AI models for deployment on their own hardware or servers. It supports LLMs and other deep learning models, with extensive testing on Intel platforms.

• Quantization methods: Apply static or dynamic quantization, SmoothQuant and weight-only quantization, or use quantization-aware training and mixed precision.
• Model support: AutoRound integration supports quantization of models including LLaMA, Qwen, DeepSeek, Flux and FramePack, with support for LLMs and vision-language models.
• Framework support: Work within PyTorch, TensorFlow or JAX; the project also supports compression with ONNX Runtime.
• Accelerator support: PyTorch workloads can use CPUs, GPUs and Intel Gaudi accelerators, including FP8 quantization on Gaudi2.

The library runs on your own hardware and is open source under Apache 2.0. Intel hardware support includes Core Ultra and Xeon processors, Gaudi accelerators, and Data Center GPU Flex and Max products. AMD CPUs, ARM CPUs and NVIDIA GPUs are also supported, but receive more limited testing.