HQQ
An open-source model quantization library that compresses language and vision models without calibration data and works with Transformers and vLLM.
HQQ is a Python library for developers compressing language and vision models to fit the memory available on their own hardware. Its Half-Quadratic Quantization method doesn't require a calibration dataset.
- Weight compression: Quantize weights at 8, 4, 3, 2 or 1 bit, and give different layers different settings to balance model size and quality.
- Inference integrations: Use quantized models with Hugging Face Transformers or vLLM, with GemLite required for the vLLM backend.
- GPU inference: Use optimized CUDA/Triton kernels for supported configurations; settings that an inference backend can't handle fall back to a native backend.
- Fine-tuning: Train through Hugging Face PEFT and LoRA, or use HQQ+ with trainable low-rank adapters to improve quality at lower bit depths.
HQQ is open source under Apache 2.0 and works with PyTorch. It supports NVIDIA GPU workflows through CUDA, including optimized inference paths whose compatibility depends on the quantization settings. Those paths require axis=1 and impose backend-specific group-size restrictions, so every compression setting doesn't receive the same acceleration.
The Transformers integration can save and load quantized models through its standard model interface. Models saved directly through the HQQ library use a different format and can't be loaded with Transformers' from_pretrained method.
Logotyper
