GPTQModel

GPTQModel is a Python toolkit for developers compressing LLMs and running them on their own hardware or servers. It combines model preparation, quality evaluation and inference so you can assess compression choices within one workflow.

• Compression methods: Work with GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8 and EXL3, with model conversion and accelerated inference in the same toolkit.
• Selective compression: Apply different settings to individual layers or leave selected modules uncompressed, with controls for mixture-of-experts models.
• Quality checks: Evaluate models after quantization to check how compression affects their output.
• Runtime integrations: Use Hugging Face Transformers, Optimum and PEFT, or deploy compatible quantized models through vLLM and SGLang.

GPTQModel supports Linux, macOS and Windows, with backends for NVIDIA, AMD and Intel GPUs, Huawei Ascend NPUs, Apple Silicon and CPUs. Compatibility depends on the hardware, compression method and runtime. Multi-GPU processing can accelerate quantization.

GGUF, FP8, EXL3 and ParoQuant use native GPTQModel runtime paths. SGLang accepts selected GPTQ and AWQ formats, so its integration doesn't cover every supported compression method. GPTQModel also loads Prism/Bonsai GGUF checkpoints for inference.