bitsandbytes

A PyTorch quantization library reduces local LLM memory use with 8-bit inference, 4-bit QLoRA fine-tuning and 8-bit optimizers.

bitsandbytes is a Python library for developers who need to run or fine-tune large language models on their own hardware with less memory. It works within PyTorch workflows.

8-bit inference: LLM.int8() roughly halves inference memory requirements while keeping higher precision for outliers to preserve model performance.
4-bit fine-tuning: QLoRA keeps the base model at 4-bit precision and trains a small set of added LoRA weights to reduce memory use.
8-bit optimizers: Quantized optimizer states reduce training memory use while aiming to retain the performance of 32-bit optimizers.
Quantized layers: Developers can use 4-bit and 8-bit linear layers in their PyTorch models.

The library is open source under the MIT license. It integrates with Hugging Face Transformers, Diffusers and PEFT.

Hardware support varies by platform and accelerator. The development branch supports CPUs and compatible NVIDIA, AMD and Intel GPUs on Linux and Windows, plus Apple Silicon CPUs and Metal on macOS. x86 CPU support requires AVX2; macOS requires Apple M1 or newer hardware. Some CPU and Metal 8-bit inference paths may lack performance optimizations. Intel Gaudi supports 8-bit inference and partial QLoRA functionality, but doesn't support 8-bit optimizers.