BitNet
A local LLM inference framework runs compatible 1-bit models on ARM and x86 CPUs, with GPU kernels and MIT-licensed code.
BitNet is Microsoft's local LLM inference framework for developers and researchers working with compatible 1-bit models. Its bitnet.cpp runtime focuses on faster inference and lower energy use on your own hardware.
- CPU and GPU inference: Run BitNet b1.58 models with optimized kernels on ARM and x86 CPUs, or use the available GPU inference kernels.
- Conversational models: BitNet-b1.58-2B-4T supports interactive chat; compatible community models include Llama3-8B-1.58-100B-tokens and the Falcon3 and Falcon-E families.
- Text embeddings: BitNet-embedding-0.6B and BitNet-embedding-270M support optimized x86 CPU inference, with the smaller model intended for devices with limited resources. Neither embedding model has ARM kernel support.
The code is open source under the MIT license, with Windows and Debian/Ubuntu build support. Local inference runs on your CPU or GPU; the online demo runs separately. Model compatibility matters: this runtime targets supported 1-bit models, rather than general-purpose LLM inference.
