PowerInfer
A local LLM inference engine that shares sparse model workloads across CPU and GPU, with NVIDIA acceleration on Linux and Windows.
PowerInfer is a local LLM inference engine for developers and researchers running large models on a PC with a consumer GPU. Its performance gains depend on sparse models, which use only part of their network for each input.
- Shared CPU and GPU workloads: It divides model computation between the processor and graphics card to reduce GPU memory demands and transfers between them.
- Sparse model support: It runs ReLU variants of Llama 2 and Falcon-40B, ProSparse Llama 2, and Bamboo-7B using its PowerInfer GGUF model format.
- Application serving: Server support and batched generation let developers use the engine as a model backend for their applications.
- llama.cpp compatibility: It can also run llama.cpp model weights, though those weights don't receive its sparse inference performance gains.
PowerInfer is open source under the MIT license and runs inference on your own hardware. Linux and Windows support x86-64 CPUs with AVX2, with or without an NVIDIA GPU. On macOS, it runs on Apple Silicon using the CPU only, where its performance improvement is limited.
Logotyper
