ExLlamaV2 and V3

ExLlamaV2 and V3 are local LLM inference libraries for developers building AI applications or hosting models on consumer GPUs. V2 is archived for now and isn't currently maintained; development continues in V3.

• Model quantization: V2 runs 4-bit GPTQ and EXL2 models, with mixed precision to balance memory use against accuracy; V3 uses EXL3 and includes model conversion tools.
• Multiple GPUs and CPU offloading: V3 can distribute inference across GPUs and offload parts of large mixture-of-experts models to the CPU when GPU resources are limited.
• Model support: V3 supports families including Llama, Mistral, Qwen and GPT-OSS, plus multimodal models such as Gemma 3 and Qwen 3-VL.
• Serving concurrent requests: Both libraries support dynamic batching and speculative decoding; V2 also supports streamed responses and prompt caching to reuse work across requests.
• Application connections: TabbyAPI provides an OpenAI-compatible API for both libraries, with local or remote access, embedding model support and Hugging Face chat templates; V3 also has a Transformers plugin.

Both libraries are free and open source under the MIT license. They run on Windows and Linux with NVIDIA GPUs through CUDA, on your own machine or a server you control. V3's CPU offloading supplements GPU inference.

For a browser interface to V2, ExUI provides single-user chat and notebook modes. V2 also works with text-generation-webui and lollms-webui, while TabbyAPI connects it to SillyTavern.