llama-cpp-python
A Python library runs GGUF models locally through llama.cpp and provides a self-hosted OpenAI-compatible server with CPU and GPU support.
llama-cpp-python is for developers adding local LLM inference to Python applications or connecting existing API clients to models on their own hardware. It brings llama.cpp into a Python library and a self-hosted server.
- Python integration: Generate text completions and chat responses with LangChain and LlamaIndex compatibility, plus direct access to llama.cpp's C API.
- OpenAI-compatible server: Connect compatible clients to your own models, support multiple models, or provide a local Copilot replacement.
- Vision and tools: Compatible models can process image inputs, call tools and produce JSON output; image-capable choices include LLaVA, moondream2 and qwen2.5-vl.
- GGUF model loading: Use local model files or download GGUF models from Hugging Face Hub through the optional huggingface-hub package.
The library is open source under the MIT license and runs on Linux, Windows and macOS. It supports CPU inference, NVIDIA acceleration through CUDA, AMD GPUs through HIP or ROCm, and Apple Silicon through Metal. Vulkan and SYCL backends are also available.
Inference runs on the machine hosting the library or server. Hugging Face model downloads require access to that external service. Vision and tool-calling support depend on the selected model.
Logotyper
