2,000+ AI tools you can run locally Add tool

llama.cpp

An MIT-licensed local LLM engine runs language and vision-language models on CPUs or GPUs, with quantization, hybrid execution and Docker support.

Screenshot of llama.cpp website

llama.cpp is a local LLM inference engine for developers building AI applications or running models on hardware they control. It supports language and vision-language models, with command-line tools and a server API for using the engine directly or connecting other software.

CPU and GPU execution: Run models on CPUs or accelerate them with Apple Silicon, NVIDIA, AMD and Intel GPU backends.
Lower memory use: Quantized models reduce memory requirements, while hybrid CPU and GPU execution can partially accelerate models that exceed available video memory.
Application integration: Use the C/C++ library or server REST API as the model engine behind an application.
Constrained output: Grammar support lets developers restrict generated output to a defined structure.

The project is open source under MIT and builds on the ggml library. It supports local execution, Docker deployment and operation on cloud hardware you choose. Hardware support includes Metal for Apple Silicon, CUDA for NVIDIA, HIP for AMD, SYCL for Intel GPUs, and Vulkan. Available memory and backend support determine which models a machine can run.

Self-Hosted Chat Web UIs Inference Libraries and Bindings Inference Servers and Clusters Local Model Runners Web Code execution GGUF Hugging Face integration MCP Multimodal input OpenAI-compatible API Prompt caching Quantization Reranking Speculative decoding Streaming inference Structured output Tool calling

Similar to llama.cpp

Gemma

138 videos

An open-weight local LLM family for phones, PCs and servers, with Ollama and LM Studio support and a JAX library for fine-tuning.

5.8KUpdated 1 month agoApache-2.0

Android#LM Studio integration#LoRA#Multilingual

Gemma is Google DeepMind's family of open-weight AI models for developers building applications on their own hardware. It includes compact models for mobile and IoT devices and larger models for reasoning on personal computers and servers.

Edge and Embedded AIEmbedding and Reranker ModelsLLM Fine-TuningOpen-Weight LLMsVision-Language Models

Ollama

261 videos

An MIT-licensed local LLM runner and API for macOS, Windows, Linux and Docker, with optional cloud inference and coding agent integrations.

186.9KUpdated 21 hours agoMIT

macOS · Windows · Linux · Docker#GGUF#llama.cpp backend#Multimodal input

Ollama runs open models for people who want local AI applications, model chat or a backend for coding assistants. It provides a command-line interface and an API, with a choice between inference on your hardware and its cloud service.

Inference Servers and ClustersLocal Model Runners

LM Studio

91 videos

A local LLM app runs downloaded models for chat and agent tasks, with MLX and llama.cpp support and voice transcription processed on your device.

lmstudio.aiComputer and Browser Agents

macOS · Windows · Linux#llama.cpp backend#MCP#MLX

LM Studio runs downloaded language models for conversations and tasks on your computer. Its Bionic agent helps people create documents, write software and carry out work using local or frontier open models.

Computer and Browser AgentsDesktop Chat AppsLocal Model RunnersPersonal AI AssistantsSpeech-to-Text and Transcription

vLLM

58 videos

An open source LLM serving engine runs models on your hardware, with an OpenAI-compatible API and support for NVIDIA, AMD and CPU inference.

96.9KUpdated 20 hours agoApache-2.0

macOS · Docker#Batch processing#Distributed execution#GGUF

vLLM serves language models on hardware you control. It's for developers and teams building model APIs that need to handle concurrent requests while using memory and compute efficiently.

GPUs, Clusters and EdgeInference Servers and Clusters

OpenCode

114 videos

An open source AI coding agent works in your terminal or IDE with local models or cloud models including Claude, GPT and Gemini.

226.7KUpdated 1 day agoMIT

macOS · Windows · Linux#Code execution#Human approval#Multi-agent workflows

OpenCode is an AI coding agent for developers who want help writing code or exploring unfamiliar projects. You can choose a local model running on your hardware or connect a cloud provider.

Autonomous Coding AgentsTerminal Coding Agents

MLX

25 videos

An open-source machine learning array framework with shared CPU and GPU memory on Apple silicon and Linux CPU or CUDA support.

29.2KUpdated 1 week agoMIT

macOS · Linux#Distributed execution#LoRA

MLX is a machine learning array framework for researchers and developers building and testing models on their own hardware. It supports model training and inference, with APIs for numerical work and neural networks.

Inference Libraries and BindingsTraining Frameworks