Tools to Run AI Models Locally

Cog packages machine learning models for researchers and developers serving them on their own infrastructure. It handles the container environment and inference API around a Python model.

Transformers is a Python library for developers and researchers running pretrained AI models or training models on their own hardware. It covers text, images, audio, video and multimodal tasks through shared model definitions.

llmfit helps people choosing models for their own computers compare what will fit in memory and how fast it might run. It checks your hardware before recommending models, so you can assess candidates before downloading them.

MLX is a machine learning array framework for researchers and developers building and testing models on their own hardware. It supports model training and inference, with APIs for numerical work and neural networks.

Kai 9000 is a personal AI assistant for people who want remembered context and tools that can act on their requests. Its responses can include interactive screens such as quizzes, recipe cards and dashboards.

SGLang is a self-hosted inference framework for teams serving AI models to applications. It supports production workloads on a single GPU or across distributed clusters, including AI agents and reinforcement learning rollouts.

vLLM serves language models on hardware you control. It's for developers and teams building model APIs that need to handle concurrent requests while using memory and compute efficiently.

LightLLM is a self-hosted LLM inference and serving framework for developers and teams running models on their own GPU servers. Its focus is serving requests efficiently: it schedules work around predicted memory use and handles text encoding, model inference and output decoding asynchronously to improve GPU utilization.

text-generation-webui, also called TextGen, runs language models on your own hardware for chat, writing and application integrations. You can use it as a desktop app or a self-hosted browser interface.

Optimum Quanto is an open-source PyTorch library for developers who want to reduce the memory needed to run models on their own hardware. It handles language models and image-generation models, with integrations for Hugging Face Transformers and Diffusers. The project is in maintenance mode, accepting minor fixes and documentation work rather than major features.

FlashAttention is an attention library for developers training or running transformer models on their own GPUs. It computes exact attention while using less memory than standard PyTorch attention, allowing longer sequences.

ExecuTorch is an on-device AI runtime for developers adding PyTorch models to apps and embedded devices. It handles inference on the user's hardware across Android, iOS, macOS, Linux and Windows, as well as embedded systems and microcontrollers.

CTranslate2 is an inference library for developers who want to run Transformer models on their own hardware or servers. It focuses on reducing execution time and memory use for applications built in C++ or Python.

Notely Voice turns speech and recordings into editable notes for lectures, meetings and interviews. It runs Whisper on your device, so speech recognition works offline after you download a model and doesn't upload audio to a cloud service.

TensorRT is NVIDIA's inference toolkit for developers deploying AI applications on NVIDIA GPUs. Its compilers, runtimes and model optimizations target lower response latency and higher throughput on local machines, self-hosted servers and edge devices.

vLLM Metal lets developers serve language models on Apple Silicon Macs through vLLM. It's for people who want to use their Mac's hardware for local AI inference while keeping vLLM's API server and request scheduling.

Lucebox is a local AI workstation for developers and teams who want to run models on their own hardware. Its inference engine is also available separately for supported AMD and NVIDIA machines.

Coqui TTS is a text-to-speech toolkit for developers and speech researchers who want to generate voices locally or train their own models. The idiap fork builds on coqui-ai/TTS.

Wordflow is an AI writing assistant for writers and researchers who want to reuse editing prompts across their work. Its browser editor lets you revise a paragraph or selected passage with a choice of local and external models.

Locally Uncensored is a desktop AI studio for people who want to chat, create media and work on code on their own PC. It includes mainstream models and variants with fewer refusal restrictions.

ColPali is a document retrieval library for developers and researchers building search or retrieval-augmented generation systems. It matches text queries to page images, using visual content alongside text. The colpali-engine package is deprecated, and maintainers recommend Sentence Transformers for new projects and production use.

OpenCLIP is a Python library for developers and researchers building image-text matching and zero-shot classification applications. It supports pretrained CLIP models and training models on your own hardware.

Vyact is a desktop workspace for people who want a local LLM to help with files, writing and code. It brings conversations and searchable documents together, with optional connections to email services and Chrome.

DeepImageJ brings pretrained neural networks into ImageJ and Fiji for life-science researchers who want to analyze microscopy images without writing code or learning how to train models. It runs image-to-image predictions on your own computer, using a standard CPU or a compatible GPU.

Tellama is a local LLM chat app for Android users who want conversations to stay on their phone. Chat with an installed compatible model works offline, and you don't need an account.

Optimum is a collection of Python packages for developers adapting Hugging Face models to their deployment hardware. It extends Transformers, Diffusers, TIMM and Sentence Transformers with tools for faster inference and training.

KoboldCpp runs language models on your own hardware for chat, creative writing and roleplay. It's a fork of llama.cpp with a bundled KoboldAI Lite browser interface.

Chatterbox TTS Server turns text into speech on your own computer or server. It's for people making narration and audiobooks, or adding speech to apps through an API.

llmserve is a terminal app for people who keep local LLM models across several folders or inference tools. It brings model discovery and server management into one interface so you can choose an existing model and an available backend.

Kaito manages LLM inference, fine-tuning and document retrieval services in Kubernetes. It's for teams that want to run models in their own cluster with less manual work sizing GPU capacity and managing deployments.

Lollms WebUI is a local AI interface for writing, coding assistance and media generation through a choice of models. The project receives minimal maintenance, and its interface has no built-in user authentication.

Ollama Helm Chart packages a self-hosted LLM service for developers and infrastructure teams that already use Kubernetes. It brings Ollama into a cluster so applications can access models running on the cluster's hardware.

KTransformers is a research framework for developers running or fine-tuning large language models on their own hardware. It focuses on mixture-of-experts (MoE) models, using CPU resources alongside GPUs to work within limited GPU memory.

MTPLX is a native Mac app, command-line tool and local LLM server for people running chat or coding models on Apple Silicon. It uses a model's own multi-token prediction heads to accelerate generation without a separate draft model.