Tools to Run AI Models Locally

RealtimeSTT is a Python library for developers building voice assistants, dictation tools and apps that turn live audio into text. It can show text while someone speaks and produce a final transcript when they finish.

MiniCPM-V is a family of local vision-language models for developers building applications that interpret images and video. It can run on phones and other hardware you control.

ncnn is a C++ inference framework for developers adding on-device AI to mobile, desktop and embedded applications. It focuses on a small memory footprint and doesn't require third-party runtime dependencies.

llama-swap is a local AI proxy for people who want several models available to their apps without keeping them all loaded. It starts the server a request needs and replaces the active server when another model is requested.

Private Mind is a mobile AI assistant for people who want to chat and read documents without sending their content to a cloud service. It runs AI and document processing on your phone.

Pinokio is a local AI app launcher for people who want to try community projects without handling each app's terminal commands themselves. It brings app discovery and management into a desktop interface.

Docker Model Runner lets developers run AI models on their own computers or servers and connect them to applications. It combines local inference with model distribution through container registries.

OlliteRT lets you use an Android phone as a private AI server for apps on your local network. It's for people who want to run models locally for chat clients or home automation without a cloud inference service.

Nexa SDK is an on-device AI framework for developers building apps that run models on users' hardware. It covers language, vision and audio tasks through a shared interface for local inference.

Flair is a free Python library for developers and researchers who need to extract information from text or train their own natural language processing models. It combines pretrained text analysis models with tools for building word and document embeddings, all within a framework built directly on PyTorch.

Sonar serves language and multimodal models for developers and teams running AI applications on their own hardware. It builds on vLLM and works with Hugging Face-compatible models.

OGX, formerly Llama Stack, is an AI application server for developers building chat apps, document search and AI agents. It brings model access and the services those applications need into a server you can host yourself.

TensorRT-LLM is an inference library for developers serving LLMs on their own NVIDIA GPUs. It supports application-level inference and server deployments, with performance optimizations for workloads spread across one or several machines.

Kokoro-FastAPI generates speech on your own machine or server using Kokoro-82M. It's for developers adding narration to reading apps, audiobooks or voice assistants through an OpenAI-compatible speech API.

OpenLLM is a self-hosted LLM server for developers connecting applications to models on their own hardware or servers. Its OpenAI-compatible interface lets existing clients use those models without adopting a separate API.

Riffado turns recordings from Plaud Note, Note Pro, and NotePin devices into searchable transcripts and summaries. It's for recorder owners who want control over AI processing and storage. Recording sync requires a Plaud account and access to Plaud's cloud, so the full workflow isn't offline.

RealtimeTTS is a Python text-to-speech library for developers building voice assistants and other apps that need spoken responses before the full text is ready. It accepts both complete text and incoming LLM output.

Vocaldi is a paid Mac app that combines voice typing with read-aloud for people who prefer speaking to typing or listening to written text. Transcription, dictation cleanup and speech synthesis run on your Mac, so the audio and text you process stay on the device. It requires Apple Silicon and macOS 14 or later.

llama.cpp is a local LLM inference engine for developers building AI applications or running models on hardware they control. It supports language and vision-language models, with command-line tools and a server API for using the engine directly or connecting other software.

Triton Inference Server, offered by NVIDIA as Dynamo-Triton, is a self-hosted AI inference server for teams deploying models in applications. It brings models from different frameworks into one serving system across on-premises, cloud and edge hardware.

InsightFace is a face and person analysis toolkit for developers building identity verification, access control and media applications. Its Python tools and recognition server can process data on your own hardware.

llama-cpp-python is for developers adding local LLM inference to Python applications or connecting existing API clients to models on their own hardware. It brings llama.cpp into a Python library and a self-hosted server.

Radiant Camera is an iPhone camera and photo editor for people who want automatic scene adjustments with room to fine-tune the result. All image processing, including AI, runs on your device. It combines capture and editing in one app, with scene detection for portraits, pets, macro shots, night scenes, and landscapes.

Qube is a desktop AI assistant for people who want to chat with models and work with private documents on their own hardware. It combines a document library with persistent memory and research tools that provide citations you can inspect.

Sandbook is an offline text-to-speech reader for iPhone and iPad, for people who want to listen to books and documents without uploading them. It runs Kokoro-82M on the device through Apple MLX, with the speech model and voices included in the app. It needs no account and has no ads or tracking.

WhisperLiveKit is a self-hosted speech-to-text server for meeting transcripts, live captions and apps that need speech recognition as people talk. It processes audio on your own machine or server.

MLX Omni Server lets developers use local models through both OpenAI-compatible and Anthropic-compatible APIs on an Apple Silicon Mac. It's for applications built around those SDKs that need inference on hardware the developer controls. Apple's MLX framework provides hardware acceleration, and all AI processing stays on the Mac.

NobodyWho Chat is an on-device AI assistant for people who want help with writing, brainstorming, coding and questions without sending their conversations to a hosted model. Language-model inference runs on the phone, and chat works offline without an account.

FastDeploy is a self-hosted inference toolkit for teams deploying language and vision-language models on Linux servers. Built on PaddlePaddle, it supports offline inference and online serving.

Harbor helps people experiment with local LLMs by connecting model backends, chat interfaces and supporting services. Its CLI and companion app manage those connections in a shared development environment.

Pico MLX Server is a macOS app for people who want to run MLX models locally and use them with an existing AI chat client. It gives MLX Server a graphical interface and exposes local models through an OpenAI-compatible API. The project is open source under the MIT license and requires macOS Sonoma or later.

FastEmbed is a Python library for developers building semantic search and retrieval-augmented generation (RAG). It turns text into vectors and evaluates search results on your own hardware without requiring a cloud embedding API.

Accelerate is a Python library for developers and researchers who write their own PyTorch training loops. It handles hardware and distributed execution so they can keep control of how their models train.

MOSS TTS generates speech for developers and creators working on narration, dubbing and other long audio content. It supports consistent voices across long passages and can clone a speaker from reference audio without separate training for that voice.