Tools to Run AI Models Locally

DeepSpeed is an open-source PyTorch library for developers and researchers training or running large AI models on their own hardware or compute clusters. It helps address memory limits and distribute workloads across devices.

ModelScope is an AI model hub with a Python library for developers and researchers who want to run models locally or adapt them to their own data. It covers language, vision, speech, multimodal tasks and scientific computing.

Locally AI is a native assistant for people who want to use language and vision models on their own Apple devices. It handles text, images and spoken conversations without sending AI processing to a cloud service.

Reef is self-hosted infrastructure for developers who want AI agents to learn from their interactions. It connects live requests and feedback to changes in model weights or the prompts, rules, and skills that guide an agent.

docTR reads text from PDFs and images for developers building document processing applications and researchers comparing OCR models. It locates individual words and recognizes their text on your own hardware.

OuteTTS is a local text-to-speech library for developers adding generated speech to their applications. It uses reference audio to reproduce a speaker's voice, including their style, emotion and accent.

WebLLM is an engine for developers building browser-based AI assistants and Chrome extensions. It runs language models on the user's device, so applications don't need a separate inference server to process prompts.

openWakeWord is a wake word detection library for developers building voice apps that listen for a chosen word or phrase. It lets an application detect voice triggers on its own hardware.

TensorSharp runs local AI models for .NET developers and people who want chat, media generation and agent tools on their own hardware. You can embed it in a C# application or use its CLI, browser interface or companion app, TensorAgent.

Text Embeddings Inference is a self-hosted server for developers building semantic search and retrieval applications. It turns text into embeddings through a REST API and can run offline once model weights are downloaded.

Google AI Edge Eloquent turns speech into edited text for people who prefer dictating notes and drafts to typing them. It uses on-device Gemma models to remove verbal filler and self-corrections.

PaddleNLP is a Python toolkit for developers training and deploying language models on their own infrastructure. Built on PaddlePaddle, it covers pretraining, fine-tuning, model compression and inference across several accelerator families.

MeloTTS is a Python text-to-speech library for developers adding spoken output to their applications. It runs locally and can generate speech in real time on a CPU, so a dedicated GPU isn't required for inference.

NVIDIA Dynamo is a self-hosted inference framework for teams serving generative AI models across multiple GPUs or server nodes. It coordinates inference engines so teams can manage capacity, response latency and cached context across a cluster.

Aiko turns recorded meetings, lectures and voice memos into text for people who want to keep their audio on their own device. It processes completed recordings rather than live speech and doesn't identify individual speakers.

RamaLama runs AI models for local chat or a self-hosted REST API. It's for developers who want a container workflow for trying models on their own hardware and serving them in production.

Transformers.js is a JavaScript library for developers adding on-device AI to web apps. It runs models in the browser, so processing inputs doesn't require a separate inference server.

Llama Desktop runs language models on your computer for private chat and use in other apps. It's for people who want a local ChatGPT alternative and developers connecting coding agents, editors or scripts to their own models.

WebBrain is an AI browser agent for people who want to ask questions about pages or delegate web tasks from a sidebar. It works in your existing browser, including signed-in tabs, and starts in read-only Ask mode.

Stable Diffusion Core ML is a developer toolkit for adding local AI image generation to Mac, iPhone, and iPad apps. It pairs model conversion tools with a Swift package for embedding generation in native software.

Civitai is a platform for creators to share AI image models, browse other people's work and exchange feedback. You can self-host its model-sharing platform, but image generation and training don't work in the local setup.

FlagEmbedding is a Python toolkit for developers building semantic search and retrieval-augmented generation (RAG). It runs embedding models to find relevant text and rerankers to refine the order of retrieved results.

Whisperian is an Android speech-to-text app for people who'd rather dictate than type, including writers, engineers and people handling everyday correspondence. It supports offline transcription as well as cloud services, and gives users configurable workflows as an alternative to WisprFlow or Superwhisper.

DeepEP handles communication between GPUs for teams training or serving mixture-of-experts (MoE) models on their own servers. It moves tokens to the GPUs that hold the selected experts and combines their outputs.

Cloak by Polyaura is an on-device AI app for people who want to chat and work with voice recordings on their iPhone or iPad without sending them to a cloud service. It works without an internet connection after the initial model download.

vllm-mlx is a local LLM inference server for developers who want to connect AI apps and coding assistants to models running on an Apple Silicon Mac. One process exposes both OpenAI-compatible and Anthropic-compatible APIs. It's open source under Apache 2.0 and runs on macOS with at least 8 GB of unified memory.

Google AI Edge Gallery is a local AI app for people who want to use and compare generative models on their own devices. Model inference runs on-device, including offline chat and media tasks.

MediaPipe is a toolkit for developers adding on-device AI to applications. It provides pretrained models and task APIs for applications that need to process images, video, text or audio on the user's device.

Osaurus runs AI agents on your Mac for tasks such as reviewing documents, organizing files, and writing code. It keeps agent memory on your computer, so you can change models without losing the context of your work.

ONNX Runtime is an AI inference and training engine for developers adding models to apps and services. It supports deployment on a user's device or a server, including applications that need the same model format across platforms.

SemanticFinder helps readers and researchers find passages by meaning in long documents. It runs the search model in your browser, so you can search your own text without sending it to a model server.

Linguist is a browser translation extension for people reading, communicating or studying across languages. You can choose on-device translation or connect a translation service that suits your needs.

Augmentoolkit turns documents into datasets for training an LLM on a particular subject. It's for people building custom models around material such as research papers or fictional lore.

Backyard AI is a desktop AI app powered by llama.cpp. The desktop app is deprecated and no longer supported, which limits its suitability for people choosing software for ongoing use.