Tools to Run AI Models Locally

Nunchaku is a local diffusion inference engine for developers and image-generation users who want to run models such as FLUX with less GPU memory. It focuses on faster generation with compressed models on NVIDIA hardware.

pyannote.audio is a Python toolkit for developers and researchers who need to separate recordings by speaker. Speaker diarization identifies who spoke when, so applications can associate parts of an audio recording with different speakers.

LLM is a command-line tool and Python library for developers who want to use language models in terminal workflows or applications. It supports models running on your machine alongside remote providers such as OpenAI, Anthropic and Google.

exo is a local LLM runner for people who want to run models too large for a single machine's memory. It joins devices into a cluster and runs inference on your own hardware.

Msty is a family of AI products for individuals, teams and organizations working with models and their own knowledge. Msty Studio is its workspace product, with separate products for agents and centralized model access.

Sentence Transformers is a Python library for developers building search and document retrieval on their own hardware. It helps match content by meaning and rank results against a query.

Fluid is a local AI assistant for Mac users who want to ask questions without sending their chats to a hosted model. It runs Meta Llama 3 on your Mac and supports offline conversations.

Icebreaker is a local AI chat app for people who want to converse with language models running on their own hardware. It uses llama.cpp, with Hugging Face also part of its model stack.

Transformer Engine is a library for developers building and running Transformer models on NVIDIA GPUs. It reduces compute and memory demands in training and inference through low-precision arithmetic and optimized model components.

Needle is an AI model for developers building device assistants that turn requests into actions. It prioritizes automation over general chat, with smaller model variants for devices that have limited resources.

Diffusers is a Python library for developers and researchers building image, video and audio generation into applications or running models on their own hardware. It supports pretrained generation pipelines as well as custom diffusion systems.

ms-swift is a Python framework for developers and researchers training and deploying language and multimodal models on their own hardware. It covers model adaptation through evaluation and serving, with support for Qwen3, DeepSeek-R1, Llama4, Mistral, Qwen3-VL and InternVL3.5.

Flash Linear Attention is a Python library for researchers and developers building language models on their own hardware. It supplies reusable components for exploring linear attention, sparse attention and state space architectures.

Fullmoon is a local LLM chat app for people who want conversations to stay on their Apple devices. It runs models on-device and works fully offline, with chat history saved locally.

GLiNER is a Python framework for extracting named entities from text using small models that run locally on CPUs and consumer hardware. It's for developers building document processing, personal data detection or search systems who need to choose their own entity types without training a model for each task.

DiffSynth-Studio is a diffusion model engine for developers and researchers generating media or training models on their own hardware. It combines inference and training, with memory-saving features for consumer GPUs.

ClearML is a self-hostable MLOps platform for researchers and engineering teams managing model development and production. It connects experiment records with datasets, training jobs and model deployment so teams can trace results back to the work that produced them.

mistral.rs is an inference engine for developers who want to run AI models on their own computer or server. It supports browser chat and inference embedded directly in Python or Rust applications.

Kokoro is a text-to-speech model and inference library for developers adding speech generation to personal projects or production applications. It runs on your own hardware and uses the compact Kokoro-82M model.

DwarfStar is a local LLM inference engine for people who want to run large DeepSeek, GLM and Qwen models on their own computers or servers. It focuses on selected models and hardware setups, and requires the project's own GGUF files rather than accepting arbitrary GGUF downloads.

ReScript is a video and audio editor for people cutting interviews, podcasts and voice notes through text. Deleting words from a transcript removes the matching sections of the recording.

Draw Things creates and edits images on iPhone, iPad and Mac. It's for artists and other creators who want to work offline and keep generation on their own hardware.

Odysseus brings AI chat, autonomous agents and personal work into one workspace on your own hardware. It's for people who want models to work with their documents and ongoing tasks in a self-hosted app.

LitGPT is a Python toolkit for developers and researchers who want to train and deploy language models on their own hardware. Its directly editable model implementations suit custom applications and research experiments.

SmolChat is an Android app for chatting with language models on your phone. It gives people who want on-device AI a chat interface for their own GGUF model files, including small language models.

Open Wispr is a voice dictation tool for people who want to speak into Mac apps while keeping audio and text on their computer. It inserts transcribed speech at the cursor in editors, browsers, terminals, email and other applications.

OpenSuperWhisper is a speech-to-text app for people who want to dictate or transcribe saved recordings on an Apple Silicon Mac. It runs Whisper and Parakeet models locally.

node-llama-cpp is a library for developers adding local AI to JavaScript and TypeScript applications. It uses llama.cpp to run GGUF models on your hardware and works in Node.js, Bun and Electron.

Apache TVM is a machine learning compiler for developers optimizing models for deployment on their own hardware. It compiles model computations into modules that applications can embed, with targets spanning data center GPUs and edge devices. Code is licensed under Apache 2.0.

Enclave AI is a local AI assistant for people who want to chat, work with documents or use voice conversations on their Apple devices. Downloaded models handle these tasks offline.

Willow Inference Server is a self-hosted speech server for developers building voice assistants, transcription apps, and accessible interfaces. It handles speech recognition and speech generation on your own hardware.

Aphelion is a desktop app for people who want character conversations, creative writing and coding help from models running on their own computer. It works offline after the first model download and doesn't require an account.

Claude Code Local lets developers use Claude Code with a model running on their own computer. It suits people who want to keep coding offline or continue working when they hit cloud usage limits.

FlashInfer is a GPU kernel library for developers building LLM inference systems on their own hardware or servers. It supplies the attention, matrix multiplication and mixture-of-experts operations used by model serving software.