Tools to Run AI Models Locally

OpenVINO is an AI inference toolkit for developers deploying models in their own applications. It converts and optimizes models for local hardware or servers, covering generative AI and conventional machine-learning workloads.

Detectron2 is a Python library for developers and researchers building object detection and image segmentation applications. It provides model training tools and reusable components for computer vision research and production software.

LMCache helps teams serving LLMs reuse context that a model has already processed. It reduces repeated work on long prompts, conversation history and retrieved documents, shortening the wait for the first token and improving throughput.

AIBrix provides infrastructure components for teams serving large language models on Kubernetes. It's aimed at organizations that need to manage inference capacity, request routing and models across a cluster.

Xinference serves AI models on your own computer or servers for developers and researchers building applications around models they control. It brings chat, document search and media generation into one platform.

FastLLM is an open-source inference engine for people running large language models on their own machines or self-hosted servers. Its main draw is hardware flexibility: large mixture-of-experts (MoE) models can spread their weights across GPU memory, system RAM and disk when they don't fit entirely on a graphics card.

MNN is a C++ framework for developers running AI models on phones, PCs and embedded devices. It supports inference and training on the device, with a small application footprint and hardware acceleration.

wllama lets developers add local LLM inference to browser apps without a separate inference server. It brings llama.cpp to JavaScript and TypeScript projects through WebAssembly, so models run on the user's device.

Moondream is a vision language model for developers building apps that interpret images, such as warehouse or manufacturing footage. Its downloadable models can run on your own hardware, including offline in an air-gapped environment.

Sidekick is a native macOS app for people who want AI answers grounded in their documents and research materials. Its built-in llama.cpp engine runs GGUF models such as Qwen3.5 without a separate model runner.

DeePMD-kit is open-source software for molecular and materials researchers who want to train and run AI interatomic potentials on their own hardware. It learns atomic interactions from quantum-mechanical reference data, then uses those models in molecular dynamics simulations of molecules, solids and metals. It runs on CPUs and GPUs, with support for CUDA and ROCm source builds, and carries the LGPL-3.0-or-later license.

whisper.cpp runs OpenAI's Whisper speech recognition models on your own hardware. It's for developers adding transcription to applications and people who need offline speech-to-text.

BentoML is a Python framework for developers who want to turn AI models into services on their own hardware or servers. It handles model serving and application logic for LLMs, image generation, audio, and embeddings.

PocketPal AI runs language models on phones and tablets for people who want an assistant they can use without sending conversations to a server. Core chat works offline after a model download and doesn't require an account.

NullHub is a self-hosted management console for people running NullClaw and its supporting tools. It brings agent instances, workflow runs and task queues into one browser dashboard.

Silero VAD is a pretrained voice activity detection model for developers who need to find speech segments in audio. It returns speech timestamps for use in audio processing pipelines.

LLM Farm is an app for on-device AI conversations and comparing language models on Apple hardware. It suits people who want offline chat or need to assess which model fits a project.

LM Studio runs downloaded language models for conversations and tasks on your computer. Its Bionic agent helps people create documents, write software and carry out work using local or frontier open models.

TabbyAPI is a self-hosted LLM API server for people who want to run ExLlamaV3 models on their own hardware. It’s built for personal projects and small groups; its maintainers don’t recommend it for production servers.

audio.cpp runs audio models locally for developers building speech tools and people generating or processing audio on their own hardware. It brings multiple audio tasks into one C++ runtime built on ggml.

GPyTorch is an open-source Python library for researchers and developers building Gaussian process models in PyTorch. It runs model inference on your own hardware and can use GPUs to accelerate computation. Its focus is on making Gaussian processes practical at larger scales while leaving room to define custom models. The library uses the MIT license.

Mamba is a Python library for researchers and developers building sequence models, including language models. Its selective state space architecture processes sequences in linear time and is designed for efficient GPU execution.

HanLP is a multilingual natural language processing toolkit for developers and researchers who need structured text analysis. Its native Python API runs models on your own hardware. The separate REST API sends text to a hosted server.

Priv AI is an on-device assistant for people who want to chat with AI and use it to examine personal health and financial information. It works offline after a model download and doesn't require an app account.

Lemonade is a local AI server for people who want to chat with models on their own hardware or use them in apps and agents. It includes a built-in interface for direct use and APIs for connecting other software.

unbagrnd removes image backgrounds locally for people who need cutouts without uploading pictures to a cloud service. It includes an app for individual images and folders, plus a self-hosted API for developers.

Sokuji translates speech for people taking part in meetings across languages. Other participants hear your translated voice as ordinary call audio, so they don't need to install anything.

Zeraix is a local AI desktop for people who want models to work with their files and carry out multi-step tasks. It combines chat, project tools, and agents in one workspace.

HQQ is a Python library for developers compressing language and vision models to fit the memory available on their own hardware. Its Half-Quadratic Quantization method doesn't require a calibration dataset.

VoxKey is a free, open-source dictation app for people who want to speak into Mac text fields without sending audio to a cloud transcription service. It requires Apple Silicon and macOS 26 or later. Speech recognition runs locally and works offline after you download a model.

Argmax OSS Swift is a speech AI SDK for developers building iOS and macOS apps that process audio on the user's device. Its separate libraries cover transcription, identifying speakers and generating spoken audio.

MONAI is a framework for researchers and developers building AI models for medical images. Its Core library provides medical-specific tools for training and evaluating models within existing PyTorch workflows.

WeNet is a speech recognition toolkit for developers adding transcription to applications or services. It covers model training and production deployment, so teams can adapt a recognizer to their audio or start with a pretrained model.

ChatterUI is an Android chat app for people who want to use a local LLM on their phone or connect to a remote model. It supports assistant and character conversations, with control over reply generation and chat formatting.