Tools to Run AI Models Locally

Infinity Embeddings is a self-hosted model server for developers building semantic search and retrieval-augmented generation applications. It runs retrieval models on your own hardware and can serve several models together.

llamafile packages a language model and its runtime in a single executable for people who want to run models locally or share a ready-to-run model with others. Recipients don't need to install a separate runtime.

Parakeety is a paid Mac dictation app for people who want to speak into the app they're already using without sending their voice to a cloud service. It runs on Apple Silicon Macs with macOS 14 or later and places the transcript at the cursor. Intel Macs aren't supported.

bitsandbytes is a Python library for developers who need to run or fine-tune large language models on their own hardware with less memory. It works within PyTorch workflows.

RKLLM is a toolkit for developers building local AI applications on Rockchip boards. It runs language and multimodal models on the chip's neural processing unit (NPU).

faster-whisper is an open source Python library for developers adding speech transcription to their software. It uses CTranslate2 to run OpenAI's Whisper models, with faster transcription and lower memory use than the original implementation in the project's benchmark comparisons.

Keras is an open-source Python deep learning framework for developers and researchers who want to build and train models without tying their code to one backend. It runs locally on Linux and macOS, with WSL2 recommended for Windows. Its Apache 2.0 license permits commercial use and modification.

Model2Vec is a Python package for developers who need text embeddings for search or classification on their own hardware. It turns sentence transformers into smaller static embedding models, trading a small drop in performance for lower model size and faster CPU inference.

rerankers is a Python library for developers building search and retrieval systems. It ranks candidate documents against a query and lets you compare reranking approaches through a shared interface.

Airux Pocket AI runs AI conversations on Android phones for people who want offline chat and photo analysis without cloud inference. You don't need an account, and downloaded models run directly on your phone.

ExLlamaV2 and V3 are local LLM inference libraries for developers building AI applications or hosting models on consumer GPUs. V2 is archived for now and isn't currently maintained; development continues in V3.

Roboflow Inference runs computer vision models on images and camera feeds for teams building inspection, monitoring and image analysis systems. You can host it on your own computer, server or edge device.

Stanza is a Python NLP library for developers and researchers who need to analyze text in multiple human languages on their own hardware. It turns raw text into linguistic annotations, such as word forms, grammatical relationships, and named entities. It's open source under Apache 2.0.

Unsloth is a desktop app for people who want to chat with models, train them on their own data, and create media on their own hardware. Its no-code interface brings these tasks into one workspace.

DeepGEMM is an open-source CUDA kernel library for developers building LLM training and inference systems on their own NVIDIA GPUs. It supplies matrix operations and specialized computations for mixture-of-experts (MoE) models.

Easy Diffusion generates and edits images on your computer through a browser interface. It's for people who want to use Stable Diffusion without assembling its software components themselves.

NeuTTS is a collection of on-device text-to-speech models for developers building voice assistants and speech-enabled apps. It generates audio on your own hardware rather than sending speech generation to a cloud API.

Atomic Chat is a local LLM app for people who want to chat with models on their own device or use them with coding assistants and AI agents. It began as a fork of Jan by Menlo Research.

LLMKube manages self-hosted LLM inference for teams and homelab owners who need several machines to serve models together. It brings Linux GPU servers and Apple Silicon Macs into the same Kubernetes-managed fleet.

SLEAP helps behavioral researchers measure animal movement in recorded video. It trains models to estimate poses for single animals or groups, with human review during dataset creation and prediction checking.

MLX VLM is a Python package for developers and researchers who want to run multimodal models on their own Mac. It supports local inference and fine-tuning through MLX.

MLX Audio is a Python library for developers building speech and audio applications on Apple Silicon Macs. It runs models locally using Apple's MLX framework, with support for generated speech, transcription and audio processing.

Torch-TensorRT compiles PyTorch models with TensorRT to reduce inference latency on NVIDIA hardware. It's for developers running models on their own machines or servers who want faster execution while keeping their PyTorch workflow. The project is open source under the BSD 3-Clause license.

dstack is a self-hosted AI compute orchestration tool for teams running development, training and inference across cloud providers and their own servers. It gives those workloads a shared interface across Kubernetes, VMs and bare-metal clusters.

OfflineLLM is an on-device AI chat app for Android users who want to run models locally without sending conversations to a cloud service. It works fully offline and has no network permissions.

Jan is a desktop AI chat app for people who want to run models on their own computer while retaining the option to use cloud providers. Local model use works offline; choosing a hosted model sends requests to that provider.

Willow is a voice assistant for people who want to control home automation through hardware and speech processing they own. It provides an alternative to Amazon Echo or Google Home on Espressif's ESP32-S3-BOX hardware family.

Slotstream runs Qwen3.8-Flash-Next on Apple Silicon Macs with too little RAM to hold the whole model. It's aimed at Macs with 16 to 64 GB of memory, for people who want local chat, answers about pictures or a backend for coding assistants.

ChuanhuChatGPT is a self-hosted web interface for people who want to chat with local LLMs and cloud models in one application. It also supports document questions and automated agent tasks.

Nativ is a macOS app for people who want AI chat and coding tools to use models running on their own Mac. It combines a chat workspace, model library, and local inference server.

LiteRT is Google's framework for developers adding machine learning and generative AI to apps that run on users' devices. It succeeds TensorFlow Lite and supports local inference without a server round trip.

KubeAI is a Kubernetes operator for teams serving AI models on their own infrastructure or cloud clusters. It manages model servers and adjusts capacity to demand, including starting models from zero running replicas.

TorchMD-NET is an open-source Python library for researchers who train neural network potentials for molecular dynamics. It exposes its models as PyTorch modules and connects them to GPU-accelerated simulation software, including ACEMD, OpenMM and TorchMD. It's available under the MIT license.

Ollama runs open models for people who want local AI applications, model chat or a backend for coding assistants. It provides a command-line interface and an API, with a choice between inference on your hardware and its cloud service.