Tools to Run AI Models Locally

vLLM Production Stack is an open source LLM serving stack for teams running models on Kubernetes GPU clusters. It lets applications move from a single vLLM instance to a distributed deployment without changing application code.

picoLLM is an on-device LLM inference SDK for developers adding text generation to their apps. Inference works offline, but a Picovoice account and AccessKey are required, and key validation needs an internet connection.

LM Format Enforcer is an open-source Python library for developers who need a local LLM to return structured data their applications can parse. It constrains text during generation, so a prompt's formatting request doesn't have to carry the whole burden. The library uses the MIT license.

MusicGen is Meta AI's model for generating music from written descriptions. It's part of AudioCraft, a PyTorch library for developers and audio researchers building music generation tools in their own computing environment.

Izwi is a local voice AI app and runtime for people and teams processing speech on their own hardware or inside infrastructure they control. It brings audio tools and voice conversations into a desktop app, browser interface, CLI and inference server.

FlashRank is a Python library that runs reranking models locally on CPU, without Torch or Transformers. It's for developers building search systems or retrieval-augmented generation (RAG) apps who want to reorder retrieved passages by relevance before passing them to an LLM.

Braina combines AI chat, dictation and computer control for people who want to work through spoken or typed commands. It can run language models on your own computer and help with writing, document research and repetitive desktop tasks.

Whisper ASR Webservice gives developers a speech-to-text API for their own applications and services. It processes audio on your machine or server and supports multilingual transcription, speech translation and language identification.

Ray Serve is a Python library for developers building inference APIs around machine learning models. It's useful when a response needs several models or application steps, rather than a single model call.

Meetily records calls and turns them into live transcripts and meeting summaries. It's for people who want to keep meeting recordings on their own computer without adding a recording bot to the call.

EmbedAnything is a local embedding pipeline for developers building semantic search and retrieval-augmented generation (RAG) applications. It turns documents and media into vectors for indexing in a vector database.

ggml is a library for developers building machine learning runtimes and local AI applications. It provides tensor computation across different kinds of hardware; GGUF is the model file format covered by the project.

Klee is a native macOS AI chat app and agent for people who want an assistant to work with files on their Mac. It runs models locally through MLX, and downloaded models can chat offline without an account or API key.

MLX LM is a Python package for developers and researchers who want to generate text and adapt language models on Apple Silicon Macs. It uses MLX for local inference and training, with access through Python or a terminal.

taOS is a self-hosted AI agent platform for people who want to run assistants on their own computers and keep their conversations, files, and memory there. It brings agent management and local model access into a browser-based desktop.

Stable Audio Tools is a Python toolkit for developers and audio researchers who want to generate audio or train models on their own recordings. It supports work with pretrained models as well as training from scratch.

1Panel is a Linux server management panel for people hosting AI software alongside websites and other applications. Its browser interface brings model and agent management together with everyday server administration.

Mooncake provides storage and data transfer components for teams running distributed LLM inference. Its Transfer Engine and Mooncake Store support shared caching and deployments that separate prompt processing from token generation.

Ultralytics YOLO is a Python computer vision library for developers building applications that interpret images and video. Models can run on hardware you control for tasks such as visual inspection, inventory monitoring and robotics perception.

KittenTTS is a Python text-to-speech library for developers adding local speech generation to apps or producing spoken audio from text. It supports voice cloning without model training and doesn't require a GPU.

FlashMLA is an attention library for developers building DeepSeek-V4.1 inference systems on their own hardware. It requires NVIDIA SM100 or SM103 GPUs, or Huawei Ascend 950 NPUs.

AI Runner is a desktop app for people who want an AI companion and an art workspace on their own hardware. Its local chat, voice and image tools work offline without an API key or subscription.

Paddler lets product and infrastructure teams serve AI models on their own servers and spread requests across multiple machines. It's built for applications that need text generation or embeddings while keeping inference within infrastructure the team controls.

MLC LLM is a compiler and deployment engine for developers running language models on their own hardware or building AI into apps. Its shared MLCEngine supports deployment across desktop computers, mobile devices and browsers.

TranslateLocally translates text on your own computer for people who want local machine translation and teams working with their own translation models. It uses Marian and Bergamot, with a desktop interface for everyday text and command-line access for batch work.

sherpa-onnx is a toolkit for developers building speech features that work offline on their own devices. It covers transcription, speech generation and audio analysis, with processing kept local.

PaddleX is a low-code toolkit for developers building AI applications with PaddlePaddle models. It brings pretrained pipelines and model training into one tool for applications that process documents, images or time-series data.

FUTO Voice Input adds private dictation to compatible Android keyboards and apps. It's for people who want speech recognition on their phone while keeping the keyboard they already use. Gboard and Samsung Keyboard don't support it as an alternative voice input provider.

llm-d is a distributed LLM inference stack for infrastructure teams serving models across a Kubernetes cluster. It coordinates model servers such as vLLM and SGLang to handle production traffic across multiple machines.

Ente Photos is a Google Photos alternative for people who want to back up and share their library without letting the storage provider read its contents. You can self-host its server or use Ente's hosted service.

kvpress is a Python library for developers and researchers reducing the memory needed for long-context LLM inference. It compresses the key-value cache that grows as a model processes longer inputs.

Haplo AI runs language models on iOS and macOS for people who want to ask questions, write code or examine photos without sending that material to a hosted assistant. After downloading a model, chat works offline without an account.

Hovor is a dictation app for people who want to speak text into their Mac apps and choose between local speech recognition and a cloud service. On supported Apple Silicon Macs, it runs downloaded Parakeet or Whisper models on the device, so on-device transcription happens on your Mac.

DL4MicEverywhere lets microscopy researchers train and use deep learning models on their own computers through graphical Jupyter notebooks. It builds on ZeroCostDL4Mic, bringing its bioimage analysis workflows to local machines and remote computing infrastructure. You don't need coding skills to use the notebook interfaces.