Tools to Run AI Models Locally

Seldon Core is an AI model serving framework for teams running machine learning models and LLM applications on Kubernetes. It manages individual models and applications with multiple processing stages.

OpenVINO Model Server is a self-hosted inference server for developers and teams serving AI models on their own machines or servers. It uses OpenVINO to run workloads on x86 CPUs, including Xeon, with acceleration for Intel integrated and discrete GPUs and NPUs. It's open source under Apache 2.0.

Maid is an Android AI chat app for people who want to chat with models on their phone and access remote models from the same interface. Local conversations work without an internet connection.

FastChat is a self-hosted LLM platform for developers and researchers who want to serve chat models on their own hardware and compare their responses. It powers Chatbot Arena and includes a locally hosted side-by-side comparison interface. The Python code is open source under Apache 2.0; model licenses apply separately, including Llama's license for Vicuna.

FunASR is a speech-to-text toolkit for developers building transcription services and voice applications. It supports offline recordings and live speech on local hardware or self-hosted servers, with capabilities that depend on the model and runtime.

Vosk is an offline speech-to-text toolkit for developers building voice interfaces or transcription software. It processes speech on the device or server running it, without sending audio to a cloud recognition service.

NativeMind is a local ChatGPT alternative for people who want AI help with reading and writing inside their browser. It processes prompts and page content on your device using local models.

Layla is an AI assistant for people who want private chat, a personal companion or character roleplay on their phone. It runs local models on Android and iOS, so you can keep using it offline after downloading a model.

Colossal-AI is a Python framework for developers and researchers training or serving large AI models on their own GPUs. It provides tools to spread workloads across GPUs and work within available memory.

PowerInfer is a local LLM inference engine for developers and researchers running large models on a PC with a consumer GPU. Its performance gains depend on sparse models, which use only part of their network for each input.

LocalAI is a self-hosted AI runtime for developers and teams that want to run models on their own hardware. It brings language, speech, vision and media generation into one server that existing AI applications can connect to.

Distributed Llama runs language models across your own networked computers. It's for people who want to share inference work and model memory across several devices instead of fitting the whole workload on one machine.

SAHI is a Python computer vision library for developers detecting small objects in large images, including aerial imagery and surveillance footage. It divides images into smaller regions for detection, helping models find objects that occupy few pixels.

KServe serves LLMs and predictive machine learning models on your own Kubernetes infrastructure. It's for teams that need to put models from different frameworks into production through one serving platform.

gollama is a terminal app for people managing a local LLM collection in Ollama. Its keyboard-driven interface helps you compare downloaded models and clear out ones you don't need.

tiny-cuda-nn is a neural network framework for developers and researchers building small models, NeRF renderers and neural graphics applications. It handles training and inference on local NVIDIA GPUs.

LLM X is a browser chat interface for people running models on their own hardware. It brings multiple model servers into one place and lets you compare their responses within a conversation.

LiteRT-LM is Google's open-source framework for developers building apps that run language models on the user's device. It uses LiteRT for local inference across mobile devices, desktops and browsers, with GPU and NPU acceleration. The project uses the Apache 2.0 license.

PyOD is a Python library for detecting unusual records and behavior in data on your own machine or server. A shared fit/predict interface lets developers and data scientists compare classical and deep learning anomaly detectors without changing the surrounding application for each algorithm. Its code uses the BSD 2-Clause license.

Olares is a self-hosted personal cloud OS for individuals and small teams that want AI agents, models and apps on hardware they own. It brings local AI and everyday file storage into a shared workspace you can access through a browser.

Transcribro lets Android users dictate text with speech recognition running on their device. It supports English only and can also provide transcription to compatible apps.

Stability Matrix is a desktop app for people who want to use several Stable Diffusion interfaces with one model library. It manages local Web UI packages and includes its own interface for generating images.

Offeline is a local AI chat app for people who want to talk to AI on their phone or tablet without sending conversations to a hosted chatbot. It processes chats on the device.

Moonshine is an on-device AI toolkit for developers building voice agents and spoken interfaces. Voice processing runs on the device without an account or API keys.

BGE Embeddings is a family of models for developers who need to find relevant passages for search or retrieval-augmented generation (RAG). It includes embedding models for finding candidate text and rerankers for assessing how closely those results match a query.

Off Grid AI is a local AI app for people who want chat, document analysis and media tools on their own phone or computer. Downloaded models work offline, and the core app doesn't require an account.

Moshi is a voice AI model and dialogue framework for researchers and developers building spoken applications. It processes speech directly and can listen while it speaks, including information such as emotion and non-verbal cues.

GPTQModel is a Python toolkit for developers compressing LLMs and running them on their own hardware or servers. It combines model preparation, quality evaluation and inference so you can assess compression choices within one workflow.

Skales is an AI agent for people who want help with files, browsing, email and coding on their own devices. It can keep working on a goal after you close the chat and resume unfinished goals after a restart.

EasyOCR is a Python library for developers adding text recognition to an application. It reads both text in photographs and dense document text.

SmolVLM is a compact vision language model for developers building applications that interpret images and answer questions about them. It can run on-device, including on laptops and smaller local setups.

DeepCell is an open-source Python library for researchers analyzing individual cells in biological images. It runs locally and uses TensorFlow to identify whole cells or nuclei and track cells through time-lapse movies. Its Apache 2.0 license allows users to adapt the library for their own research.

Bergamot Translator is a local AI translation library for developers building translation into browser or native applications. People who want to translate web content can use it through the separate TranslateLocally extension for Firefox.

MFLUX is a Python toolkit for creators and developers who want to generate and edit images on an Apple Silicon Mac. It provides a command-line interface for working with models locally.