Tools to Run AI Models Locally
The software that loads a model and runs it on your machine: one-step apps like Ollama for a laptop, and servers like vLLM for a GPU box.
Subcategories
An open-source local AI chatbot for Windows, macOS and Linux. Run models without a GPU or cloud API, and chat privately with your documents.
77.4KUpdated 1 year agoMIT
macOS · Windows · Linux · Docker#GGUF#llama.cpp backend#OpenAI-compatible API
GPT4All is a local AI chatbot for people who want to run language models on their own desktop or laptop and keep conversations on their machine. Its LocalDocs feature lets you ask questions about your own documents without sending them to a cloud service. It suits developers, teams and individuals who want control over their models and data.
Chat With Your DocumentsDesktop Chat AppsInference Libraries and BindingsInference Servers and ClustersLocal Model Runners+0
A Python package for local LLM inference and fine-tuning on Apple Silicon, built on MLX with Hugging Face model support and an MIT license.
7.2KUpdated 1 day agoMIT
macOS#Batch processing#Distributed execution#Hugging Face integration
MLX LM is an open-source Python package for generating text and fine-tuning language models locally on Apple Silicon Macs. Built on MLX, it suits developers and researchers who want to work with models through Python or a terminal, including adapting models to their own tasks. The package uses the MIT license.
Inference Libraries and BindingsLLM Fine-TuningLocal Model RunnersQuantization and Model ConversionTerminal Chat Clients+0
Self-hosted AI inference operator for Kubernetes with vLLM, Ollama and an OpenAI-compatible API. Runs on CPUs, GPUs or TPUs under Apache 2.0.
1.3KUpdated 1 day agoApache-2.0
Web#LoRA#Multimodal input#Ollama integration
KubeAI is an open source Kubernetes operator for teams serving AI models on their own infrastructure or cloud clusters. It manages model servers and scales them with demand, including starting from zero running replicas. It uses the Apache 2.0 license and can run on CPUs, GPUs or TPUs, including in a local Kubernetes cluster.
Embedding ServersInference Servers and ClustersKubernetes and Cloud-Native AILLM Gateways and RoutersSpeech-to-Text and Transcription+0
Self-hosted text-to-speech server runs Chatterbox models on CPU or GPU, with voice cloning, audiobook generation and an OpenAI-compatible API.
1.5KUpdated 4 months agoMIT
macOS · Windows · Linux · Docker · Web#Hugging Face integration#Multilingual#OpenAI-compatible API
Chatterbox TTS Server runs Resemble AI's speech models on your own computer or server, with a browser interface and an OpenAI-compatible API. It's for people producing narration and audiobooks, or developers adding speech to voice agents and other apps. The project is open source under the MIT license.
Inference Servers and ClustersText-to-Speech and AudiobooksVoice Cloning and Voice Changers+0
Self-hosted speech-to-text API that runs in Docker on CPU or CUDA GPUs, with Whisper, Faster Whisper and WhisperX. Open source under MIT.
3.3KUpdated 2 months agoMIT
Docker · Web#Multilingual#Speaker diarization#Voice activity detection
Whisper ASR Webservice turns Whisper speech recognition into a self-hosted API for developers adding transcription to their apps or services. It runs in Docker on your own machine or server, with CPU processing or CUDA GPU acceleration. The Python project is open source under the MIT license.
Inference Servers and ClustersSpeech-to-Text and TranscriptionSubtitles and DubbingTranslation+0
Local LLM library runs GGUF models through llama.cpp in Node.js, Bun and Electron. MIT licensed, with GPU support and JSON schema enforcement.
2.2KUpdated 3 days agoMIT
macOS · Windows · Linux#Batch processing#GGUF#Guardrails
node-llama-cpp is an open source library for developers adding local LLM inference to JavaScript and TypeScript applications. It connects Node.js, Bun and Electron to llama.cpp, running GGUF models on your own machine. Its MIT license allows use in commercial projects.
Inference Libraries and BindingsLLM Libraries and SDKsTerminal Chat Clients+0
An MIT-licensed translation app for Windows, macOS and Linux that runs Marian and Bergamot models locally and connects to Chrome and Firefox.
631Updated 2 years agoMIT
macOS · Windows · Linux · Web · Browser Extension#Batch processing#Multilingual#Quantization
TranslateLocally is an open source machine translation app for Windows, macOS and Linux. It's for people who want to translate text on their own device, including those handling material they don't want to send to a remote translation service. The desktop app has an MIT license and uses Marian and Bergamot translation models.
On-Device and In-Browser AITranslation+0
A local AI app launcher for Windows, macOS, and Linux that helps you find and run community projects. It's open source under the MIT license.
8.2KUpdated 4 weeks agoMIT
macOS · Windows · Linux#Code execution
Pinokio is a local launcher for people who want to try AI apps on their own computers. It pairs a catalog of community projects with scripts that handle the commands each app needs. Pinokio runs on Windows, macOS, and Linux, and its code is open source under the MIT license.
One-Click AI Installers+0
Open-source speaker diarization toolkit with local PyTorch models, CUDA GPU support, and an optional hosted service that processes audio on pyannoteAI servers.
10.6KUpdated 3 months agoMIT
#Hugging Face integration#Speaker diarization#Voice activity detection
pyannote.audio is a Python toolkit that separates an audio recording into timed segments labeled by speaker. It's for developers and researchers who need to track who spoke when, with pretrained models that run on their own hardware. The toolkit is open source under the MIT license.
Fine-Tuning and TrainingInference Libraries and BindingsVoice, Speech and Music+0
Open-source text-to-speech toolkit for local speech generation, voice cloning and model training on Linux, macOS and Windows, licensed under MPL-2.0.
2.3KUpdated 4 months agoMPL-2.0
macOS · Windows · Linux · Docker#Multilingual#Streaming inference#Voice cloning
Coqui TTS (idiap fork) is a local text-to-speech library for developers and speech researchers who want pretrained voices or tools to train their own models. It builds on coqui-ai/TTS, continuing the original unmaintained project. The Python toolkit is open source under the Mozilla Public License 2.0 (MPL-2.0).
Inference Libraries and BindingsText-to-Speech and AudiobooksTraining FrameworksVoice Cloning and Voice Changers+0
A self-hosted inference framework that coordinates NVIDIA GPU clusters with vLLM, SGLang or TensorRT-LLM and exposes an OpenAI-compatible API.
8.2KUpdated 20 hours ago
#Distributed execution#Multimodal input#OpenAI-compatible API
NVIDIA Dynamo is a self-hosted inference framework for teams serving models across multiple GPUs or server nodes. It coordinates SGLang, TensorRT-LLM and vLLM, adding cluster-level scheduling and request routing above those engines. Its focus is large deployments where GPU capacity, response latency and repeated computation affect serving costs.
GPU Management and MonitoringInference Servers and ClustersKubernetes and Cloud-Native AILLM Gateways and Routers+0
Open-source Python toolkit for semantic search and RAG, with BGE embedding models, multilingual rerankers, evaluation and fine-tuning under MIT.
12.2KUpdated 1 month agoMIT
#Multilingual#Semantic search
BGE Embeddings is a family of embedding models and rerankers for developers building semantic search and retrieval-augmented generation (RAG). Developed by the Beijing Academy of Artificial Intelligence, it includes the MIT-licensed Python toolkit FlagEmbedding for running inference, evaluating retrieval and fine-tuning models.
Embedding and Reranker ModelsFine-Tuning and TrainingInference Libraries and BindingsRAG and Vector Search+0
Open-source AI inference toolkit for local or self-hosted deployment on Linux, Windows and macOS, with CPU, Intel GPU and NPU support.
10.9KUpdated 24 hours agoApache-2.0
macOS · Windows · Linux#Hugging Face integration#Multimodal input#ONNX
OpenVINO is an Apache 2.0 licensed toolkit for developers who want to run AI models locally or serve them on their own infrastructure. It converts and optimizes models for inference, with support for x86 and ARM CPUs, Intel integrated and discrete GPUs, and Intel NPUs. Its runtime works on Linux, Windows and macOS.
Edge and Embedded AIInference Libraries and BindingsInference Servers and ClustersOn-Device and In-Browser AIQuantization and Model Conversion+0
A Python library for local embeddings and reranking, using ONNX Runtime with CPU or GPU support. Open source under Apache 2.0.
3.2KUpdated 24 hours agoApache-2.0
#Batch processing#Multilingual#ONNX
FastEmbed is a Python library that generates embeddings on your own hardware for semantic search and retrieval-augmented generation (RAG). It's for developers who need to turn text into searchable vectors without relying on a cloud embedding API. It can run on a CPU or use GPU acceleration, and its Apache 2.0 license makes it open source.
Inference Libraries and BindingsRAG and Vector Search+0
A PyTorch library for training and inference on local machines or clusters, with CPU, GPU and TPU support. Open source under Apache 2.0.
9.9KUpdated 1 day agoApache-2.0
#Distributed execution
Accelerate is a Python library for developers and researchers who write their own PyTorch training loops and want to use the same code on a local machine or a distributed cluster. It handles the hardware-specific work while leaving the training logic under your control.
GPUs, Clusters and EdgeInference Libraries and BindingsTraining Frameworks+0
Open-source computer vision library for local detection, segmentation and tracking, with AGPL-3.0 licensing and exports to ONNX, TensorRT and CoreML.
62.1KUpdated 24 hours agoAGPL-3.0
#ONNX
Ultralytics YOLO is an open-source Python computer vision library for developers building applications that analyze images and video on their own hardware. It supports local and edge deployment, including NVIDIA Jetson, Raspberry Pi and mobile phones. A separate hosted platform provides browser-based annotation, cloud GPU training and managed prediction endpoints.
Computer Vision ModelsEdge and Embedded AIInference Libraries and BindingsTraining Frameworks+0
Self-hosted AI model serving platform for Kubernetes. Serve LLMs and predictive models with vLLM, Hugging Face support and an OpenAI-compatible API.
6KUpdated 23 hours agoApache-2.0
#Hugging Face integration#ONNX#OpenAI-compatible API
KServe is an open source platform for teams serving LLMs and predictive machine learning models on their own Kubernetes infrastructure. It puts both kinds of workloads under a common serving API, so teams can manage different model frameworks through the same platform. It uses the Apache 2.0 license.
Inference Servers and ClustersKubernetes and Cloud-Native AI+0
A local LLM engine that runs models in the browser with WebGPU acceleration, OpenAI API compatibility, and an Apache 2.0 license.
19.2KUpdated 2 weeks agoApache-2.0
Web · Browser Extension#OpenAI-compatible API#Streaming inference#Structured output
WebLLM runs language models directly in a user's browser, using WebGPU for GPU acceleration. It's an open-source engine for developers building web-based AI assistants and Chrome extensions that process prompts on the user's device rather than an inference server. The project uses the Apache 2.0 license.
Inference Libraries and BindingsLLM Libraries and SDKsOn-Device and In-Browser AI+0
Python library for running and training embedding and reranker models locally, with Apache 2.0 licensing and pretrained models on Hugging Face.
19.1KUpdated 1 week agoApache-2.0
#Hugging Face integration#Multilingual#Multimodal input
Sentence Transformers is an open-source Python library for developers building semantic search and document retrieval on their own hardware. It runs embedding and reranker models locally, turning content into numerical representations for similarity comparisons and scoring results against a query. The library uses the Apache 2.0 license.
Inference Libraries and BindingsRAG and Vector SearchTraining Frameworks+0
Open-source Rust ML framework for running models locally on CPUs, NVIDIA GPUs or in browsers, with Apache 2.0 licensing and quantized LLM support.
21.1KUpdated 2 days agoApache-2.0
macOS · Web#GGUF#Hugging Face integration#Multilingual
Candle is a Rust machine learning framework for developers who want to embed local AI in applications or deploy models on their own servers. It produces lightweight binaries that don't need Python in production, making it a candidate for serverless inference where a large runtime can slow startup. Its API uses tensor operations familiar to PyTorch developers.
Inference Libraries and BindingsOn-Device and In-Browser AITraining Frameworks+0
Self-hosted embedding and reranking API with MIT licensing, Hugging Face models, and CPU, NVIDIA, AMD and Apple MPS support.
2.9KUpdated 6 months agoMIT
macOS · Docker#Batch processing#Hugging Face integration#Multimodal input
Infinity Embeddings is a self-hosted server for developers building semantic search and retrieval-augmented generation applications. It runs embedding and reranking models on your own hardware, with support for image and audio search alongside text. It's open source under MIT.
Embedding ServersInference Servers and Clusters+0
Chat with local or hosted models, compare answers side by side and ask questions over your documents in Msty Studio's AI workspace.
msty.appAI Notes and Knowledge Bases
Web#llama.cpp backend#MLX#Ollama integration
Msty Studio is a private AI workspace for chatting with local or hosted models and working with your own documents. It offers a desktop app alongside web and team deployment options. Local model support includes MLX, llama.cpp and Ollama.
AI Notes and Knowledge BasesAutomation and No-Code AIDesktop Chat AppsLLM Gateways and RoutersRun Models Locally+0
Run AI models through Docker Desktop, Docker Engine or a standalone binary, with local inference and OpenAI and Ollama compatible APIs.
655Updated 2 days agoApache-2.0
macOS · Windows · Linux#GGUF#Hugging Face integration#llama.cpp backend
Docker Model Runner lets developers run and serve AI models on their own computer or server using Docker Desktop, Docker Engine or the standalone dmr binary. It pulls models from Docker Hub, OCI registries, and Hugging Face, then stores them locally. Inference runs locally too.
Inference Servers and ClustersLocal Model Runners+0
An open-source Python library for local speech transcription with Whisper models. It runs on CPUs or NVIDIA GPUs and uses CTranslate2.
25.6KUpdated 3 hours agoMIT
#Batch processing#Hugging Face integration#Quantization
faster-whisper is a Python library for people building local speech transcription into their own software. It runs OpenAI's Whisper models through CTranslate2, with faster processing and lower memory use than the original Whisper implementation in the project's comparisons. It runs on a CPU. NVIDIA GPUs are supported too, and the code is open source under the MIT license.
Inference Libraries and BindingsSpeech-to-Text and Transcription+0
Local AI app for Android and iOS. Run Gemma 4 on-device, ask questions about photos, transcribe audio and compare model performance.
24.8KUpdated 21 hours agoApache-2.0
iOS · Android#Agent Skills#Hugging Face integration#Multilingual
Google AI Edge Gallery is an open-source app for people who want to try generative AI on their own phone. It runs model inference locally, so offline chat, image analysis and audio tasks don't send your inputs to a server. It supports Android and iOS and uses the Apache 2.0 license.
Mobile Chat AppsOn-Device and In-Browser AISpeech-to-Text and TranscriptionTranslation+0
Chat with models on your iPhone or iPad through Liquid LEAP, or connect to your own Ollama or LM Studio server. Cloud providers are optional.
liquid.aiMobile Chat Apps
macOS · iOS
Liquid Apollo is an AI chat app for iPhone and iPad. Liquid LEAP runs smaller language models directly on your device, so local conversations do not depend on a cloud service. The app also runs on compatible Apple Silicon Macs.
Mobile Chat AppsOn-Device and In-Browser AI+0
A local AI desktop workspace for Apple Silicon Macs, Windows and Linux x64, with GGUF and MLX models, document search and optional cloud providers.
46Updated 5 days agoAGPL-3.0
macOS · Windows · Linux · Browser Extension#Code execution#GGUF#llama.cpp backend
Vyact is an open source desktop workspace for people who want to use a local LLM with their documents, email and code. It runs on Apple Silicon Macs, Windows and Linux x64 under the AGPL-3.0 license. Intel Macs aren't supported.
AI Notes and Knowledge BasesBrowser AI SidebarsChat With Your DocumentsDesktop Chat AppsLocal Model RunnersWriting and Email Assistants+0
An AI dictation app for iPhone and iPad that uses Gemma models locally. It supports offline transcription and optional cloud text features.
apps.apple.comDictation and Voice Typing
iOS#Multilingual#Works offline
Google AI Edge Eloquent is an AI dictation app for iPhone and iPad that turns spoken thoughts into edited text using on-device Gemma models. It's for people who prefer speaking to typing but don't want every hesitation or self-correction carried into their notes and drafts.
Dictation and Voice TypingOn-Device and In-Browser AISpeech-to-Text and Transcription+0
