Tools to Run AI Models Locally
The software that loads a model and runs it on your machine: one-step apps like Ollama for a laptop, and servers like vLLM for a GPU box.
Subcategories
A self-hosted speech API for transcription, translation, and speech generation, compatible with OpenAI clients and deployable on CPU or GPU.
3.8KUpdated 7 months agoMIT
Docker#OpenAI-compatible API#Streaming inference
Speaches is a self-hosted speech server for developers building transcription services and voice applications. Its OpenAI-compatible API lets existing clients and SDKs work with speech models running on hardware you control.
Inference Servers and ClustersSpeech-to-Text and TranscriptionText-to-Speech and AudiobooksTranslation+0
A self-hosted LLM serving toolkit with text and vision models, quantization, and OpenAI-compatible APIs, licensed under Apache 2.0.
8.1KUpdated 1 day agoApache-2.0
#Batch processing#Distributed execution#Hugging Face integration
LMDeploy is an open-source toolkit for developers who want to run language and vision-language models on their own GPU hardware. It supports offline batch processing and model servers for applications.
Inference Libraries and BindingsInference Servers and ClustersQuantization and Model Conversion+0
A local AI playground lets you interact with AI without relying on cloud processing.
liquid.aiMobile Chat Apps
macOS · iOS
Liquid Apollo is a local AI playground for people who want to interact with AI without relying on cloud processing. It focuses on low-latency interaction.
Mobile Chat AppsOn-Device and In-Browser AI+0
A self-hosted ML model serving framework for Linux and macOS, with Python model code, dynamic batching and CPU/GPU pipelines. Open source under Apache 2.0.
904Updated 2 weeks agoApache-2.0
macOS · Linux · Docker#Batch processing#OpenAI-compatible API#Reranking
Mosec is a self-hosted model serving framework for developers turning trained machine learning models into APIs. It runs on Linux and macOS, on your own hardware or cloud servers, and is open source under the Apache 2.0 license. Its main focus is handling concurrent requests while making better use of CPU and GPU resources.
Inference Servers and Clusters+0
An open-source image classification and recognition toolkit supports custom model training, image retrieval, and server or on-device deployment.
5.9KUpdated 1 month agoApache-2.0
Android#ONNX#Quantization#Semantic search
PaddleClas is a PaddlePaddle toolkit for developers and researchers building image classification and recognition applications on their own hardware. It covers fixed-category classifiers and image matching for large or frequently changing catalogs.
Computer Vision ModelsLoRA and Image Model TrainingOn-Device and In-Browser AIQuantization and Model ConversionRAG and Vector Search+0
A command-line Hub client that downloads models and datasets to a local cache, manages repositories, and accesses hosted inference.
4.1KUpdated 4 days agoApache-2.0
#Hugging Face integration
Hugging Face CLI is a terminal tool for developers finding models and datasets for local AI projects or publishing their own work. It connects to the Hugging Face Hub, where those resources are hosted.
Model Hubs and Downloaders+0
A local AI music model and synthesis engine for Apple Silicon, with a standalone macOS app and an AUv3 plugin for digital audio workstations.
1.9KUpdated 4 months agoApache-2.0
macOS#MLX#Streaming inference
Magenta RealTime is a local AI music model and synthesis engine for musicians playing AI instruments and developers building interactive music apps. It generates a continuous audio stream on supported Apple Silicon Macs.
Inference Libraries and BindingsMusic and Sound ModelsMusic Generation+0
A self-hosted LLM inference server with an OpenAI-compatible chat API, multi-GPU support and an Apache 2.0 license; its repository is archived.
10.9KUpdated 8 months agoApache-2.0
Linux · Docker#Batch processing#Distributed execution#Hugging Face integration
Text Generation Inference (TGI) serves LLMs on your own hardware for developers building chat apps and text generation services. The project is archived and no longer receives updates.
Inference Servers and Clusters+0
A local LLM acceleration library for Intel CPUs, GPUs and NPUs, with Ollama and llama.cpp integration; the project is archived.
8.8KUpdated 10 months agoApache-2.0
Windows · Linux · Docker#Distributed execution#GGUF#Hugging Face integration
Intel IPEX-LLM is a library for developers running or fine-tuning language models on Intel hardware. The project is archived and no longer maintained, and Intel has identified known security issues in the code.
Inference Libraries and BindingsLLM Fine-Tuning+0
LogotyperGot an idea for an AI tool? I'm Eugene, the designer and developer behind Localhosted. I'll help you build it, from the first sketch to launch.
Start your project
