Self-Hosted LLM Inference Servers

Nativ is a macOS app for people who want AI chat and coding tools to use models running on their own Mac. It combines a chat workspace, model library, and local inference server.

KubeAI is a Kubernetes operator for teams serving AI models on their own infrastructure or cloud clusters. It manages model servers and adjusts capacity to demand, including starting models from zero running replicas.

Ollama runs open models for people who want local AI applications, model chat or a backend for coding assistants. It provides a command-line interface and an API, with a choice between inference on your hardware and its cloud service.

exo is a local LLM runner for people who want to run models too large for a single machine's memory. It joins devices into a cluster and runs inference on your own hardware.

ms-swift is a Python framework for developers and researchers training and deploying language and multimodal models on their own hardware. It covers model adaptation through evaluation and serving, with support for Qwen3, DeepSeek-R1, Llama4, Mistral, Qwen3-VL and InternVL3.5.

ClearML is a self-hostable MLOps platform for researchers and engineering teams managing model development and production. It connects experiment records with datasets, training jobs and model deployment so teams can trace results back to the work that produced them.

mistral.rs is an inference engine for developers who want to run AI models on their own computer or server. It supports browser chat and inference embedded directly in Python or Rust applications.

DwarfStar is a local LLM inference engine for people who want to run large DeepSeek, GLM and Qwen models on their own computers or servers. It focuses on selected models and hardware setups, and requires the project's own GGUF files rather than accepting arbitrary GGUF downloads.

LitGPT is a Python toolkit for developers and researchers who want to train and deploy language models on their own hardware. Its directly editable model implementations suit custom applications and research experiments.

Willow Inference Server is a self-hosted speech server for developers building voice assistants, transcription apps, and accessible interfaces. It handles speech recognition and speech generation on your own hardware.

Claude Code Local lets developers use Claude Code with a model running on their own computer. It suits people who want to keep coding offline or continue working when they hit cloud usage limits.

vLLM Production Stack is an open source LLM serving stack for teams running models on Kubernetes GPU clusters. It lets applications move from a single vLLM instance to a distributed deployment without changing application code.

Whisper ASR Webservice gives developers a speech-to-text API for their own applications and services. It processes audio on your machine or server and supports multilingual transcription, speech translation and language identification.

Ray Serve is a Python library for developers building inference APIs around machine learning models. It's useful when a response needs several models or application steps, rather than a single model call.

taOS is a self-hosted AI agent platform for people who want to run assistants on their own computers and keep their conversations, files, and memory there. It brings agent management and local model access into a browser-based desktop.

Mooncake provides storage and data transfer components for teams running distributed LLM inference. Its Transfer Engine and Mooncake Store support shared caching and deployments that separate prompt processing from token generation.

Paddler lets product and infrastructure teams serve AI models on their own servers and spread requests across multiple machines. It's built for applications that need text generation or embeddings while keeping inference within infrastructure the team controls.

MLC LLM is a compiler and deployment engine for developers running language models on their own hardware or building AI into apps. Its shared MLCEngine supports deployment across desktop computers, mobile devices and browsers.

llm-d is a distributed LLM inference stack for infrastructure teams serving models across a Kubernetes cluster. It coordinates model servers such as vLLM and SGLang to handle production traffic across multiple machines.

Speaches is a self-hosted speech server for developers building transcription services and voice applications. Its OpenAI-compatible API lets existing clients and SDKs work with speech models running on hardware you control.

LMDeploy is an open-source toolkit for developers who want to run language and vision-language models on their own GPU hardware. It supports offline batch processing and model servers for applications.

Mosec is a self-hosted model serving framework for developers turning trained machine learning models into APIs. It runs on Linux and macOS, on your own hardware or cloud servers, and is open source under the Apache 2.0 license. Its main focus is handling concurrent requests while making better use of CPU and GPU resources.

Text Generation Inference (TGI) serves LLMs on your own hardware for developers building chat apps and text generation services. The project is archived and no longer receives updates.