Self-Hosted LLM Inference Servers

Cog packages machine learning models for researchers and developers serving them on their own infrastructure. It handles the container environment and inference API around a Python model.

SGLang is a self-hosted inference framework for teams serving AI models to applications. It supports production workloads on a single GPU or across distributed clusters, including AI agents and reinforcement learning rollouts.

vLLM serves language models on hardware you control. It's for developers and teams building model APIs that need to handle concurrent requests while using memory and compute efficiently.

LightLLM is a self-hosted LLM inference and serving framework for developers and teams running models on their own GPU servers. Its focus is serving requests efficiently: it schedules work around predicted memory use and handles text encoding, model inference and output decoding asynchronously to improve GPU utilization.

vLLM Metal lets developers serve language models on Apple Silicon Macs through vLLM. It's for people who want to use their Mac's hardware for local AI inference while keeping vLLM's API server and request scheduling.

Lucebox is a local AI workstation for developers and teams who want to run models on their own hardware. Its inference engine is also available separately for supported AMD and NVIDIA machines.

KoboldCpp runs language models on your own hardware for chat, creative writing and roleplay. It's a fork of llama.cpp with a bundled KoboldAI Lite browser interface.

Chatterbox TTS Server turns text into speech on your own computer or server. It's for people making narration and audiobooks, or adding speech to apps through an API.

llmserve is a terminal app for people who keep local LLM models across several folders or inference tools. It brings model discovery and server management into one interface so you can choose an existing model and an available backend.

Kaito manages LLM inference, fine-tuning and document retrieval services in Kubernetes. It's for teams that want to run models in their own cluster with less manual work sizing GPU capacity and managing deployments.

Ollama Helm Chart packages a self-hosted LLM service for developers and infrastructure teams that already use Kubernetes. It brings Ollama into a cluster so applications can access models running on the cluster's hardware.

KTransformers is a research framework for developers running or fine-tuning large language models on their own hardware. It focuses on mixture-of-experts (MoE) models, using CPU resources alongside GPUs to work within limited GPU memory.

MTPLX is a native Mac app, command-line tool and local LLM server for people running chat or coding models on Apple Silicon. It uses a model's own multi-token prediction heads to accelerate generation without a separate draft model.

OpenVINO is an AI inference toolkit for developers deploying models in their own applications. It converts and optimizes models for local hardware or servers, covering generative AI and conventional machine-learning workloads.

AIBrix provides infrastructure components for teams serving large language models on Kubernetes. It's aimed at organizations that need to manage inference capacity, request routing and models across a cluster.

Xinference serves AI models on your own computer or servers for developers and researchers building applications around models they control. It brings chat, document search and media generation into one platform.

FastLLM is an open-source inference engine for people running large language models on their own machines or self-hosted servers. Its main draw is hardware flexibility: large mixture-of-experts (MoE) models can spread their weights across GPU memory, system RAM and disk when they don't fit entirely on a graphics card.

BentoML is a Python framework for developers who want to turn AI models into services on their own hardware or servers. It handles model serving and application logic for LLMs, image generation, audio, and embeddings.

TabbyAPI is a self-hosted LLM API server for people who want to run ExLlamaV3 models on their own hardware. It’s built for personal projects and small groups; its maintainers don’t recommend it for production servers.

Lemonade is a local AI server for people who want to chat with models on their own hardware or use them in apps and agents. It includes a built-in interface for direct use and APIs for connecting other software.

Reef is self-hosted infrastructure for developers who want AI agents to learn from their interactions. It connects live requests and feedback to changes in model weights or the prompts, rules, and skills that guide an agent.

TensorSharp runs local AI models for .NET developers and people who want chat, media generation and agent tools on their own hardware. You can embed it in a C# application or use its CLI, browser interface or companion app, TensorAgent.

Text Embeddings Inference is a self-hosted server for developers building semantic search and retrieval applications. It turns text into embeddings through a REST API and can run offline once model weights are downloaded.

NVIDIA Dynamo is a self-hosted inference framework for teams serving generative AI models across multiple GPUs or server nodes. It coordinates inference engines so teams can manage capacity, response latency and cached context across a cluster.

RamaLama runs AI models for local chat or a self-hosted REST API. It's for developers who want a container workflow for trying models on their own hardware and serving them in production.

Llama Desktop runs language models on your computer for private chat and use in other apps. It's for people who want a local ChatGPT alternative and developers connecting coding agents, editors or scripts to their own models.

vllm-mlx is a local LLM inference server for developers who want to connect AI apps and coding assistants to models running on an Apple Silicon Mac. One process exposes both OpenAI-compatible and Anthropic-compatible APIs. It's open source under Apache 2.0 and runs on macOS with at least 8 GB of unified memory.

Augmentoolkit turns documents into datasets for training an LLM on a particular subject. It's for people building custom models around material such as research papers or fictional lore.

SwarmLLM is a ChatGPT alternative for people who want to run AI models on their own hardware. It can use a single computer or combine several machines to handle a model that won't fit on one.

WhisperLive turns speech into text as audio arrives, using OpenAI's Whisper models. It's for developers and teams who want to run transcription on their own machine or server and connect it to live audio or recorded files.

Magnitude is a local LLM engine for people who want to run AI agents on their own hardware. It adapts model execution to your machine and connects to coding assistants you already use.

GPT4All is a local AI chatbot for people who want language models and document conversations on their own computer. Model inference and LocalDocs processing run on the device without a cloud connection.

Deltafin runs full Kimi K3 on a single device for researchers and developers exploring large models on their own hardware. It preserves the original expert weights and complete expert bank, but responses are slow and long conversations take substantially longer than short completions.

Pooled runs a single open model across laptops, desktops and phones in browser tabs. It lets people combine their devices' memory to use a model that wouldn't fit on one device, for shared chat or coding tasks.