Self-Hosted LLM Gateways and Routers

Portkey Gateway routes application requests across model providers through an OpenAI-compatible API. It's for developers who need shared routing and failure controls when working with multiple AI backends.

MCP Gateway & Registry gives platform teams a shared catalog and access gateway for the tools their AI agents and coding assistants use. It addresses scattered server connections, credentials and access policies across an organization.

Pydantic AI is a Python SDK for developers building agents into their applications. It uses Pydantic validation to give application code structured results and check the arguments agents pass to tools.

LLMKube manages self-hosted LLM inference for teams and homelab owners who need several machines to serve models together. It brings Linux GPU servers and Apple Silicon Macs into the same Kubernetes-managed fleet.

CliRelay is an AI gateway for teams that want to manage coding assistants through a shared service on their own server. It builds on CLIProxyAPI and brings subscription accounts and provider API keys under central control.

Agentgateway is an open-source proxy for platform teams connecting applications and agents to models, tools and other agents. It brings routing and access controls together across AI protocols and ordinary service traffic.

KubeAI is a Kubernetes operator for teams serving AI models on their own infrastructure or cloud clusters. It manages model servers and adjusts capacity to demand, including starting models from zero running replicas.

Msty is a family of AI products for individuals, teams and organizations working with models and their own knowledge. Msty Studio is its workspace product, with separate products for agents and centralized model access.

APIPark is an AI gateway and API developer portal for teams managing model services and business APIs. It gives applications a common request format and lets teams share APIs with controlled access.

Agent Router is an open source AI gateway for teams connecting agents to model providers and MCP tools. It gives applications a shared entry point while platform teams manage access and routing.

Bifrost is an AI gateway for developers and teams building applications that use local LLMs or multiple cloud providers. It gives those applications a shared API for model access and a central place to manage usage.

Headroom reduces the context that LLM apps and coding agents send to a model. It's for developers whose prompts fill up with repetitive tool results, logs, file contents and retrieved documents. Compression runs locally, but model requests still go to the chosen LLM provider.

AQBot is a desktop AI workspace for people who want model chat, coding agents and document search in one app. It keeps application data and user files locally, while requests to cloud models go to their providers.

vLLM Production Stack is an open source LLM serving stack for teams running models on Kubernetes GPU clusters. It lets applications move from a single vLLM instance to a distributed deployment without changing application code.

Codex Router runs on your computer and connects the Codex App and CLI to external model providers or your own self-hosted endpoints. It's for developers who want a different model within their existing coding workflow. Cloud routes send model requests to remote services and need internet access; paid providers still charge for access or usage.

Claude Code Router is a local gateway for developers using coding agents such as Claude Code, Codex and OpenCode. It keeps provider selection and request routing in one app, while the selected cloud provider processes model requests remotely. Those requests leave your machine.

MCP Context Forge is a gateway and registry for teams connecting AI clients to multiple tools, agents and services. It gives them one place to discover available services, control access and inspect calls.

aichat is a terminal AI assistant for people who want to discuss files, generate shell commands and use model-powered tools from the command line. It connects to Ollama, OpenAI, Claude, Gemini and OpenAI-compatible API providers.

LLMux is a self-hosted AI gateway for developers who want to use local models and cloud providers through one interface while controlling where requests go. It blocks remote inference by default and logs permitted calls that leave the machine.

Paddler lets product and infrastructure teams serve AI models on their own servers and spread requests across multiple machines. It's built for applications that need text generation or embeddings while keeping inference within infrastructure the team controls.

Torrix is a self-hosted LLM monitoring tool for teams investigating AI spending and failed requests. It keeps request records in your infrastructure and brings usage analytics and individual traces into one dashboard.

llm-d is a distributed LLM inference stack for infrastructure teams serving models across a Kubernetes cluster. It coordinates model servers such as vLLM and SGLang to handle production traffic across multiple machines.

LunarGate is a self-hosted AI gateway for teams whose apps use multiple model providers. It gives those apps one OpenAI-compatible endpoint and keeps routing, retries and fallback rules in your infrastructure. The gateway is open source under the MIT license and runs on macOS and Linux, with a container image also available.

TensorZero is a self-hosted platform for developers who need to compare and improve models in LLM applications. The project is archived and no longer maintained.