Inference Gateway

Inference Gateway is a self-hosted LLM proxy for developers building apps or AI agents that use multiple model providers. It gives those clients one OpenAI-compatible API so they can switch providers without rewriting their application integration.

• Model routing: Connect local backends such as Ollama and llama.cpp alongside OpenAI, Anthropic, Groq, Google and other cloud providers, with model aliases that can route requests to a pool of deployments.
• MCP tools: Let models discover and call tools from connected MCP servers while the gateway handles execution, including access to filesystems, databases and external APIs.
• Agent delegation: Discover specialized agents through Agent-to-Agent (A2A), delegate tasks within a conversation and stream their results.
• Access and guardrails: Centralize provider keys, rate limits and audit logging, with OIDC authentication, policy checks, and secret and personal data detection.
• Monitoring: Track requests through Prometheus metrics, OpenTelemetry traces and structured logs, with Grafana dashboards for inspection.

The project is open source under Apache 2.0 and runs as a standalone binary, in Docker or on Kubernetes. Its Kubernetes Operator manages gateways, agents and MCP servers. The gateway sends no analytics or telemetry home and can run in an air-gapped environment with local backends; choosing a cloud provider sends model requests to that provider.

MCP connections require servers that support the stateless protocol revision used by the gateway. Servers that require a legacy initialization handshake or a persistent session aren't compatible.