LLMux

LLMux is a self-hosted AI gateway for developers who want to use local models and cloud providers through one interface while controlling where requests go. It blocks remote inference by default and logs permitted calls that leave the machine.

• Local and cloud backends: Connect Ollama, llama.cpp and vLLM, plus Anthropic, Gemini, Cohere, Bedrock, Azure and OpenAI-compatible providers.
• Routing and fallback: Model aliases, retries, fallback chains and least-cost selection let applications choose providers without handling each connection separately.
• Access and cost controls: Per-key budgets, rate limits and model restrictions govern usage, while response-level cost accounting and an optional dashboard show spending and activity.
• Response caching: Exact-match and semantic caches reuse responses, with storage in local memory or shared through Redis.
• Embedded or standalone operation: Run the gateway inside a Go application or as a standalone service with an OpenAI-compatible API; language packages also support embedding or a supervised child process.

Chat supports provider-specific translation, including tool calling, vision and streaming. Embeddings work through providers with compatible embedding APIs. Image, audio and other forwarded routes require OpenAI-compatible passthrough providers rather than native adapters.

LLMux is open source under Apache 2.0, requires no gateway account and has no telemetry. Local inference can stay on your machine, but pricing-catalog sync contacts OpenRouter and LiteLLM by default. Disabling that sync leaves the built-in pricing data available offline.