Olla
A self-hosted LLM proxy that routes requests across Ollama, vLLM and other backends, with automatic failover and a built-in monitoring dashboard.

Olla puts a single API endpoint in front of existing inference servers. It routes requests across mixed backends and switches servers when one fails before a response starts.
- Load balancing: Distribute requests across healthy servers, with automatic failover and recovery checks.
- Shared model catalogue: Discover models across Ollama, LM Studio, vLLM, SGLang, llama.cpp and other backends, with aliases for differently named copies of the same model.
- Conversation affinity: Keep multi-turn conversations on the backend holding their cached context to avoid repeated prompt processing.
- Embeddings and client APIs: Route embedding and reranking requests to Text Embeddings Inference and support OpenAI- and Anthropic-compatible clients.
- Monitoring: Inspect backend health, latency, traffic and loaded models through a browser dashboard, Prometheus metrics and request logs.
Olla uses Apache 2.0 and runs on Linux, macOS and Windows, with Docker support. It requires separate inference backends to serve models and integrates with OpenWebUI. LiteLLM can bridge requests to hosted providers such as OpenAI, Anthropic and Bedrock; those requests leave the local inference servers.