OpenZiti LLM Gateway
A self-hosted LLM gateway for Linux, macOS and Windows that connects Ollama, vLLM and cloud APIs through an OpenAI-compatible interface. Apache 2.0 licensed.
OpenZiti LLM Gateway is a self-hosted API proxy for developers and teams connecting AI applications to inference servers across private networks and cloud services. Its optional zrok integration uses OpenZiti for identity-based access and end-to-end encryption, so clients and backends can connect across network boundaries without opening inbound ports.
The gateway works with Ollama, vLLM, llama-server and SGLang, alongside OpenAI and Anthropic. Applications use an OpenAI-compatible interface, and the gateway translates requests for Anthropic's Messages API. It supports streamed responses across providers. Inference runs on the selected backend: local routes use your own servers, while OpenAI and Anthropic routes send requests to those cloud services.
Requests can follow an explicit model choice or use automatic selection based on prompt content. Semantic routing combines keyword rules with embedding comparisons and an LLM classifier for ambiguous requests. Open WebUI users can also select an automatic routing model. For teams with several inference machines, weighted load balancing distributes traffic across a mixed pool of backends; health checks and automatic failover skip unavailable servers.
Virtual API keys control client access and can restrict which models each client uses. The open source gateway runs on Linux, macOS and Windows under the Apache License 2.0. Its single Go executable doesn't require a database or message queue. Prometheus metrics report request latency, token usage, routing decisions and backend health.
