Portkey Gateway

Portkey Gateway routes application requests across model providers through an OpenAI-compatible API. It's for developers who need shared routing and failure controls when working with multiple AI backends.

• Provider routing: Connect to Ollama, OpenAI, Anthropic and Google Gemini, with conditional routing and load balancing across providers or API keys.
• Failure handling: Retry failed calls, fall back to another model or provider, and apply timeouts to slow requests.
• Input and output checks: Apply built-in or custom guardrails to prompts and model replies according to your chosen rules.
• Media and streaming: Call vision, speech and image generation models, use supported streaming responses and access OpenAI realtime APIs.
• Response caching: Reuse cached model responses to reduce repeated calls.

The gateway code is open source under the MIT license. Portkey offers hosted operation and enterprise private deployments on AWS, Azure, GCP, OpenShift and Kubernetes. The gateway connects to model providers; deploying it privately doesn't make those providers' models part of the gateway itself.

Agent framework integrations include AutoGen, CrewAI, LangChain and LlamaIndex.