Higress

Higress is an AI gateway for teams building AI apps and agents that need shared controls across model requests and tool calls. It brings model APIs, MCP services and Kubernetes inference workloads behind one entry point.

• Model routing: Connect to OpenAI, Claude and other model providers through a common protocol, with load balancing, model fallback and inference-aware routing on Kubernetes.
• MCP tools: Host MCP servers or turn OpenAPI descriptions into remote MCP services, with shared authentication, authorization and audit logs for tool calls.
• Usage and caching: Track token consumption, enforce rate limits and reuse responses through exact or semantic caching for repeated or similar requests.
• Input and output controls: Apply privacy protection to inputs sent to external models and filter model responses before they reach users.
• Streaming and extensions: Handle streaming requests and responses, including SSE, and extend traffic handling with Wasm plugins written in Go, Rust or JavaScript.

The free, open-source edition uses the Apache 2.0 license and can run locally in Docker without registration. A built-in console provides a management interface. Self-hosting the gateway doesn't keep external model requests on your machine: they still go to the provider you select. Alibaba Cloud also offers a paid, fully managed gateway.

Higress is built on Istio and Envoy. Configuration updates can take effect without interrupting long-lived connections.