vLLM Production Stack
A self-hosted LLM serving stack built on vLLM, with an OpenAI-compatible API, request routing and GPU cluster monitoring under Apache 2.0.
vLLM Production Stack is an open source LLM serving stack for teams running models on Kubernetes GPU clusters. It lets applications move from a single vLLM instance to a distributed deployment without changing application code.
- Compatible API: Applications access models through the same OpenAI-compatible API that vLLM provides.
- Request routing: The router directs requests to backends running different models, supports model aliases, and distributes traffic through round-robin or session-based routing.
- Context reuse: Session-based routing and KV cache offloading give teams ways to reuse model context across requests.
- Cluster monitoring: Prometheus and Grafana show healthy instances, response latency, time to first token, queued and active requests, and GPU cache usage and hit rates.
- Backend discovery: Kubernetes service discovery and fault tolerance help the router track available serving instances.
The stack requires a running Kubernetes environment with GPUs, so it's aimed at server deployments rather than desktop chat. Teams host the serving engines, router and monitoring components on their own infrastructure. The project uses the Apache 2.0 license.
Logotyper
