AIBrix
Open-source LLM serving infrastructure for Kubernetes that routes requests, scales capacity and supports multi-node inference with vLLM integrations.
AIBrix provides infrastructure components for teams serving large language models on Kubernetes. It's aimed at organizations that need to manage inference capacity, request routing and models across a cluster.
- Routing and scaling: Direct requests across models and replicas, and adjust inference resources as demand changes.
- Distributed inference: Serve workloads across multiple nodes and separate prompt processing from response generation with prefill-decode disaggregation.
- LoRA serving: Manage lightweight model adaptations at high density and load them dynamically.
- Cache capacity: Offload cached inference state and reuse it across engines to manage memory across a deployment.
- Operations and measurement: Manage model downloads, standardize inference metrics, detect GPU hardware failures and benchmark performance and resource efficiency.
AIBrix is open source under Apache 2.0 and supports self-hosted Kubernetes deployments. Its serving integrations include vLLM Semantic Router and vLLM-Omni for multimodal workloads. Batch inference supports jobs that don't require interactive responses.
Logotyper
