Ray Serve
A self-hosted Python model serving library that combines models and application logic, with LLM streaming, request batching and multi-GPU support.
Ray Serve is a Python library for developers building inference APIs around machine learning models. It's useful when a response needs several models or application steps, rather than a single model call.
- Model composition: Combine multiple models and Python application logic into one inference service, with separate deployments for its component parts.
- Framework support: Serve PyTorch, TensorFlow, Keras, Scikit-learn and Hugging Face Transformers models alongside arbitrary Python functions.
- LLM serving: Stream responses, batch incoming requests dynamically, and distribute serving across multiple GPUs or machines.
- Shared compute: Use Ray's scheduling to share GPU resources among model deployments through fractional GPU allocation.
- HTTP interfaces: FastAPI integration handles request parsing and validation for inference endpoints.
Ray Serve is open source under Apache 2.0. You can run an application locally and scale it to a cluster using Ray, keeping the serving workload on hardware you choose. Its distributed serving support suits services that outgrow one machine, while local operation also supports development and smaller applications.
The library's application interfaces are Python-based, so developers define model connections and business logic in code. Models from different supported frameworks can participate in the same service.
Logotyper
