KServe
A self-hosted AI model serving platform runs LLMs and predictive models on Kubernetes with vLLM, Hugging Face support and an OpenAI-compatible API.
KServe serves LLMs and predictive machine learning models on your own Kubernetes infrastructure. It's for teams that need to put models from different frameworks into production through one serving platform.
- LLM serving: Run Hugging Face models with vLLM or llm-d and expose an OpenAI-compatible API, with GPU acceleration including NVIDIA GPUs.
- Model memory: Model caching reduces repeated loading, while KV cache offloading to CPU memory or disk helps handle longer sequences.
- Predictive models and pipelines: Serve TensorFlow, PyTorch, scikit-learn, XGBoost and ONNX models, and combine preprocessing, predictions and explanations in InferenceGraph pipelines or ensembles.
- Scaling and rollouts: Adjust capacity to request volume, scale idle predictive workloads to zero, and compare deployments with canary rollouts and A/B testing, subject to deployment mode.
- Explanations and monitoring: Use feature attribution to examine predictions, log request payloads, and detect drift, outliers and adversarial inputs.
KServe is open source under Apache 2.0 and runs on a self-hosted Kubernetes cluster. It also integrates with Kubeflow. The standard Kubernetes deployment mode doesn't support canary deployment or request-based autoscaling with scale-to-zero.
Logotyper
