Kaito

Kaito manages LLM inference, fine-tuning and document retrieval services in Kubernetes. It's for teams that want to run models in their own cluster with less manual work sizing GPU capacity and managing deployments.

• Model deployment: Run Hugging Face models supported by vLLM, including workloads that use LoRA adapters, with inference settings selected for single-node or distributed operation.
• GPU provisioning: Estimate a model's GPU memory needs and provision capacity through node provisioners compatible with Karpenter-core APIs.
• Demand-based scaling: Use KEDA and vLLM metrics to adjust model replicas as request load changes.
• Request routing: Connect compatible gateways through Gateway API Inference Extension to route requests with awareness of cached model context.
• Document retrieval: Deploy a LlamaIndex-based RAG service with local or remote embeddings, an optional LLM endpoint, and a vector database.

Model workloads run in your Kubernetes cluster, and vLLM is the only supported inference engine. Kaito can store inference model files on GPU nodes' built-in local NVMe storage without requiring separate inference storage.

Document retrieval uses an in-memory FAISS database by default, with Qdrant or Milvus available for persistent storage. Local embeddings run within your infrastructure; remote embedding services receive document content for processing.