KubeAI

KubeAI is a Kubernetes operator for teams serving AI models on their own infrastructure or cloud clusters. It manages model servers and adjusts capacity to demand, including starting models from zero running replicas.

• Model serving: Serve LLMs through vLLM and Ollama, transcribe audio with FasterWhisper, generate embeddings with Infinity and rerank search results with cross-encoder models.
• OpenAI-compatible API: Use OpenAI client libraries for chat, text generation, embeddings and audio transcription against models running in your cluster.
• Scaling and routing: Route requests with shared prompt prefixes to help vLLM replicas reuse cached work, queue requests while models start and retry requests after backend failures.
• Models and adapters: Automate model downloads and storage mounts, and manage dynamic LoRA adapters across replicas.
• Event workloads: Connect Kafka and PubSub to model-serving workloads; a bundled Open WebUI also provides browser chat.

KubeAI runs in Kubernetes, including local clusters, and supports CPUs, GPUs and TPUs. Its model catalog includes presets for common GPU types. Autoscaling doesn't require Istio, Knative or the Prometheus metrics adapter.

Models scaled to zero take time to start before they can answer the first request. The model proxy and operator ship together but can run independently. KubeAI is open source under Apache 2.0.