Kubernetes Tools for AI Inference and MLOps

Kubeflow is a self-hosted AI platform for teams developing and running machine learning workloads on Kubernetes. Its modular projects cover interactive development, training and the work of moving models into production.

Higress is an AI gateway for teams building AI apps and agents that need shared controls across model requests and tool calls. It brings model APIs, MCP services and Kubernetes inference workloads behind one entry point.

Optio coordinates coding agents and automated work for teams using their own infrastructure. A shared feed tracks pull requests, recurring jobs, interactive sessions and agents that stay available between tasks.

Kaito manages LLM inference, fine-tuning and document retrieval services in Kubernetes. It's for teams that want to run models in their own cluster with less manual work sizing GPU capacity and managing deployments.

Ollama Helm Chart packages a self-hosted LLM service for developers and infrastructure teams that already use Kubernetes. It brings Ollama into a cluster so applications can access models running on the cluster's hardware.

AIBrix provides infrastructure components for teams serving large language models on Kubernetes. It's aimed at organizations that need to manage inference capacity, request routing and models across a cluster.

Xinference serves AI models on your own computer or servers for developers and researchers building applications around models they control. It brings chat, document search and media generation into one platform.

BentoML is a Python framework for developers who want to turn AI models into services on their own hardware or servers. It handles model serving and application logic for LLMs, image generation, audio, and embeddings.

Flawless provides CISRE, a self-hosted AI SRE platform for teams investigating Kubernetes incidents. It connects evidence collection and diagnosis with approved changes and checks that the affected resource has recovered.

Kong Gateway gives platform teams a shared place to manage API, LLM and MCP requests. It helps teams apply access and traffic policies across services and AI applications on infrastructure they control.

Archestra Community Edition is a self-hosted AI platform for teams managing access to company tools and models. It gives coding assistants, internal agents, and staff chat a shared place to enforce policies and track activity.

NVIDIA Dynamo is a self-hosted inference framework for teams serving generative AI models across multiple GPUs or server nodes. It coordinates inference engines so teams can manage capacity, response latency and cached context across a cluster.

Helix is a paid, self-hosted platform for engineering teams coordinating coding agents. Agents work on separate tasks in parallel, while the team shares a view of their progress and reviews their work.

HolmesGPT is a self-hosted AI agent for DevOps and SRE teams investigating production incidents. It queries live observability data to find root causes across Kubernetes, VMs, bare metal, cloud services and databases.

OpenShell is a self-hosted runtime for teams that need autonomous AI agents to work with files and APIs under explicit access limits. It runs agents in isolated sandboxes and enforces policies at the kernel level.

Vald is a distributed vector search engine for developers building similarity search into applications. It runs on a self-hosted Kubernetes cluster and finds approximate nearest neighbors in dense vectors using NGT. It's open source under the Apache 2.0 license, with an architecture designed to spread large vector collections across multiple machines.

AX is a self-hosted AI agent orchestrator for teams running autonomous tasks on Kubernetes. It manages agents that retain state and need access to models, repositories and tools while they work.

OpenLLM is a self-hosted LLM server for developers connecting applications to models on their own hardware or servers. Its OpenAI-compatible interface lets existing clients use those models without adopting a separate API.

Triton Inference Server, offered by NVIDIA as Dynamo-Triton, is a self-hosted AI inference server for teams deploying models in applications. It brings models from different frameworks into one serving system across on-premises, cloud and edge hardware.

Seldon Core is an AI model serving framework for teams running machine learning models and LLM applications on Kubernetes. It manages individual models and applications with multiple processing stages.

OpenVINO Model Server is a self-hosted inference server for developers and teams serving AI models on their own machines or servers. It uses OpenVINO to run workloads on x86 CPUs, including Xeon, with acceleration for Intel integrated and discrete GPUs and NPUs. It's open source under Apache 2.0.

NVIDIA DCGM monitors and manages NVIDIA data-center GPUs for infrastructure teams running GPU clusters. It helps teams investigate slow jobs, detect hardware problems and control GPU power and clock settings.

LocalAI is a self-hosted AI runtime for developers and teams that want to run models on their own hardware. It brings language, speech, vision and media generation into one server that existing AI applications can connect to.

ToolHive runs and manages Model Context Protocol (MCP) servers for developers and platform teams that want control over the tools their AI clients can access. It supports individual desktop use and shared deployments on your own infrastructure.

KServe serves LLMs and predictive machine learning models on your own Kubernetes infrastructure. It's for teams that need to put models from different frameworks into production through one serving platform.

Olares is a self-hosted personal cloud OS for individuals and small teams that want AI agents, models and apps on hardware they own. It brings local AI and everyday file storage into a shared workspace you can access through a browser.

SkyPilot manages AI training, inference and development workloads across infrastructure you control. It's for AI teams sharing compute and infrastructure teams managing jobs across clusters and cloud providers.

Weft is a language and framework for developers who want to run AI workflow orchestration on their own machine. Its runtime requires Docker, and its graph editor works in VS Code and compatible forks. Self-hosting the workflow doesn't make its model calls local: the included WhatsApp support bot uses OpenRouter for inference.

LLMKube manages self-hosted LLM inference for teams and homelab owners who need several machines to serve models together. It brings Linux GPU servers and Apple Silicon Macs into the same Kubernetes-managed fleet.

Agentgateway is an open-source proxy for platform teams connecting applications and agents to models, tools and other agents. It brings routing and access controls together across AI protocols and ordinary service traffic.

dstack is a self-hosted AI compute orchestration tool for teams running development, training and inference across cloud providers and their own servers. It gives those workloads a shared interface across Kubernetes, VMs and bare-metal clusters.

OpenSandbox gives developers isolated environments for AI agents that need to execute code or interact with browsers and developer tools. It runs on your own infrastructure, locally with Docker or on a Kubernetes cluster.

CubeSandbox is a self-hosted sandbox service for teams that need AI agents to execute code on their own infrastructure. Each sandbox has its own operating system kernel inside a hardware-isolated MicroVM.

KubeAI is a Kubernetes operator for teams serving AI models on their own infrastructure or cloud clusters. It manages model servers and adjusts capacity to demand, including starting models from zero running replicas.