Kubernetes Tools for AI Inference and MLOps
Serve models and run ML pipelines on Kubernetes with platforms such as KServe, Kubeflow and Ray Serve.
A self-hosted AI agent sandbox platform for Linux with Firecracker isolation, snapshot forks and an E2B-compatible API, licensed under MIT.
3.7KUpdated 1 day agoMIT
macOS · Linux · Docker#Code execution#Distributed execution#Resumable workflows
AgentENV runs isolated environments for AI agents across your own servers. It's for teams building parallel agent workflows or reinforcement learning systems that need many separate sandboxes.
Guardrails, Security and SandboxingKubernetes and Cloud-Native AI+0
Self-hosted AI gateway routes OpenAI-compatible requests to cloud models, vLLM and MCP tools, with centralized credentials and fallback routing.
2.4KUpdated 4 days agoApache-2.0
Docker#LLM tracing#MCP#Multi-user access
Agent Router is an open source AI gateway for teams connecting agents to model providers and MCP tools. It gives applications a shared entry point while platform teams manage access and routing.
Kubernetes and Cloud-Native AILLM Gateways and RoutersLLM Observability and Tracing+0
A self-hostable MLOps platform that tracks experiments, versions datasets and automates ML jobs, with model serving backed by NVIDIA Triton.
6.9KUpdated 2 weeks agoApache-2.0
Docker · Web#Code execution#Git integration#Multi-user access
ClearML is a self-hostable MLOps platform for researchers and engineering teams managing model development and production. It connects experiment records with datasets, training jobs and model deployment so teams can trace results back to the work that produced them.
Experiment Tracking and MLOpsGPU Management and MonitoringInference Servers and ClustersKubernetes and Cloud-Native AILLM Fine-Tuning+0
A self-hosted code interpreter Python library that runs AI-generated code in Docker, Podman or Kubernetes and captures generated plots.
1.1KUpdated 1 week agoMIT
Docker#Code execution#MCP#Tool calling
LLM Sandbox is a Python library for developers whose AI agents or applications need to execute generated code. It puts execution in containers on infrastructure you control, with support for local deployments and Kubernetes clusters.
Guardrails, Security and SandboxingKubernetes and Cloud-Native AILLM Libraries and SDKsMCP Servers and Agent Skills+0
A self-hosted LLM serving stack built on vLLM, with an OpenAI-compatible API, request routing and GPU cluster monitoring under Apache 2.0.
2.7KUpdated 1 week agoApache-2.0
Web#OpenAI-compatible API#Prompt caching
vLLM Production Stack is an open source LLM serving stack for teams running models on Kubernetes GPU clusters. It lets applications move from a single vLLM instance to a distributed deployment without changing application code.
Inference Servers and ClustersKubernetes and Cloud-Native AILLM Gateways and RoutersLLM Observability and Tracing+0
A self-hosted AI gateway connects MCP servers, agents and REST/gRPC APIs through one endpoint, with access controls and OpenTelemetry tracing.
4.8KUpdated 23 hours agoApache-2.0
Windows · Docker · Web#LLM tracing#MCP#Multi-user access
MCP Context Forge is a gateway and registry for teams connecting AI clients to multiple tools, agents and services. It gives them one place to discover available services, control access and inspect calls.
Kubernetes and Cloud-Native AILLM Gateways and RoutersLLM Observability and TracingMCP Servers and Agent SkillsPrompt Management and Playgrounds+0
A self-hosted Python model serving library that combines models and application logic, with LLM streaming, request batching and multi-GPU support.
44.3KUpdated 18 hours agoApache-2.0
#Batch processing#Hugging Face integration#ONNX
Ray Serve is a Python library for developers building inference APIs around machine learning models. It's useful when a response needs several models or application steps, rather than a single model call.
Inference Servers and ClustersKubernetes and Cloud-Native AI+0
An open-source AI agent training framework using reinforcement learning with existing agent harnesses, local execution and Kubernetes support.
19KUpdated 3 days agoMIT
#Code execution#MCP#Multimodal input
Agent Lightning is a Python framework for developers and researchers training AI agents through reinforcement learning. It keeps the agent's existing tools, context and task environment involved in training, so feedback comes from the workflow the agent actually uses.
Kubernetes and Cloud-Native AILLM Fine-TuningTraining Frameworks+0
An open-source LLM inference stack for self-hosted Kubernetes clusters that coordinates vLLM and SGLang across GPUs, TPUs, XPUs and CPUs.
5.3KUpdated 2 days agoApache-2.0
#Batch processing#Distributed execution#OpenAI-compatible API
llm-d is a distributed LLM inference stack for infrastructure teams serving models across a Kubernetes cluster. It coordinates model servers such as vLLM and SGLang to handle production traffic across multiple machines.
Inference Servers and ClustersKubernetes and Cloud-Native AILLM Gateways and Routers+0
An open-source GPU computing platform for local AI training and inference on AMD hardware, with Linux and Windows support and PyTorch, JAX and vLLM.
6.8KUpdated 2 months agoMIT
Windows · Linux
ROCm is AMD's GPU computing platform for developers running AI training, inference and scientific workloads on their own workstations or servers. It provides the software needed to use AMD GPUs for these workloads, including local LLM inference.
GPU Management and MonitoringKubernetes and Cloud-Native AI+0
LogotyperGot an idea for an AI tool? I'm Eugene, the designer and developer behind Localhosted. I'll help you build it, from the first sketch to launch.
Start your project
