Infrastructure for Running AI on Your Hardware

SGLang is a self-hosted inference framework for teams serving AI models to applications. It supports production workloads on a single GPU or across distributed clusters, including AI agents and reinforcement learning rollouts.

vLLM serves language models on hardware you control. It's for developers and teams building model APIs that need to handle concurrent requests while using memory and compute efficiently.

ExecuTorch is an on-device AI runtime for developers adding PyTorch models to apps and embedded devices. It handles inference on the user's hardware across Android, iOS, macOS, Linux and Windows, as well as embedded systems and microcontrollers.

Kubeflow is a self-hosted AI platform for teams developing and running machine learning workloads on Kubernetes. Its modular projects cover interactive development, training and the work of moving models into production.

TensorRT is NVIDIA's inference toolkit for developers deploying AI applications on NVIDIA GPUs. Its compilers, runtimes and model optimizations target lower response latency and higher throughput on local machines, self-hosted servers and edge devices.

Higress is an AI gateway for teams building AI apps and agents that need shared controls across model requests and tool calls. It brings model APIs, MCP services and Kubernetes inference workloads behind one entry point.

Lucebox is a local AI workstation for developers and teams who want to run models on their own hardware. Its inference engine is also available separately for supported AMD and NVIDIA machines.

TensorFlow Model Optimization Toolkit is an open-source Python library for developers preparing TensorFlow and Keras models for deployment on mobile, IoT and other edge devices, or in the cloud. It helps reduce inference latency and cost, and adapt models to limits on memory, processing power, energy use and storage.

Optio coordinates coding agents and automated work for teams using their own infrastructure. A shared feed tracks pull requests, recurring jobs, interactive sessions and agents that stay available between tasks.

Kaito manages LLM inference, fine-tuning and document retrieval services in Kubernetes. It's for teams that want to run models in their own cluster with less manual work sizing GPU capacity and managing deployments.

YoMo is a self-hosted framework for developers building AI agents that call application tools. It supports distributing inference and tool execution across locations closer to an application's users.

Ollama Helm Chart packages a self-hosted LLM service for developers and infrastructure teams that already use Kubernetes. It brings Ollama into a cluster so applications can access models running on the cluster's hardware.

OpenVINO is an AI inference toolkit for developers deploying models in their own applications. It converts and optimizes models for local hardware or servers, covering generative AI and conventional machine-learning workloads.

LMCache helps teams serving LLMs reuse context that a model has already processed. It reduces repeated work on long prompts, conversation history and retrieved documents, shortening the wait for the first token and improving throughput.

AIBrix provides infrastructure components for teams serving large language models on Kubernetes. It's aimed at organizations that need to manage inference capacity, request routing and models across a cluster.

Xinference serves AI models on your own computer or servers for developers and researchers building applications around models they control. It brings chat, document search and media generation into one platform.

MNN is a C++ framework for developers running AI models on phones, PCs and embedded devices. It supports inference and training on the device, with a small application footprint and hardware acceleration.

Moondream is a vision language model for developers building apps that interpret images, such as warehouse or manufacturing footage. Its downloadable models can run on your own hardware, including offline in an air-gapped environment.

BentoML is a Python framework for developers who want to turn AI models into services on their own hardware or servers. It handles model serving and application logic for LLMs, image generation, audio, and embeddings.

OpenVoiceOS is a voice assistant platform for people building smart speakers or adding voice control to their own devices. Its core is a fork of MycroftAI/mycroft-core, and most classic Mycroft skills remain compatible.

Flawless provides CISRE, a self-hosted AI SRE platform for teams investigating Kubernetes incidents. It connects evidence collection and diagnosis with approved changes and checks that the affected resource has recovered.

py-xiaozhi is a Python AI voice client for people who want to try Xiaozhi conversations on a computer or build voice interaction into hardware projects. It builds on the xiaozhi-esp32 project and lets you use a desktop microphone and speakers without dedicated Xiaozhi hardware.

NVIDIA NIXL is a data transfer library for developers building AI inference systems such as NVIDIA Dynamo. It provides a common interface for moving data between inference components.

jetson-stats monitors NVIDIA Jetson boards and controls their power and cooling settings. It's for developers running workloads, including local AI, who need to inspect resource use and adjust how their board operates.

Kong Gateway gives platform teams a shared place to manage API, LLM and MCP requests. It helps teams apply access and traffic policies across services and AI applications on infrastructure they control.

Marbor is a self-hosted AI inference gateway for operators serving models across their own GPU fleet. It places requests using live hardware and model state to reduce cold loads and avoid deployments that can't serve the requested model.

DeepSpeed is an open-source PyTorch library for developers and researchers training or running large AI models on their own hardware or compute clusters. It helps address memory limits and distribute workloads across devices.

LiteRT Torch is a Python library for converting PyTorch models into .tflite files for the LiteRT runtime. Developers can prepare models on Linux and deploy them into Android, iOS and IoT applications that run inference on the device. Code is licensed under Apache 2.0.

Archestra Community Edition is a self-hosted AI platform for teams managing access to company tools and models. It gives coding assistants, internal agents, and staff chat a shared place to enforce policies and track activity.

NVIDIA Dynamo is a self-hosted inference framework for teams serving generative AI models across multiple GPUs or server nodes. It coordinates inference engines so teams can manage capacity, response latency and cached context across a cluster.

DeepEP handles communication between GPUs for teams training or serving mixture-of-experts (MoE) models on their own servers. It moves tokens to the GPUs that hold the selected experts and combines their outputs.

NVIDIA cuML is a Python machine learning library for data scientists whose scikit-learn, UMAP or HDBSCAN workloads take too long on a CPU. It runs on NVIDIA GPUs in Linux or WSL 2, on a workstation or across a cluster. The project is open source under Apache 2.0.

MediaPipe is a toolkit for developers adding on-device AI to applications. It provides pretrained models and task APIs for applications that need to process images, video, text or audio on the user's device.

PicoClaw is a self-hosted personal AI assistant that can use local models through Ollama or vLLM. It's aimed at people who want an agent on their own hardware, including small Linux boards and Android phones. Hosted providers are optional, but choosing OpenAI, Anthropic, Google Gemini or DeepSeek sends model requests to those services and requires internet access and API keys.