Infrastructure for Running AI on Your Hardware

Weft is a language and framework for developers who want to run AI workflow orchestration on their own machine. Its runtime requires Docker, and its graph editor works in VS Code and compatible forks. Self-hosting the workflow doesn't make its model calls local: the included WhatsApp support bot uses OpenRouter for inference.

RKLLM is a toolkit for developers building local AI applications on Rockchip boards. It runs language and multimodal models on the chip's neural processing unit (NPU).

Roboflow Inference runs computer vision models on images and camera feeds for teams building inspection, monitoring and image analysis systems. You can host it on your own computer, server or edge device.

DeepGEMM is an open-source CUDA kernel library for developers building LLM training and inference systems on their own NVIDIA GPUs. It supplies matrix operations and specialized computations for mixture-of-experts (MoE) models.

LLMKube manages self-hosted LLM inference for teams and homelab owners who need several machines to serve models together. It brings Linux GPU servers and Apple Silicon Macs into the same Kubernetes-managed fleet.

Agentgateway is an open-source proxy for platform teams connecting applications and agents to models, tools and other agents. It brings routing and access controls together across AI protocols and ordinary service traffic.

dstack is a self-hosted AI compute orchestration tool for teams running development, training and inference across cloud providers and their own servers. It gives those workloads a shared interface across Kubernetes, VMs and bare-metal clusters.

NullClaw runs AI agents on your own hardware or server for coding work and automated background tasks. It's aimed at developers who want an assistant that can carry out multi-step jobs with controls over its access and actions.

Willow is a voice assistant for people who want to control home automation through hardware and speech processing they own. It provides an alternative to Amazon Echo or Google Home on Espressif's ESP32-S3-BOX hardware family.

OpenSandbox gives developers isolated environments for AI agents that need to execute code or interact with browsers and developer tools. It runs on your own infrastructure, locally with Docker or on a Kubernetes cluster.

CubeSandbox is a self-hosted sandbox service for teams that need AI agents to execute code on their own infrastructure. Each sandbox has its own operating system kernel inside a hardware-isolated MicroVM.

LiteRT is Google's framework for developers adding machine learning and generative AI to apps that run on users' devices. It succeeds TensorFlow Lite and supports local inference without a server round trip.

KubeAI is a Kubernetes operator for teams serving AI models on their own infrastructure or cloud clusters. It manages model servers and adjusts capacity to demand, including starting models from zero running replicas.

AgentENV runs isolated environments for AI agents across your own servers. It's for teams building parallel agent workflows or reinforcement learning systems that need many separate sandboxes.

exo is a local LLM runner for people who want to run models too large for a single machine's memory. It joins devices into a cluster and runs inference on your own hardware.

macmon is a local system monitor for people checking how demanding workloads, including local AI, use an Apple Silicon Mac. Its terminal interface shows live hardware readings and a process list without requiring administrator privileges.

Agent Router is an open source AI gateway for teams connecting agents to model providers and MCP tools. It gives applications a shared entry point while platform teams manage access and routing.

Needle is an AI model for developers building device assistants that turn requests into actions. It prioritizes automation over general chat, with smaller model variants for devices that have limited resources.

nvtop is a terminal monitor for people checking GPU activity on local AI machines and compute servers. Its htop-style interface shows multiple GPUs and accelerators, including the processes using them.

Max Headbox is a local AI agent for people who want a voice-controlled assistant that can carry out actions on a Raspberry Pi. It processes voice and runs models on your own hardware without relying on cloud AI providers.

ClearML is a self-hostable MLOps platform for researchers and engineering teams managing model development and production. It connects experiment records with datasets, training jobs and model deployment so teams can trace results back to the work that produced them.

LLM Sandbox is a Python library for developers whose AI agents or applications need to execute generated code. It puts execution in containers on infrastructure you control, with support for local deployments and Kubernetes clusters.

Apache TVM is a machine learning compiler for developers optimizing models for deployment on their own hardware. It compiles model computations into modules that applications can embed, with targets spanning data center GPUs and edge devices. Code is licensed under Apache 2.0.

vLLM Production Stack is an open source LLM serving stack for teams running models on Kubernetes GPU clusters. It lets applications move from a single vLLM instance to a distributed deployment without changing application code.

MCP Context Forge is a gateway and registry for teams connecting AI clients to multiple tools, agents and services. It gives them one place to discover available services, control access and inspect calls.

Ray Serve is a Python library for developers building inference APIs around machine learning models. It's useful when a response needs several models or application steps, rather than a single model call.

taOS is a self-hosted AI agent platform for people who want to run assistants on their own computers and keep their conversations, files, and memory there. It brings agent management and local model access into a browser-based desktop.

Mooncake provides storage and data transfer components for teams running distributed LLM inference. Its Transfer Engine and Mooncake Store support shared caching and deployments that separate prompt processing from token generation.

Ultralytics YOLO is a Python computer vision library for developers building applications that interpret images and video. Models can run on hardware you control for tasks such as visual inspection, inventory monitoring and robotics perception.

Agent Lightning is a Python framework for developers and researchers training AI agents through reinforcement learning. It keeps the agent's existing tools, context and task environment involved in training, so feedback comes from the workflow the agent actually uses.

llm-d is a distributed LLM inference stack for infrastructure teams serving models across a Kubernetes cluster. It coordinates model servers such as vLLM and SGLang to handle production traffic across multiple machines.

OpenLIT helps developers investigate failures and assess the quality of existing AI applications and coding agents. It connects model requests with the tools, retrieval and agent steps involved in producing an answer.

Wyoming Satellite connects a microphone and audio playback device to Home Assistant for a self-hosted voice assistant. The project is archived and no longer maintained; Linux Voice Assistant replaces it using the ESPHome protocol.

ROCm is AMD's GPU computing platform for developers running AI training, inference and scientific workloads on their own workstations or servers. It provides the software needed to use AMD GPUs for these workloads, including local LLM inference.