GPU Monitoring and Management Tools

Kaito manages LLM inference, fine-tuning and document retrieval services in Kubernetes. It's for teams that want to run models in their own cluster with less manual work sizing GPU capacity and managing deployments.

AIBrix provides infrastructure components for teams serving large language models on Kubernetes. It's aimed at organizations that need to manage inference capacity, request routing and models across a cluster.

jetson-stats monitors NVIDIA Jetson boards and controls their power and cooling settings. It's for developers running workloads, including local AI, who need to inspect resource use and adjust how their board operates.

Marbor is a self-hosted AI inference gateway for operators serving models across their own GPU fleet. It places requests using live hardware and model state to reduce cold loads and avoid deployments that can't serve the requested model.

NVIDIA Dynamo is a self-hosted inference framework for teams serving generative AI models across multiple GPUs or server nodes. It coordinates inference engines so teams can manage capacity, response latency and cached context across a cluster.

gpustat is a local terminal monitor for people running AI workloads or sharing accelerator hardware. It helps you check device activity and identify the jobs using memory without opening a separate dashboard.

NVIDIA Container Toolkit lets developers and server operators build and run containers that use NVIDIA GPUs. It provides GPU access for containerized applications on a Linux machine or server.

NVIDIA DCGM monitors and manages NVIDIA data-center GPUs for infrastructure teams running GPU clusters. It helps teams investigate slow jobs, detect hardware problems and control GPU power and clock settings.

Olares is a self-hosted personal cloud OS for individuals and small teams that want AI agents, models and apps on hardware they own. It brings local AI and everyday file storage into a shared workspace you can access through a browser.

nvitop is a terminal GPU monitor for people running AI workloads on NVIDIA hardware. It shows GPU activity alongside the processes using it, so you can check resource use and investigate busy devices.

SkyPilot manages AI training, inference and development workloads across infrastructure you control. It's for AI teams sharing compute and infrastructure teams managing jobs across clusters and cloud providers.

LLMKube manages self-hosted LLM inference for teams and homelab owners who need several machines to serve models together. It brings Linux GPU servers and Apple Silicon Macs into the same Kubernetes-managed fleet.

dstack is a self-hosted AI compute orchestration tool for teams running development, training and inference across cloud providers and their own servers. It gives those workloads a shared interface across Kubernetes, VMs and bare-metal clusters.

macmon is a local system monitor for people checking how demanding workloads, including local AI, use an Apple Silicon Mac. Its terminal interface shows live hardware readings and a process list without requiring administrator privileges.

nvtop is a terminal monitor for people checking GPU activity on local AI machines and compute servers. Its htop-style interface shows multiple GPUs and accelerators, including the processes using them.

ClearML is a self-hostable MLOps platform for researchers and engineering teams managing model development and production. It connects experiment records with datasets, training jobs and model deployment so teams can trace results back to the work that produced them.

OpenLIT helps developers investigate failures and assess the quality of existing AI applications and coding agents. It connects model requests with the tools, retrieval and agent steps involved in producing an answer.

ROCm is AMD's GPU computing platform for developers running AI training, inference and scientific workloads on their own workstations or servers. It provides the software needed to use AMD GPUs for these workloads, including local LLM inference.