NVIDIA Dynamo
A self-hosted inference framework that coordinates NVIDIA GPU clusters with SGLang, TensorRT-LLM or vLLM and provides an OpenAI-compatible API.
NVIDIA Dynamo is a self-hosted inference framework for teams serving generative AI models across multiple GPUs or server nodes. It coordinates inference engines so teams can manage capacity, response latency and cached context across a cluster.
- Independent scaling: Prompt processing and token generation can run in separate GPU pools, with automatic resource adjustments to meet latency targets as demand changes.
- Cache-aware routing: Requests go to workers based on their load and cached context, reducing repeated prompt processing.
- Memory and replica startup: Cached context can move between GPU memory, CPU memory, SSDs and remote storage, while ModelExpress transfers model weights between GPUs to shorten replica startup.
- Model workloads: SGLang, TensorRT-LLM and vLLM support multimodal inference and tool calling through Dynamo; FastVideo and SGLang Diffusion support video generation.
- Performance planning: AI Perf benchmarks serving performance, and AISimulate estimates deployment behavior without requiring a running GPU cluster.
Dynamo exposes an OpenAI-compatible API. For Kubernetes deployments, Grove schedules related components together and accounts for hardware connections across hosts and racks. Cache offloading through the KV Block Manager works with TensorRT-LLM and vLLM. Optional S3 or Azure Blob storage puts offloaded cache in those external services.
The framework targets NVIDIA GPU fleets. For a single model on one GPU, the inference engine alone is usually sufficient.
Logotyper
