NVIDIA Dynamo explained: distributed LLM serving

NVIDIA Dynamo is presented as an open source framework for distributed generative AI inference across multiple GPUs and nodes. The speaker explains its role above serving engines such as vLLM and TensorRT-LLM: Dynamo coordinates requests, memory and workers rather than replacing those engines.

The explanation starts with prefill, which processes the prompt, and decode, which generates the response one token at a time. The speaker describes prefill as compute intensive and decode as more dependent on memory. Separating these phases lets teams scale their workers independently according to prompt length and demand for output.

KV cache is another focus. It stores information that avoids repeated computation of earlier context. The overview explains how Dynamo can route requests to workers with useful cached context and manage cache blocks across workers. Its architecture discussion also covers data transfer between GPUs and nodes, Kubernetes topology planning, and model weight caching through Model Express.

The speaker cites NVIDIA's claim of up to seven times more requests on Blackwell under benchmark conditions, while stressing that results depend on the model, hardware, workload and deployment. For people who run models locally on a single GPU, the speaker says Dynamo is probably unnecessary. The intended audience is teams operating large inference clusters, including workloads with long context and agents that repeatedly call tools.