NVIDIA Blackwell and Dynamo: agent scaling and deployment

This NVIDIA presentation explains how to assess and deploy AI agent workloads on server infrastructure. The speakers argue that token throughput alone misses the repeated model requests, tool calls and growing context within a task. They discuss measuring concurrent agents against responsiveness targets, alongside throughput and cost.

The benchmark example is AgentPerf, which replays coding trajectories from public repositories using DeepSeek V4 Pro. NVIDIA reports that GB300 NVL72 supported up to 40 times more concurrent agents per GPU than Hopper. At a target of about 20 tokens per second per user, the cited figures are roughly 57 agents per GPU versus 1.5. These are results presented for the benchmark configuration, rather than a guarantee for every deployment. The speaker says the results do not cover DGX Spark.

The deployment discussion covers self-hosted inference with NVIDIA Dynamo. It explains how distributing model experts and replicating attention layers affects request routing. Dynamo's router considers both KV cache location and worker load so repeated requests can reuse context while the system balances traffic.

The Kubernetes example requires the Dynamo operator and a Dynamo Graph Deployment YAML file. It specifies KV-aware routing, six prefill replicas and one decode replica using 16 GPUs with data and expert parallelism. The walkthrough concerns rack-scale infrastructure; the speakers also point to open source documentation and tutorials.