NVIDIA Triton: inference serving and deployment overview
Learn how NVIDIA Triton serves models on CPUs and GPUs, handles HTTP or gRPC requests, and uses Model Analyzer to compare deployment configurations.
Player not loading? Watch on YouTube
This Portuguese-language overview is summarized in English from its complete Portuguese captions.
NVIDIA solutions architect João Paulo introduces Triton Inference Server as a way to standardize model inference deployments. He frames the problem around serving several machine learning models while making efficient use of compute resources and exposing them through scalable services. This is an architectural overview rather than a step-by-step installation tutorial.
The speaker describes Triton as open source and says it supports deployment in the cloud, in a local data center, or at the edge, using CPUs or GPUs. He discusses support for multiple model frameworks and workloads that include real-time queries, batch processing and streaming. These options make the overview relevant to self-hosted inference, though the video does not specify hardware requirements or demonstrate an offline setup.
The technical explanation follows requests through standard HTTP or gRPC interfaces into automatic scheduling queues for individual models. João Paulo also introduces NVIDIA Model Analyzer, which he says tests configuration options to identify suitable batch and concurrency settings for a workload. He closes with access options: NVIDIA's official GitHub code and an optimized Docker container distributed through NGC. He describes monthly container updates, but provides no commands, benchmark results or version-specific deployment instructions.
Design Blog
