Triton Inference Server

Triton Inference Server, offered by NVIDIA as Dynamo-Triton, is a self-hosted AI inference server for teams deploying models in applications. It brings models from different frameworks into one serving system across on-premises, cloud and edge hardware.

• Multiple frameworks: Serve TensorRT, PyTorch, ONNX, OpenVINO, Python and RAPIDS FIL models, with custom backends for other execution needs.
• Batching and concurrency: Run models concurrently and group requests through dynamic batching, with sequence batching for workloads that retain state between requests.
• Model pipelines: Combine models with preprocessing and postprocessing for ensemble workloads, including audio and video streams.
• Performance analysis: Use Model Analyzer to profile model configurations and compare serving performance.
• Application integration: Connect through HTTP or gRPC with Python and C++ clients, integrate Kubernetes for scaling, and collect monitoring metrics through Prometheus.

Triton is open source under BSD-3-Clause. It runs on x86 and ARM CPUs, NVIDIA GPUs and supported non-NVIDIA accelerators, including AWS Inferentia. Linux Docker containers support x86 and ARM; releases are also available for Windows and NVIDIA Jetson.

NVIDIA offers commercial support through NVIDIA AI Enterprise. The separate NVIDIA Dynamo project provides complementary LLM capabilities such as prefix caching and disaggregated serving.