Triton Inference Server vs FastAPI for model deployment

This Russian-language walkthrough is summarized in English from its complete Russian captions.

The speaker compares Triton Inference Server with FastAPI for deploying machine learning models as services. His proposed architecture puts inference inside Triton and uses FastAPI as an outer API layer when an application needs more flexible routes. He recommends Triton strongly, but the walkthrough does not establish that it is the best choice for every deployment.

The self-hosted example uses Docker containers for Triton, FastAPI, Prometheus and Grafana. Prometheus collects Triton's metrics from port 8002, while Grafana displays GPU memory use and other measurements through a local dashboard on port 3000. The speaker also shows monitoring attached to FastAPI and says that approach takes more code.

The tutorial explains model folders, input and output configuration, and pipelines that connect a detector to a classifier. For conditional logic, the speaker prefers a Python script that can call several models. He warns that passing requests between components can involve memory leaks.

Dynamic batching receives the most attention. In a hypothetical example, Triton waits up to 20 milliseconds for a batch of 100 images, sends a smaller batch when the wait expires, or sends a full batch sooner. The speaker notes that queue handling can affect latency. The closing discussion covers handoffs between ML and application teams and prediction differences that can arise when porting Python pipelines to C++.