Player not loading? Watch on YouTube
This tutorial builds a Python pipeline for analyzing basketball footage with RF-DETR, SAM2 and SmolVLM2. It covers player detection, team assignment, jersey recognition and shot outcomes, then maps player positions onto a top-down court. The workflow uses open source models, but the demonstrated training runs on Roboflow and the notebooks run in Google Colab. It does not establish an offline setup or hardware requirements.
Supervision supplies the video frame generator, converts inference outputs into detections and keypoints, filters classes, and draws boxes, labels and masks. The walkthrough configures its annotators and reuses them as tracking and team assignments are added.
RF-DETR detects players and jersey-number regions. Its player boxes initialize SAM2 tracking without manual prompts. The speaker chooses the large SAM2 checkpoint for tracking quality and identifies it as the pipeline's slowest component. Mask cleanup removes disconnected regions that could disrupt later analysis.
Team assignment uses SigLIP embeddings, UMAP and K-means with two clusters. The clusters need an external mapping to actual team names. For jersey OCR, the speaker reports that fine-tuning SmolVLM2 raises accuracy on their test set from 56% to 86%. Multiple frames help resolve hidden numbers and incorrect readings before roster lookup.
Court mapping uses a YOLO 11 model with 33 landmarks and requires at least four corresponding point pairs. Jumping and overlapping players can distort projected positions. Trajectory cleanup needs the full sequence, so this step suits postgame analysis rather than live tracking. A shot event tracker classifies attempts using shooting detections and a timed check for the ball in the basket.
