MATLAB tutorial: label images with CLIP and Grounding DINO

This tutorial demonstrates automated aerial image labeling with vision-language models in MATLAB. The speaker introduces CLIP, Moondream and Grounding DINO as available in Computer Vision Toolbox for R2026a. The example selects video frames containing airplanes and labels commercial jets and private jets.

CLIP compares text and image embeddings to find relevant frames. The workflow loads a pretrained model, computes similarity scores and keeps frames above a chosen threshold. The speaker recommends short phrases over single words. An initial airplane prompt misses some relevant frames, so Moondream generates captions for the selected images. A word cloud helps identify terms for a revised prompt, "an aerial view of airplanes at an airport." The demonstration reports that this search retrieves more airplane frames.

Grounding DINO then produces bounding boxes for the requested aircraft classes. The tutorial explains both programmatic access and the Image Labeler app, where users choose class names, a model and a threshold before running detection and accepting results. The boxes can support object detector training. For segmentation labels, the example passes them to SAM2 to generate pixel-level masks. The workflow includes result review before acceptance; the demonstrated retrieval improvement does not establish accuracy across other datasets.