CLIP tutorial: training, zero-shot inference and code

This tutorial explains CLIP, short for contrastive language image pre-training. It follows the image and text encoders into a shared embedding space, where the training objective brings matching pairs closer together and separates mismatched pairs. The walkthrough uses 512-dimensional vectors and covers L2 normalization, cosine similarity, a learned temperature parameter and cross-entropy loss.

For zero-shot classification, the speaker turns class names into prompts such as "a photo of a car" and compares their text embeddings with an image embedding. The explanation distinguishes this approach from training on labeled examples for the downstream classification task.

A Colab notebook example compares two pictures of the same person, one with a hat and one without. The speaker subtracts their image vectors, normalizes the difference and compares it with text vectors for hat, cup, cat and boat. Hat receives about 60% in the displayed result, an illustration of semantic information in the embeddings rather than a general accuracy measure.

The final technical section explains linear probing: a linear classifier learns from labeled data while the CLIP encoder stays frozen. The reported comparisons concern particular evaluations, rather than a universal advantage over other architectures.