CLIP

CLIP is an image and text model for developers and researchers who want to classify pictures without building a labeled dataset for each task. You define the categories in natural language, and the model scores how closely each description matches an image.

• Classification without retraining: Change the candidate labels to apply the same model to different classification tasks without supplying new training examples.
• Image and text features: Use numerical representations of pictures and text in your own applications, or train a separate classifier on the image features.
• Recognition beyond objects: The model has demonstrated text recognition, geographic classification and action recognition in videos, though results vary by task.

CLIP runs locally through PyTorch on a CPU or CUDA GPU. It can load a local checkpoint or download model weights when needed. The project uses the MIT license.

Label wording can affect predictions. CLIP struggles with counting, estimating distances and distinguishing closely related categories such as car models or flower species. Its accuracy also drops on images unlike those in its training data.