Florence-2

Florence-2 is Microsoft's vision model for developers building applications that describe images, extract text or locate objects on their own hardware. One model handles several image tasks selected through text prompts.

• Image captions: Generate brief or detailed descriptions of a whole image, or describe individual regions with their locations.
• Object detection: Return object labels and bounding boxes, with optional confidence scores, or propose image regions for further processing.
• Text extraction: Read text from images with OCR and optionally return coordinates that connect each text region to its position in the picture.
• Phrase grounding and segmentation: Match phrases in a supplied caption to image regions and identify the areas occupied by objects.

The open-source model uses the MIT license and runs locally with PyTorch and Hugging Face Transformers. It supports CPU processing or CUDA GPUs, and Hugging Face distributes its weights in Safetensors format.

The family includes base and large models, plus versions fine-tuned across multiple tasks. Developers can use it without task-specific training or fine-tune it for their applications. The supplied Florence-2-large checkpoint has undergone additional pretraining, but Microsoft cautions that this training may be insufficient.