Ferret
A local vision-language model connects text with specific image regions, uses Vicuna checkpoints, and runs GPU inference with a Gradio browser interface.
Ferret is a local vision-language model for researchers studying how AI understands specific parts of an image. It supports questions about selected regions and can identify image regions in its answers. Its code and data are restricted to research use.
- Image region references: Ask about particular objects or details using region references in different forms, without relying on a fixed set of object labels.
- Region location in answers: The model can connect its responses to locations within an image, supporting conversations about fine-grained visual details.
- Training and evaluation: The project provides training code, the GRIT instruction-tuning dataset, and Ferret-Bench for evaluating region references and location alongside meaning, knowledge, and reasoning.
Ferret uses Vicuna as its base language model and LLaVA's pretrained visual projector. It provides 7B and 13B checkpoint weight differences. The local demo runs inference on a GPU and exposes a Gradio interface in the browser.
The GRIT dataset carries a CC BY-NC 4.0 license, and models trained on it are restricted to research purposes. Apple's released weight differences carry a CC-BY-NC license. Use must also follow the applicable LLaMA, Vicuna, and GPT-4 license terms.
Logotyper
