LLaVA

LLaVA is a vision-language model project for researchers and developers who want to ask questions about images on their own hardware.

• Image conversations: Models can describe pictures, answer questions about their contents, reason about visual details and read text in images.
• Local model comparison: The Gradio browser interface lets you chat with images and compare checkpoints, with inference running on your machine or self-hosted server.
• Model families: The original models pair CLIP with Vicuna; LLaVA-NeXT includes Llama 3 and Qwen 1.5 variants and supports video tasks.
• Research and training: Public checkpoints, visual instruction datasets and training code support work on multimodal models.

GPU inference supports multiple GPUs and 4-bit or 8-bit quantization. Quantization may let some models run with as little as 12GB of VRAM, but it can reduce accuracy. Apple M1 and M2 devices can use the MPS backend. SGLang provides a higher-throughput GPU serving option, though its LLaVA integration doesn't support 4-bit quantization. Hosted demos run separately from local inference.

The linked repository covers LLaVA and LLaVA-NeXT research code and model downloads. Its code uses Apache 2.0. The current usage notice requires compliance with the licenses of each dataset and checkpoint, including the chosen base language model; it does not impose an additional blanket research-only restriction on the Apache-licensed code.