Molmo

Molmo is Ai2's family of vision-language models for working with images and text on your own hardware. It's aimed at developers and researchers who want to train models or evaluate them on visual tasks.

• Model choices: MolmoE-1B uses OLMoE, Molmo-7B-O uses OLMo, and Molmo-7B-D and Molmo-72B use Qwen2, all paired with an OpenAI CLIP vision encoder.
• Training and adaptation: The Python code supports pretraining and multitask training, with pretrained and fine-tuned weights available for each model.
• Vision encoder alternatives: Custom training can use SigLIP, MetaCLIP or DINOv2 to compare image encoders.
• Visual evaluation: Evaluate models with high-resolution images and distribute training or evaluation across machines; VLMEvalkit also supports the Hugging Face models, though it doesn't cover every dataset, including PixMo-Count.

The open-source code uses the Apache 2.0 license and builds on the OLMo codebase. Model weights are available through Hugging Face and in a format compatible with this repository. A separate vLLM implementation provides faster inference than the repository's evaluation path. Ai2 also offers a hosted public demo of Molmo-7B-D.

Evaluating Molmo-72B with this codebase requires multiple nodes. Models and training data require downloads, and training can run without W&B. Training with this repository can differ from the published models because some image downloads may fail and PixMo-Clocks isn't included by default.