LMMS Eval

LMMS Eval is a Python toolkit for researchers and model teams comparing AI models on text, image, video and audio tasks. It uses a shared evaluation pipeline to reduce differences caused by separate benchmark implementations.

• Multimodal benchmarks: Evaluate image understanding with MMMU, MME, MMBench and MathVista, or video understanding with VideoMME and EgoSchema, alongside audio and text tasks.
• Model compatibility: Test Qwen2.5-VL, Qwen3-VL, LLaVA-OneVision, InternVL-2 and VILA, with vLLM and SGLang backends supporting models such as Qwen3.5.
• Statistical comparisons: Use confidence intervals, clustered standard errors and paired comparisons to assess differences beyond a single accuracy score.
• Custom evaluations: Add models and datasets, including conversations that mix text, images, video and audio.
• Remote evaluation jobs: An optional web interface supports interactive configuration, while a self-hosted HTTP server accepts jobs and returns their status and results from dedicated GPU nodes.

Evaluations can run on your own hardware or call an OpenAI-compatible endpoint. Requests to an external model API go to that endpoint. Async serving, adaptive batching and video processing optimizations support large evaluation runs. The server processes queued jobs sequentially and is intended for trusted environments; exposure to untrusted networks requires additional security layers.