OpenCompass
An open-source LLM evaluation framework that compares local models and cloud APIs on shared benchmarks, with support for multi-GPU evaluation.
OpenCompass is an LLM evaluation framework for researchers and developers comparing models for their applications. It lets you assess models on your own hardware alongside cloud APIs in the same experiment.
- Model comparison: Evaluate Hugging Face AutoModel-compatible models such as Qwen, Gemma and InternLM, plus OpenAI, Claude and Gemini APIs or inference engines with an OpenAI-compatible interface.
- Benchmark coverage: Compare performance on knowledge, reasoning, coding, science, language, long-context and safety tasks using recommended dataset configurations.
- Evaluation methods: Use zero-shot, few-shot and chain-of-thought prompts, with LLM judges available for some datasets. You can add custom models, datasets and evaluation strategies.
- Larger experiments: Distribute evaluation work across GPUs or split a Hugging Face model across multiple GPUs. Experiment records retain the settings used, and reports show results while evaluations run.
The Python framework runs locally and uses the Apache 2.0 license. Local model evaluation uses your own hardware; evaluating cloud APIs sends requests to the selected provider. Datasets can be prepared in advance, downloaded from OpenCompass storage or loaded on demand through ModelScope.
Logotyper
