LightEval
An open-source LLM evaluation toolkit for local or served models, with custom benchmarks and sample-level results, under the MIT license.
LightEval is a Python toolkit from Hugging Face for developers and researchers comparing LLMs or investigating where their responses fail. It evaluates models already loaded in memory as well as models accessed through a service.
- Benchmark coverage: Test knowledge, math, coding, instruction following and long-context behavior with benchmarks such as MMLU, GPQA, GSM8K, LiveCodeBench, IFEval and RULER.
- Individual results: Save and inspect responses for each test sample alongside overall scores to identify specific failures and compare models.
- Custom evaluations: Define tasks and metrics for cases that existing benchmarks don't cover.
- Multilingual testing: Evaluate language-specific performance with ArabicMMLU, CMMLU, GPQA-fr, RUMMLU and other benchmarks.
Local evaluation can run on CPU or one or more GPUs through Hugging Face Accelerate. LightEval also supports Inspect AI as an evaluation backend. Evaluating a served model sends requests to the service hosting that model.
The project is open source under the MIT license. It targets macOS and Linux; Windows is untested and unsupported.
Logotyper
