lm-evaluation-harness

lm-evaluation-harness is a Python framework for researchers and developers comparing language models on shared benchmarks. Public prompts help make results comparable across model checkpoints and research papers.

• Benchmark evaluation: Test generative models on standard academic tasks, including few-shot evaluations that provide examples in the prompt.
• Local model support: Evaluate Hugging Face transformers and vLLM models, GGUF weights through transformers, or models served by llama.cpp.
• Adapted models: Compare LoRA adapters supported by Hugging Face PEFT and transformers models with steering vectors applied.
• Custom evaluation: Define your own prompts and metrics when standard benchmarks don't cover the behavior you need to measure.

The framework is open source under the MIT license and supports local models and benchmark datasets. OpenAI and TextSynth integrations send evaluation requests to those external services. It also supplies the evaluation backend for Hugging Face's Open LLM Leaderboard.

Hugging Face evaluation can use multiple GPUs to distribute examples or split model weights. Native multi-node evaluation isn't supported. The llama.cpp backend requires a compatible OpenAI-style endpoint and doesn't support rolling log-likelihood evaluations used by perplexity tasks.