Ragas

Python LLM evaluation library with local text-comparison metrics, datasets and tracking. Its model-based quickstart uses OpenAI cloud scoring. Apache 2.0.

Ragas is a Python library for developers evaluating LLM applications and retrieval-augmented generation systems. Its non-LLM metrics compute text comparisons in your Python environment without a model, including string similarity, BLEU and ROUGE. Its model-based quickstart uses OpenAI's GPT-4o and an API key, sending evaluation inputs to OpenAI's cloud service. Local text scoring and cloud model judgments are separate evaluation options.

The library combines traditional metrics with LLM judgments. Developers can use its existing metrics or define criteria for their application, such as checking summary accuracy. Model-based scoring can return both a result and the reason for the judgment, so developers can examine how an evaluator assessed an output.

Built-in dataset management and result tracking support comparisons across evaluation runs. Ragas also generates test datasets for teams that don't already have evaluation examples. LangChain and LlamaIndex integrations connect it to application workflows, and it also supports integrations with observability tools.

Ragas is open source under Apache 2.0. The library collects anonymized usage analytics and provides an opt-out. Its collection code is open source, and published aggregate analytics exclude personal and company-identifying information.

LLM Libraries and SDKs Datasets and Synthetic Data LLM Evaluation and Testing