DeepEval

An open-source Python LLM evaluation framework with local test runs, explainable scores and custom criteria for agents, RAG pipelines and chatbots.

DeepEval is an LLM evaluation framework for developers testing AI agents, RAG pipelines and chatbots. It helps compare prompts and models and catch quality regressions within Python tests or CI/CD.

Agent and retrieval tests: Evaluate whole applications, complete agent runs or individual steps, including tool calls, retrieval and handoffs, with scores that include explanations.
Quality metrics: Check task completion, hallucination, answer relevance and grounding in retrieved context, alongside conversation, image and audio evaluations.
Custom scoring: Define criteria in plain English with G-Eval or build conditional scoring with DAG metrics.
Test data generation: Create synthetic test cases from a knowledge base and simulate conversations with different user personas.

DeepEval is open source under Apache 2.0. Tests run in your own environment through Pytest, Python scripts or notebooks. Integrations include LangChain, LangGraph, CrewAI, LlamaIndex and OpenAI Agents.

Some metrics use local NLP models; LLM judges use your chosen model or provider. Cloud judges such as OpenAI, Claude and Gemini require sending evaluation content to those services.

Confident AI is a separate hosted platform for shared reports, dataset management and production monitoring. Connecting DeepEval to it sends traces and evaluation results to a shared cloud workspace.

Datasets and Synthetic Data LLM Evaluation and Testing LLM Observability and Tracing LLM tracing Multimodal input