Evidently

An open-source Python framework evaluates LLMs, RAG and predictive models with custom metrics and a self-hosted monitoring dashboard.

Evidently helps AI teams check output quality and catch regressions in chatbots, RAG applications, AI agents and predictive models. Its Python library handles text and tabular data for development evaluations and production monitoring.

LLM and RAG evaluation: Check factuality, retrieval relevance, format compliance, PII and toxicity with rules, classifiers or LLM judges, and define custom evaluations for your application.
Data and model checks: Measure classification, regression and ranking performance, detect missing values and duplicates, and compare data distributions for drift.
Regression tests: Turn evaluation reports into test suites with pass/fail conditions for CI/CD checks and data validation, including conditions derived from a reference dataset.
Reports and monitoring: Explore interactive reports, export results as HTML or JSON, and track metrics and test outcomes over time in a dashboard.

The library and self-hosted monitoring UI are open source under Apache 2.0. You can host the dashboard on your own server; Evidently Cloud is a separate hosted service. A Python interface lets you add custom metrics.

LLM Evaluation and Testing LLM Observability and Tracing Web