Evidently
Open-source Python framework for LLM evaluation, RAG testing and ML monitoring, with a self-hosted dashboard and an Apache 2.0 license.
Evidently is an Apache 2.0 Python library for evaluating, testing and monitoring ML and LLM systems. It works with tabular and text data, including predictive models and RAG applications. You can run one-off evaluations or self-host its open-source monitoring UI. Evidently Cloud is a separate hosted service.
Reports compute data quality, distribution drift and model-quality metrics. Text and LLM evaluations include sentiment, toxicity, semantic similarity and retrieval relevance, with rules, classifiers and model-based judges. A Python interface supports custom metrics.
Adding pass/fail conditions turns a report into a test suite for regression checks, CI/CD or data validation. Reports can be viewed interactively or exported as HTML, JSON or Python dictionaries. The monitoring UI displays metrics and test results over time.
Local deployment of the dashboard does not determine where a model-based judge runs. Choose evaluator models and their connection settings to match your data-handling requirements.
