Open-Source LLM Evaluation Tools

MCPJam helps MCP server developers check how their tools and apps behave across AI clients before deployment. It combines hands-on debugging with repeatable tests for differences between ChatGPT, Claude, Cursor, and Copilot.

RD Agent is a self-hosted AI research agent for data scientists, researchers and quantitative developers. It runs experiments on a Linux machine or server, while chat and embedding requests use providers through LiteLLM, including OpenAI and Azure OpenAI. Those remote configurations require API credentials and internet access; hosting the agent yourself doesn't move their inference onto your hardware.

Soup is a local LLM fine-tuning tool for developers and teams adapting models to their own datasets. It brings training, evaluation and export into one workflow that runs on your own hardware without requiring a cloud service.

MindWalk helps developers review coding-agent sessions by showing where an agent explored and changed a repository. It turns Claude Code, Codex and pi logs into a visual replay for comparing the agent's work with the task you assigned.

NVIDIA NeMo Agent Toolkit is an open-source Python library for developers building self-hosted servers that expose AI tools and workflows through MCP or A2A. It also measures and evaluates agent workflows across frameworks. Its introductory NVIDIA NIM workflow uses hosted inference and requires an NVIDIA account and API key.

Argilla is a data annotation and feedback tool for AI engineers and domain experts building training and evaluation datasets. It gives reviewers a shared interface for checking examples and contributing judgments that inform model development.

UQLM is a Python library for developers and researchers assessing the reliability of LLM answers. Its confidence scores estimate the likelihood of errors or hallucinations; they don't verify that an answer is factually correct.

Harbor helps people experiment with local LLMs by connecting model backends, chat interfaces and supporting services. Its CLI and companion app manage those connections in a shared development environment.

Latitude is an AI agent monitoring platform for teams investigating production failures. It links recurring problems to the conversations and tool calls behind them, then gives coding agents the context to work on a fix.

FastChat is a self-hosted LLM platform for developers and researchers who want to serve chat models on their own hardware and compare their responses. It powers Chatbot Arena and includes a locally hosted side-by-side comparison interface. The Python code is open source under Apache 2.0; model licenses apply separately, including Llama's license for Vicuna.

Lunary is an LLM observability and prompt management platform for teams building AI agents and chatbots. It helps developers investigate production errors and review how people interact with their applications.

Distilabel is a Python framework for engineers building synthetic datasets and model-generated feedback with local LLM backends such as Ollama, llama.cpp and Hugging Face Transformers. It also connects to cloud services, including OpenAI and Anthropic; requests through those integrations go to the remote provider. Where inference runs depends on the backend you choose.

Tracely helps developers trace AI agents and turn production failures into tests that catch the same problem before another release. It uses recorded runs as test cases, so teams can build regression coverage from failures their agents have actually encountered.

lm-evaluation-harness is a Python framework for researchers and developers comparing language models on shared benchmarks. Public prompts help make results comparable across model checkpoints and research papers.

LMMS Eval is a Python toolkit for researchers and model teams comparing AI models on text, image, video and audio tasks. It uses a shared evaluation pipeline to reduce differences caused by separate benchmark implementations.

Ragas is a Python library for developers evaluating LLM applications and retrieval-augmented generation systems. Its non-LLM metrics compute text comparisons in your Python environment without a model, including string similarity, BLEU and ROUGE. Its model-based quickstart uses OpenAI's GPT-4o and an API key, sending evaluation inputs to OpenAI's cloud service. Local text scoring and cloud model judgments are separate evaluation options.

Giskard Hub is an AI agent testing platform available hosted or on-premise for organizations that need testing within their own environment. Developers can also run its open-source Python library locally, but the library's LLM judges and attack generators require a model provider and API key, with integrations for OpenAI and Anthropic. Local execution doesn't make those evaluations offline.

KADATH is a self-hosted AI agent runtime for developers and researchers comparing agents against a measurable goal. It runs successive generations of agents and selects stronger approaches, but model calls require OpenAI's cloud API, an API key and internet access.

Mastra is a TypeScript framework for developers building self-hosted AI agents and applications with local models through LM Studio. It supports gpt-oss, Qwen3 and DeepSeek through an OpenAI-compatible model server. The framework can also call OpenAI, Anthropic and Gemini, which send model requests to those remote providers.

Prompt flow is a Python toolkit for developers who want to run and evaluate LLM application workflows in their own development environment. Its chatbot examples call OpenAI or Azure OpenAI remotely using API credentials. Workflow execution and evaluation can run locally, while those model requests leave the machine.

Dingo combines rule checks, LLM judges and agent-based evaluation to assess AI training data, fine-tuning datasets and production systems. It's an open-source Python tool under Apache 2.0 for data engineers, ML practitioners and researchers who need to find data problems and compare model or application quality. Evaluation can run locally or across a Spark cluster.

VoltAgent is a TypeScript AI agent framework for developers building assistants and automated workflows with local models through Ollama. Its VoltOps console can be self-hosted or used as a cloud service. Model execution depends on the provider: Ollama supports local inference, while OpenAI, Anthropic and Google connections send requests to those remote services.

FinGPT is a financial LLM framework for researchers and developers building tools that interpret market news and company information. It includes trained models, financial datasets, and tools for adapting models to specific finance tasks.

Molmo is Ai2's family of vision-language models for working with images and text on your own hardware. It's aimed at developers and researchers who want to train models or evaluate them on visual tasks.

EvalScope is a Python evaluation framework for teams comparing AI models or assessing a model service before deployment. It connects scores to the prompts, predictions and reviews behind them so you can examine individual results.

LightEval is a Python toolkit from Hugging Face for developers and researchers comparing LLMs or investigating where their responses fail. It evaluates models already loaded in memory as well as models accessed through a service.

Gauntlet Loop is an AI agent skill for people who want generated websites, writing or software judged against a specific example. It writes a prompt that asks an agent to build, compare and revise work against a reference you choose.

Ringer coordinates parallel AI coding agents on your machine. It's for developers who want to divide implementation work among workers and judge the results with executable checks.

NVIDIA Object Oriented Agents (NOOA) is a Python framework for developers building AI agents in their own projects. It supports local models through Ollama and vLLM via LiteLLM, with Qwen3 examples for both backends. It also supports Anthropic and OpenAI models, which send requests to those remote providers.

RankLLM is a Python toolkit for search researchers and developers who want to compare how AI models reorder retrieved documents by relevance. Its main focus is listwise reranking, where a model considers several candidates together. It supports local inference and optional cloud APIs, and it's open source under the Apache 2.0 license.

Opik is an LLM observability and evaluation platform for teams building AI agents and retrieval-based applications. It helps developers inspect failures and compare behavior across development, testing and production.

Inspect AI is a Python framework for researchers and developers evaluating language models and AI agents. It measures coding, reasoning, knowledge, behavior and multimodal understanding, including tasks where agents must take actions to succeed.

Rasa Open Source is a Python framework for developers building text and voice assistants on their own servers. It provides natural-language understanding, dialogue management and integrations with messaging channels.

GuideLLM benchmarks LLM deployments so engineering and ML teams can assess capacity and response times under expected traffic. It can test self-hosted servers through OpenAI-compatible or vLLM-native interfaces.