Dingo

Dingo combines rule checks, LLM judges and agent-based evaluation to assess AI training data, fine-tuning datasets and production systems. It's an open-source Python tool under Apache 2.0 for data engineers, ML practitioners and researchers who need to find data problems and compare model or application quality. Evaluation can run locally or across a Spark cluster.

Rule checks catch duplicates, empty content, formatting problems and personal information. Model judges assess meaning, completeness and relevance, with support for local Llama3 and Qwen models as well as cloud services such as OpenAI, DeepSeek and Kimi. Local evaluation can use local files and models; choosing a cloud model sends evaluation requests to that service. Text and image checks include vision-language models such as InternVL and Gemini.

For RAG systems, Dingo measures how closely answers follow their context and query, alongside retrieval precision, recall and relevance. It includes HHEM-2.1-Open for consistency checks. Agent trace evaluation examines task completion, tool use and adherence to plans, with monitoring for recovery errors, latency and token use.

Data can come from SQL databases, HuggingFace datasets or S3 storage. Different fields can have separate checks and reports, and custom rules, prompts and models extend the evaluation. JSON and visual reports show scores and issue distributions. A Python SDK and MCP server connect evaluation to applications and editors such as Cursor. A separate hosted SaaS offering provides a web interface, access controls and interactive trend reports.

Agent judges that use web-search tools need network access and the configured search service; those workflows are separate from local rule checks and local-model evaluation.