DeepEval, Inspect AI and Azure: agent evaluation tutorial

This tutorial compares DeepEval, Inspect AI and the Azure AI Evaluation SDK through Python examples. The speaker argues that AI agent evaluation should inspect the full trajectory: a correct final answer can conceal poor tool selection, incorrect arguments or an inefficient sequence of actions. The theory section covers model-based judges, human calibration, benchmarks, latency and token costs.

The DeepEval examples use custom correctness criteria, expected tool calls and a mock travel agent. Trace decorators attach metrics to individual functions. Separate test scripts show how to bring evaluations into a pytest-based CI/CD pipeline. These examples use mock data, which the speaker explains viewers can replace with their own agent inputs and responses.

Inspect AI organizes evaluations around datasets, solvers and scorers. Its examples cover tool-assisted questions, multiple tools in one request and a ReAct loop. A custom refusal scorer checks harmful requests against explicit criteria, and the log viewer exposes inputs, outputs and evaluation details. The speaker describes Inspect AI as open source with an MIT license.

The Azure demonstration covers task adherence, tool-call accuracy and JSONL batch evaluation. It uses an Azure subscription and model deployment configuration. The speaker says the library is in preview and notes binary scoring in the SDK used. Azure safety evaluators are not tested because the demo lacks the required AI Foundry project connection.