DeepEval and LangChain: build and evaluate a refund agent

This tutorial builds a refund support AI agent with LangChain and evaluates it with DeepEval. The Python project uses an OpenAI API key; it does not demonstrate how to run models locally. The description lists Python 3.12+, while the speaker recommends versions 3.11 to 3.13 during setup with uv.

The agent reads purchase records from a Python dictionary, checks dates against a 30-day refund policy, calculates amounts for requested items, and generates simulated refund tickets. It does not transfer money. The speaker reserves actual payments for human review. Pydantic defines structured responses, and LangGraph's in-memory checkpointer supports conversations without permanent storage.

The evaluation workflow covers seven metrics: task completion, step efficiency, plan quality, plan adherence, tool correctness, argument correctness, and answer correctness through GEval. Test scenarios include unknown customers, unpurchased items, mixed refund eligibility, and requests for purchase information alone.

The speaker inspects traces and scoring explanations before changing tools and prompts. Some low scores appear to reflect missing evidence or unsuitable evaluation criteria. Disabling argument correctness as a pass/fail condition changes the reported pass rate; subsequent prompt edits address how the agent responds to invalid usernames.