Langfuse tutorial: trace and evaluate support agents

Annabell Schäfer and Lotte Verheyden demonstrate Langfuse tracing and monitoring through Specs, a sample phone-support AI agent. The presenters describe Langfuse as open source and model agnostic. This workshop uses a Langfuse account and OpenAI API calls; it does not demonstrate local model inference.

The sample application uses TypeScript and the OpenAI Agents SDK. Its traces contain agent observations, model generations and tool calls. The workshop starts with an already instrumented application, then focuses on choosing signals that help explain failed support conversations.

A judge evaluator checks user disagreement against conversation history and the latest user message. A Python evaluator detects all-caps messages as a possible frustration signal; the presenter also notes support for TypeScript evaluators. A separate scope check compares the system prompt with the user's request. Explicit JSON path mappings determine which messages each evaluator receives.

The presenters seed example production traces and use the Langfuse CLI and skill with Codex to investigate them. The reports identify retrieval failures, retry overhead and gaps in evaluation coverage. A proposed change removes the search-help-library tool. The closing discussion explains how representative failures can become datasets for offline experiments and regression checks, with human review helping catch cases that automatic evaluators miss.