Langfuse tutorial: agent evals and CI quality gates

Annabell Schäfer and Doneyli De Jesus explain how to evaluate an AI agent when a successful request does not guarantee a correct outcome. Their example is a support assistant that claims to issue a refund without calling the required tool. They present Langfuse as an open source platform for tracing execution and measuring output quality.

The banking assistant demo records model calls and tool activity. A configured masking function redacts sensitive information before traces reach Langfuse. The presenters build evaluation datasets through CSV uploads and selected production traces, then use annotation queues to collect domain-expert feedback. They recommend deriving failure categories from those comments and combining LLM-as-judge evaluators with deterministic code checks. They also explain how to compare judge scores with human labels.

A prompt change triggers a GitHub CI job that runs an experiment against a dataset and checks score thresholds. The demo leaves deployment decisions for review rather than automatically merging a passing change. Dashboards help narrow investigations to recurring failures, such as unresolved disputes. The presenters state that the demo uses fictional data.

The closing Q&A covers a self-hosted deployment option on Amazon infrastructure and identifies ClickHouse as a required data store. This is an evaluation workflow, rather than a guide to running models on personal hardware.