Langfuse: improving agents with traces and evals

Marc Klingen, co-founder of Langfuse, explains an open source reference stack for improving an AI agent through production feedback. The workflow connects online tracing and monitoring with offline datasets, experiments and evaluations. Production failures become test cases, proposed changes face those tests, and later user feedback informs another round.

Klingen describes teams using AI to propose implementation fixes and, increasingly, to maintain datasets and evaluation criteria. He argues that people should still review those criteria and set goals: an automated loop can overfit to a dataset or spend effort on failures outside the application's intended scope. User edits, approvals and requests for changes can supply feedback without separate manual labeling.

The demo applies this approach to Langfuse's changelog writer. A coding assistant uses Langfuse agent skills to investigate frequent human edits and identifies internal jargon in customer documentation. It proposes revised datasets and evaluators, changes the writer's instructions, then compares a V2 candidate with the V1 baseline. Klingen reports better customer-facing language with unchanged accuracy and format compliance. The writer raises pull requests rather than publishing directly to production.

The talk closes with the data requirements behind these loops. Klingen argues for retaining detailed traces and owning the data layer, since automated analysis can query far more historical data than a person can review.