Phoenix and Claude: agent evals tutorial with CI

Laurie Voss walks through an evaluation loop for a financial analysis AI agent built with the Claude Agent SDK. The TypeScript workshop requires Node 22 or later and an Anthropic API key. Phoenix runs on the participant's laptop; Voss also describes running it on a server for a team. This self-hosted tracing setup still uses Claude API calls and live web searches.

The tutorial starts with traces of research and report-writing turns, then adds a deterministic check for requested stock tickers. Voss examines a report whose analyst percentages total 116%, tracing the error to figures combined from different search results. He contrasts a correctness judge that rejects fresh financial data with a faithfulness judge given the research notes and raw search results.

A custom actionability judge checks for investment recommendations. Human labels, development and test splits, and agreement metrics help assess the judge itself. In Voss's experiments, prompt changes improve actionability but leave faithfulness poor. His comparison of Claude Haiku 4.5 and 5.5 reports better faithfulness with the latter, alongside a tendency to recommend holding.

The final section reuses the evaluators in npm test and describes GitHub Actions with Phoenix as a service container. The example requires ticker checks to pass and an average actionability score of at least 0.8. Voss recommends smaller regression sets for commits and continued evaluation of sampled production traffic.