Phoenix and OpenInference: debug agents with traces and eval

This tutorial uses Phoenix and OpenInference to inspect an AI agent that approves or denies procurement requests. The example runs on Microsoft Agent Framework with Azure OpenAI on Microsoft Foundry. It demonstrates observability and evaluation around a cloud model workflow, rather than local inference.

The speaker adds an OpenTelemetry trace provider and a span processor that converts framework telemetry to OpenInference conventions. Traces go to Phoenix, described as a free, open source observability and evaluation system. The walkthrough follows prompts, tool inputs and outputs, token usage, timing and cost through a purchase approval. It also describes adding a processor to remove client names before exporting traces.

For correctness checks, the tutorial extracts trace evidence and passes it to an LLM judge. The judge labels each recommendation supported or unsupported using only the tool outputs. One failure involves a $6,800 purchase approved without evidence of the required manager approval.

The initial ten-case evaluation passes four cases. The speaker reports nine passes after switching to GPT 5.4 mini without changing the prompts or tools, while warning that newer models do not guarantee better results. The closing section describes using Phoenix skills with a coding assistant to propose changes and rerun experiments in a sandbox. These scores describe the demonstrated runs, not general reliability benchmarks.