Tracely

Tracely helps developers trace AI agents and turn production failures into tests that catch the same problem before another release. It uses recorded runs as test cases, so teams can build regression coverage from failures their agents have actually encountered.

• Trace inspection: Follow conversations, agent handoffs, model calls and tool use alongside inputs, outputs, token costs and latency.
• Failure detection: Structural checks and LLM judges evaluate incoming runs, while failure clustering groups related problems into issues with suggested fixes and evaluators.
• Recorded regression tests: Preserve a failing run with its inputs and recorded tool and model responses, then replay it deterministically offline without API keys or fresh model calls.
• Pull request gates: Run regression cases in CI and report failures through GitHub commit statuses and pull request comments, with checks that can block a merge.
• Judge review and editor access: Compare evaluator verdicts with human labels, and let Claude Code or Cursor inspect traces, investigate clusters and create evaluators through MCP.

Tracely is free, open source under the MIT license, and available as a self-hosted Docker Compose deployment or a hosted service. It accepts OTLP traces and integrates with OpenAI, Anthropic, LangChain, LangGraph and LiteLLM.

Self-hosting keeps trace storage on your infrastructure, but LLM judges require your own OpenRouter key and send evaluation requests to that external service. Structural evaluators work without a model key. Recorded replay works offline; live replay makes real tool and model calls.