Open-Source LLM Observability and Tracing

Rasa Open Source is a Python framework for developers building text and voice assistants on their own servers. It provides natural-language understanding, dialogue management and integrations with messaging channels.

PenguinHarness builds runnable AI agent applications from plain-language requirements on your computer or server. It's for developers who want agents to handle app creation, evaluation, tuning and deployment.

Agent Console helps developers and teams monitor Claude Code and Codex sessions through a local dashboard. It reads the agents' transcripts on your machines so you can compare usage and spot sessions that need attention.

Pezzo helps developers and teams manage prompts and investigate problems in LLM applications. It brings prompt editing and monitoring into one place for projects where several people contribute to AI behavior.

Hermes HUD is a browser dashboard for people who already run the Hermes AI agent and want to inspect its activity and persistent memory. It requires a running agent and reads its local data.

MartinLoop runs around coding agents to keep software jobs within agreed boundaries and check their results. It's for developers and engineering teams who want to delegate longer tasks while retaining limits, recovery information and a record of what happened.

Helicone helps developers inspect and debug LLM applications, including agents, chatbots, and document processing pipelines. It combines request monitoring with a gateway for routing calls across AI providers.

APIPark is an AI gateway and API developer portal for teams managing model services and business APIs. It gives applications a common request format and lets teams share APIs with controlled access.

Pi Extensible Workflows is an extension for developers who want repeatable processes involving several AI agents. It coordinates resumable workflows inside the Pi host environment and lets you inspect work while it runs.

Codenotch is a desktop monitor for people who use AI coding assistants and local LLMs. A small display at a screen edge keeps usage readings and session states visible while you work in other apps.

Agent Router is an open source AI gateway for teams connecting agents to model providers and MCP tools. It gives applications a shared entry point while platform teams manage access and routing.

Bifrost is an AI gateway for developers and teams building applications that use local LLMs or multiple cloud providers. It gives those applications a shared API for model access and a central place to manage usage.

OrcaReplay is a local AI agent debugger for developers investigating failed coding runs or testing how another model handles the same task. It captures what happened beyond the chat transcript, including failed commands and changed files.

Mirascope is an open-source development library for building AI applications with local models or cloud providers. Its documented local backends include Ollama, vLLM and LM Studio, plus direct inference through MLX on Apple Silicon Macs. Developers can use a shared model interface while keeping inference on their own hardware or sending requests to OpenAI, Anthropic or Google.

AgentField is a self-hosted AI agent platform for developers building services that need to coordinate tasks under shared access rules. It combines execution, identity and audit records in infrastructure you control.

TruLens helps developers assess AI agents and retrieval-augmented generation (RAG) apps, then trace poor results to the steps that caused them. It connects evaluation scores with execution traces so teams can compare changes in quality, latency and cost.

vLLM Production Stack is an open source LLM serving stack for teams running models on Kubernetes GPU clusters. It lets applications move from a single vLLM instance to a distributed deployment without changing application code.

Claude Code Router is a local gateway for developers using coding agents such as Claude Code, Codex and OpenCode. It keeps provider selection and request routing in one app, while the selected cloud provider processes model requests remotely. Those requests leave your machine.

Coze Loop helps developers test and monitor AI agents, with tools for investigating failed runs and comparing prompt changes. Its self-hosted edition brings prompt development, evaluation, and execution traces into one browser interface.

MCP Context Forge is a gateway and registry for teams connecting AI clients to multiple tools, agents and services. It gives them one place to discover available services, control access and inspect calls.

Arize Phoenix helps developers investigate failures in AI agents and LLM applications and compare proposed fixes. It connects recorded application activity with evaluations and repeatable experiments.

Apache Burr is a self-hosted Python framework for developers building AI agents, chatbots, and other applications with branching decisions. It models application logic as a state machine, so developers can see which action ran, what changed, and why execution took a particular path. Its library and monitoring UI run on your own infrastructure, and its open-source code uses the Apache 2.0 license.

AgentGUI gives researchers and developers a visual workspace for supervising long-running AI agents. Its pixel-art office represents tasks as desks, so you can follow several jobs and intervene while they run.

Claude DevTools is a local debugger for developers who want to understand what happened during a Claude Code session. It reads saved logs from terminal and IDE sessions without modifying Claude Code.

Torrix is a self-hosted LLM monitoring tool for teams investigating AI spending and failed requests. It keeps request records in your infrastructure and brings usage analytics and individual traces into one dashboard.

Evidently helps AI teams check output quality and catch regressions in chatbots, RAG applications, AI agents and predictive models. Its Python library handles text and tabular data for development evaluations and production monitoring.

Atlas is a desktop workspace for developers who use coding agents across projects and want to keep the context behind their code changes. It brings agent conversations, code review and project history into one window.

Better Harness helps developers and teams assess how coding agents carry out work. It examines the workflow around a code change to find unclear goals, missing validation and delivery checks that a diff review can miss.

claudectl manages multiple Claude Code sessions for developers who need to track parallel work and intervene when an agent gets stuck. Its terminal dashboard brings session activity, approvals and spending into one view.

DeepEval is an LLM evaluation framework for developers testing AI agents, RAG pipelines and chatbots. It helps compare prompts and models and catch quality regressions within Python tests or CI/CD.

OpenLIT helps developers investigate failures and assess the quality of existing AI applications and coding agents. It connects model requests with the tools, retrieval and agent steps involved in producing an answer.

Langfuse helps teams debug LLM applications and AI agents, then test changes against production examples. It brings traces, evaluations and prompt revisions into one place.

TensorZero is a self-hosted platform for developers who need to compare and improve models in LLM applications. The project is archived and no longer maintained.