LLMOps: Gateways, Observability and Guardrails

Sandcastle is a TypeScript library for developers building automated coding workflows in scripts, CI pipelines, or custom tools. It coordinates sandboxed AI coding agents and manages where their Git changes land.

llama-swap is a local AI proxy for people who want several models available to their apps without keeping them all loaded. It starts the server a request needs and replaces the active server when another model is requested.

Chidori is an AI agent framework for developers building long-running workflows that need crash recovery or human approval. Agents use ordinary async TypeScript, with saved runs that developers can revisit for debugging and testing.

Agno is a framework and runtime for developers building customer-facing or internal AI agents. It combines agent development, service hosting and a web interface for managing runs on infrastructure you control.

Presidio is a self-hosted framework for developers and organizations that need to remove personally identifiable information (PII) from data. It detects sensitive details such as names, credit card numbers and phone numbers, then applies configurable de-identification.

MCPJam helps MCP server developers check how their tools and apps behave across AI clients before deployment. It combines hands-on debugging with repeatable tests for differences between ChatGPT, Claude, Cursor, and Copilot.

sandboxd is an AI app builder for teams hosting coding agents, playgrounds or app previews on their own server. It turns prompts into running apps, each with an isolated workspace and its own preview URL.

OGX, formerly Llama Stack, is an AI application server for developers building chat apps, document search and AI agents. It brings model access and the services those applications need into a server you can host yourself.

AX is a self-hosted AI agent orchestrator for teams running autonomous tasks on Kubernetes. It manages agents that retain state and need access to models, repositories and tools while they work.

RD Agent is a self-hosted AI research agent for data scientists, researchers and quantitative developers. It runs experiments on a Linux machine or server, while chat and embedding requests use providers through LiteLLM, including OpenAI and Azure OpenAI. Those remote configurations require API credentials and internet access; hosting the agent yourself doesn't move their inference onto your hardware.

Gondolin is a local sandbox for developers whose AI agents execute generated code. It separates that code from the host while letting developers control access to files, network destinations and API credentials.

Soup is a local LLM fine-tuning tool for developers and teams adapting models to their own datasets. It brings training, evaluation and export into one workflow that runs on your own hardware without requiring a cloud service.

Manifest LLM Gateway connects AI agents and apps to local models, cloud providers and subscriptions through a shared endpoint. It's for builders and teams who want to choose where requests go, handle provider failures and track inference usage.

MindWalk helps developers review coding-agent sessions by showing where an agent explored and changed a repository. It turns Claude Code, Codex and pi logs into a visual replay for comparing the agent's work with the task you assigned.

RocketRide is an AI pipeline builder and runtime for developers creating document processing, search, and agent workflows. Its visual editor sits in VS Code, alongside the application code that uses those workflows.

NVIDIA NeMo Agent Toolkit is an open-source Python library for developers building self-hosted servers that expose AI tools and workflows through MCP or A2A. It also measures and evaluates agent workflows across frameworks. Its introductory NVIDIA NIM workflow uses hosted inference and requires an NVIDIA account and API key.

Argilla is a data annotation and feedback tool for AI engineers and domain experts building training and evaluation datasets. It gives reviewers a shared interface for checking examples and contributing judgments that inform model development.

Workstream is a local developer dashboard for people who track code reviews and project work across several services. It runs on your machine and opens in a browser, with local SQLite storage. The project is open source under Apache License 2.0.

UQLM is a Python library for developers and researchers assessing the reliability of LLM answers. Its confidence scores estimate the likelihood of errors or hallucinations; they don't verify that an answer is factually correct.

Prompty is a prompt format and developer toolkit for building AI applications. It keeps prompt text, model settings, inputs and tool definitions in a portable Markdown file that developers can share across editors and language runtimes.

Harbor helps people experiment with local LLMs by connecting model backends, chat interfaces and supporting services. Its CLI and companion app manage those connections in a shared development environment.

Best Claude HUD keeps session and repository information visible for developers working in Claude Code's terminal interface. It helps you check the active model, available context and working tree without leaving the conversation.

Latitude is an AI agent monitoring platform for teams investigating production failures. It links recurring problems to the conversations and tool calls behind them, then gives coding agents the context to work on a fix.

Claude Code Agent Monitor gives developers one place to follow coding agents and inspect their work across Claude Code, Cursor and Codex. It keeps conversations and activity available for review after a session ends.

Seldon Core is an AI model serving framework for teams running machine learning models and LLM applications on Kubernetes. It manages individual models and applications with multiple processing stages.

FastChat is a self-hosted LLM platform for developers and researchers who want to serve chat models on their own hardware and compare their responses. It powers Chatbot Arena and includes a locally hosted side-by-side comparison interface. The Python code is open source under Apache 2.0; model licenses apply separately, including Llama's license for Vicuna.

Lunary is an LLM observability and prompt management platform for teams building AI agents and chatbots. It helps developers investigate production errors and review how people interact with their applications.

Distilabel is a Python framework for engineers building synthetic datasets and model-generated feedback with local LLM backends such as Ollama, llama.cpp and Hugging Face Transformers. It also connects to cloud services, including OpenAI and Anthropic; requests through those integrations go to the remote provider. Where inference runs depends on the backend you choose.

Tracely helps developers trace AI agents and turn production failures into tests that catch the same problem before another release. It uses recorded runs as test cases, so teams can build regression coverage from failures their agents have actually encountered.

lm-evaluation-harness is a Python framework for researchers and developers comparing language models on shared benchmarks. Public prompts help make results comparable across model checkpoints and research papers.

LMMS Eval is a Python toolkit for researchers and model teams comparing AI models on text, image, video and audio tasks. It uses a shared evaluation pipeline to reduce differences caused by separate benchmark implementations.

NVIDIA NemoClaw is a self-hosted stack for people who want to run AI agents with controls over network access and credentials. It supports OpenClaw, Hermes and LangChain Deep Agents Code inside NVIDIA OpenShell sandboxes.

Ragas is a Python library for developers evaluating LLM applications and retrieval-augmented generation systems. Its non-LLM metrics compute text comparisons in your Python environment without a model, including string similarity, BLEU and ROUGE. Its model-based quickstart uses OpenAI's GPT-4o and an API key, sending evaluation inputs to OpenAI's cloud service. Local text scoring and cloud model judgments are separate evaluation options.

LocalAI is a self-hosted AI runtime for developers and teams that want to run models on their own hardware. It brings language, speech, vision and media generation into one server that existing AI applications can connect to.