LLMOps: Gateways, Observability and Guardrails

Omit Redact is beta software for offline PII redaction on Windows 10 and 11, built for people preparing sensitive files to share in legal, healthcare, financial and public sector work. Local language models detect names, organisations and places in seven languages, alongside pattern checks for IDs and account numbers. Files and processing stay on your device, with no account, cloud service or telemetry.

OrcaReplay is a local AI agent debugger for developers investigating failed coding runs or testing how another model handles the same task. It captures what happened beyond the chat transcript, including failed commands and changed files.

ms-swift is a Python framework for developers and researchers training and deploying language and multimodal models on their own hardware. It covers model adaptation through evaluation and serving, with support for Qwen3, DeepSeek-R1, Llama4, Mistral, Qwen3-VL and InternVL3.5.

Mirascope is an open-source development library for building AI applications with local models or cloud providers. Its documented local backends include Ollama, vLLM and LM Studio, plus direct inference through MLX on Apple Silicon Macs. Developers can use a shared model interface while keeping inference on their own hardware or sending requests to OpenAI, Anthropic or Google.

Headroom reduces the context that LLM apps and coding agents send to a model. It's for developers whose prompts fill up with repetitive tool results, logs, file contents and retrieved documents. Compression runs locally, but model requests still go to the chosen LLM provider.

LLM Sandbox is a Python library for developers whose AI agents or applications need to execute generated code. It puts execution in containers on infrastructure you control, with support for local deployments and Kubernetes clusters.

nono is a security runtime for people running terminal AI agents on their own machines or servers. It uses operating system boundaries to restrict an agent's access to files, network services and credentials.

LitGPT is a Python toolkit for developers and researchers who want to train and deploy language models on their own hardware. Its directly editable model implementations suit custom applications and research experiments.

Ouroboros is a locally installed workflow engine for developers who want to define requirements and check an AI coding agent's output. It runs through a terminal CLI, a Claude Code plugin or an MCP server. Your chosen coding runtime supplies the agent and model access. When used with Claude through Claude Code's standard cloud connection, model requests go to that external service.

AQBot is a desktop AI workspace for people who want model chat, coding agents and document search in one app. It keeps application data and user files locally, while requests to cloud models go to their providers.

AgentField is a self-hosted AI agent platform for developers building services that need to coordinate tasks under shared access rules. It combines execution, identity and audit records in infrastructure you control.

Reverify helps developers and reverse engineers check AI-generated claims against actual binary files and code behavior. It returns verified, refuted or inconclusive verdicts with evidence, so an assistant's proposed explanation can be checked before you rely on it.

Zerobox is a local process sandbox for developers running AI agents, generated code, builds or tests with restricted access to their machine. It builds on OpenAI Codex's sandbox runtime and can contain a whole agent or individual tool calls.

Agentic Security is an LLM vulnerability scanner for developers, researchers and security teams testing AI models and agent workflows. It probes how systems respond to hostile inputs and attempts to bypass their safety controls.

TruLens helps developers assess AI agents and retrieval-augmented generation (RAG) apps, then trace poor results to the steps that caused them. It connects evaluation scores with execution traces so teams can compare changes in quality, latency and cost.

vLLM Production Stack is an open source LLM serving stack for teams running models on Kubernetes GPU clusters. It lets applications move from a single vLLM instance to a distributed deployment without changing application code.

MaxText is an LLM training library for researchers and engineers building models from scratch or adapting existing ones. It provides a reference implementation teams can modify for their own model designs, with training that scales from a single host to large clusters.

TrustLLM is a research toolkit for teams comparing language models beyond answer quality. It evaluates local LLMs and models accessed through APIs for behaviors such as hallucination, harmful responses, bias, and information leakage.

Codex Router runs on your computer and connects the Codex App and CLI to external model providers or your own self-hosted endpoints. It's for developers who want a different model within their existing coding workflow. Cloud routes send model requests to remote services and need internet access; paid providers still charge for access or usage.

Claude Code Router is a local gateway for developers using coding agents such as Claude Code, Codex and OpenCode. It keeps provider selection and request routing in one app, while the selected cloud provider processes model requests remotely. Those requests leave your machine.

SillyTavern is a locally installed interface for AI hobbyists who want control over prompts and access to text, image and voice models in one place. It's an independently developed fork of TavernAI and connects to a separate model backend or API provider.

Coze Loop helps developers test and monitor AI agents, with tools for investigating failed runs and comparing prompt changes. Its self-hosted edition brings prompt development, evaluation, and execution traces into one browser interface.

MCP Context Forge is a gateway and registry for teams connecting AI clients to multiple tools, agents and services. It gives them one place to discover available services, control access and inspect calls.

OpenCompass is an LLM evaluation framework for researchers and developers comparing models for their applications. It lets you assess models on your own hardware alongside cloud APIs in the same experiment.

Noi is an AI browser for people who work across AI chats and want to keep separate tasks organized. It combines multiple workspaces with reusable prompts and access to local tools.

1Panel is a Linux server management panel for people hosting AI software alongside websites and other applications. Its browser interface brings model and agent management together with everyday server administration.

aichat is a terminal AI assistant for people who want to discuss files, generate shell commands and use model-powered tools from the command line. It connects to Ollama, OpenAI, Claude, Gemini and OpenAI-compatible API providers.

Arize Phoenix helps developers investigate failures in AI agents and LLM applications and compare proposed fixes. It connects recorded application activity with evaluations and repeatable experiments.

Apache Burr is a self-hosted Python framework for developers building AI agents, chatbots, and other applications with branching decisions. It models application logic as a state machine, so developers can see which action ran, what changed, and why execution took a particular path. Its library and monitoring UI run on your own infrastructure, and its open-source code uses the Apache 2.0 license.

AgentGUI gives researchers and developers a visual workspace for supervising long-running AI agents. Its pixel-art office represents tasks as desks, so you can follow several jobs and intervene while they run.

LLMux is a self-hosted AI gateway for developers who want to use local models and cloud providers through one interface while controlling where requests go. It blocks remote inference by default and logs permitted calls that leave the machine.

Paddler lets product and infrastructure teams serve AI models on their own servers and spread requests across multiple machines. It's built for applications that need text generation or embeddings while keeping inference within infrastructure the team controls.

Claude DevTools is a local debugger for developers who want to understand what happened during a Claude Code session. It reads saved logs from terminal and IDE sessions without modifying Claude Code.

Torrix is a self-hosted LLM monitoring tool for teams investigating AI spending and failed requests. It keeps request records in your infrastructure and brings usage analytics and individual traces into one dashboard.