LLMOps: Gateways, Observability and Guardrails

FlagEmbedding is a Python toolkit for developers building semantic search and retrieval-augmented generation (RAG). It runs embedding models to find relevant text and rerankers to refine the order of retrieved results.

DEEIX Chat is a self-hosted AI workspace for individuals and teams that want one place to use multiple model providers. It combines everyday chat with controls for organizations managing shared model access and usage.

PyRIT is an open-source Python framework for testing generative AI systems, with a self-hosted CoPyRIT browser interface. Its backend serves the interface locally, and Docker deployment is available. Targets and scorers use configured endpoints, including Ollama or cloud providers. The documented OpenAI configuration sends requests to OpenAI; hosting the testing interface locally does not make every model local.

ServiceStack AI Server is a self-hosted gateway for developers and teams whose applications use several AI providers. It runs in a Docker container and gives apps one API integration for local models, cloud LLMs and media processing. Teams can change providers without changing their client integrations.

Obelisk makes past coding agent sessions searchable for developers who need to recover decisions or revisit earlier work. A desktop browser and an agent skill give people and coding assistants access to the same local history.

MLflow helps developers and teams inspect AI applications, compare changes and track model experiments. It brings agent debugging and evaluation into a platform you can run locally or on your own servers.

Plano is a self-hosted AI gateway for developers building applications with multiple agents. It handles shared routing, orchestration and monitoring outside the agents, so developers don't need to build those functions into each one.

Omnigent coordinates AI agents for developers and teams who want to combine coding assistants, control their access, and supervise work in shared sessions. It runs locally or on a self-hosted server, with remote cloud sandboxes as another option.

Coding Tools MCP gives AI chat apps and coding agents access to a repository on your machine. It's for developers who want to use an existing MCP client to inspect code, make edits and check a running project.

PanGuard AI checks skills and MCP servers for threats before AI agents load them, then watches their actions at runtime. It's for developers and teams who want protection against prompt injection, poisoned tools and malicious skills.

TokensBurned tracks AI coding activity for developers who want usage statistics on their GitHub profile. Its client processes records locally, but the profile cards depend on a hosted service that receives aggregate data.

FlashRAG is an open source Python toolkit for researchers and developers comparing retrieval-augmented generation (RAG) methods or building their own. It combines implemented research methods with prepared benchmark datasets and reported results, so experiments don't have to start with a separate codebase for each approach. It uses the MIT license.

Guardrails AI is an open-source Python validation framework that runs inside your application or as a self-hosted service. It's for developers who need to check LLM inputs and responses before their apps use them. It provides a validation layer around model interactions; it doesn't supply local model inference.

Quicksand gives AI agents a Linux virtual machine for executing code and working with desktop apps on your own hardware. It's for developers who need a separate operating system for agent tasks, controlled through an async Python API.

Helix is a paid, self-hosted platform for engineering teams coordinating coding agents. Agents work on separate tasks in parallel, while the team shares a view of their progress and reviews their work.

llm-fw inspects model requests for developers and teams using AI tools, SDKs and agents. It blocks suspected prompt injection and removes outgoing secrets before forwarding approved traffic to the model endpoint.

PentAGI automates authorized penetration testing for security engineers, researchers and red teams. Its specialized AI agents turn a target and a plain-language goal into testing work you can follow through a web interface.

CascadeFlow is a library for developers who need to control AI agent spending and behavior during a run. It works inside the application, assessing model calls, tool calls and sub-agent handoffs before deciding how execution should continue.

LazyLLM is a Python framework for developers building applications with multiple AI agents. It brings application workflows, model training and evaluation together so teams can refine a prototype using task-specific data.

AgentSight is a local AI agent monitoring tool for developers investigating failed runs, slow steps or unexpected changes. It connects prompts and model calls to what agents actually do on the machine, including activity their own logs may miss.

Aiursoft Ollama Gateway gives teams controlled access to shared local LLM servers. It connects AI applications and coding assistants to Ollama or OpenAI-compatible backends through a central gateway.

AI Infra Guard is a self-hosted AI security testing platform for developers and security teams checking agent workflows, model responses and AI infrastructure. It combines vulnerability scans and adversarial prompt tests in a web interface.

Rivet AgentOS is a runtime for developers building applications where AI agents need a working environment for code and tools. It runs inside an existing backend process, with WebAssembly isolation instead of a separate virtual machine for each agent.

Laminar helps developers inspect AI agent runs, find recurring failures and test fixes. It connects production traces with debugging and evaluation tools.

LLMLingua is a Python library for developers building LLM applications that need to fit lengthy input into a model's context window or reduce token costs. It uses smaller models to remove less useful text before the prompt reaches the answering model.

MASFactory is a Python framework for developers building workflows in which several AI agents collaborate. It combines natural-language design, code authoring and visual editing so teams can define the work and inspect how agents carry it out.

Kiln is a desktop workbench for teams building and testing AI applications. Engineers, QA staff and subject matter experts can work with the same tasks and datasets, including contributing ratings and examples without writing code.

OpenRath is a Python runtime for developers building multi-agent workflows that need an inspectable record of their work. It keeps context and evidence together so engineering and research teams can trace how a result was produced.

Ferret is a local vision-language model for researchers studying how AI understands specific parts of an image. It supports questions about selected regions and can identify image regions in its answers. Its code and data are restricted to research use.

PasteGuard is a privacy proxy for teams that use AI with customer records, client files or project secrets. It runs locally or on your own server and checks requests before they reach a model provider.

OpenShell is a self-hosted runtime for teams that need autonomous AI agents to work with files and APIs under explicit access limits. It runs agents in isolated sandboxes and enforces policies at the kernel level.

LeanCTX helps developers control the information their AI coding tools receive. It runs locally between those tools and project files or command output, while your existing application keeps control of the model and workflow.

All-in-One Sandbox gives AI agents a place to browse websites, execute code and work with files. It's for developers building agents that need these tools together, with a shared filesystem instead of separate environments for each task.

nanochat is a toolkit for researchers and developers who want to train an LLM and inspect the pipeline themselves. Its small Python codebase covers the stages between preparing a tokenizer and chatting with the resulting model.