LLMOps: Gateways, Observability and Guardrails

LLMKube manages self-hosted LLM inference for teams and homelab owners who need several machines to serve models together. It brings Linux GPU servers and Apple Silicon Macs into the same Kubernetes-managed fleet.

Semantica is a Python toolkit for teams building AI systems that need shared context, traceable decisions and policy checks. It connects facts and decisions to their evidence in queryable knowledge graphs.

CliRelay is an AI gateway for teams that want to manage coding assistants through a shared service on their own server. It builds on CLIProxyAPI and brings subscription accounts and provider API keys under central control.

Opik is an LLM observability and evaluation platform for teams building AI agents and retrieval-based applications. It helps developers inspect failures and compare behavior across development, testing and production.

Chatless is a desktop AI chat client for people who want local models and cloud services in one app. It keeps conversation history on your computer and lets you ask questions about your own documents.

Agentgateway is an open-source proxy for platform teams connecting applications and agents to models, tools and other agents. It brings routing and access controls together across AI protocols and ordinary service traffic.

Inspect AI is a Python framework for researchers and developers evaluating language models and AI agents. It measures coding, reasoning, knowledge, behavior and multimodal understanding, including tasks where agents must take actions to succeed.

Rasa Open Source is a Python framework for developers building text and voice assistants on their own servers. It provides natural-language understanding, dialogue management and integrations with messaging channels.

GuideLLM benchmarks LLM deployments so engineering and ML teams can assess capacity and response times under expected traffic. It can test self-hosted servers through OpenAI-compatible or vLLM-native interfaces.

Fabric organizes reusable AI prompts by task for people who want repeatable ways to analyze content, write, or work with code. Its community-contributed prompts, called Patterns, provide a starting point without requiring you to write each prompt yourself.

Nox is a security scanner for developers building LLM applications, RAG systems and agents with tool access. It traces untrusted input into model calls and model output into dangerous operations, including paths across functions and files.

OpenSandbox gives developers isolated environments for AI agents that need to execute code or interact with browsers and developer tools. It runs on your own infrastructure, locally with Docker or on a Kubernetes cluster.

CubeSandbox is a self-hosted sandbox service for teams that need AI agents to execute code on their own infrastructure. Each sandbox has its own operating system kernel inside a hardware-isolated MicroVM.

AgentScope is a Python framework for developers building AI agent applications on their own servers. It includes an SDK, a FastAPI backend and a web UI for applications with multiple users and separate sessions.

PenguinHarness builds runnable AI agent applications from plain-language requirements on your computer or server. It's for developers who want agents to handle app creation, evaluation, tuning and deployment.

KubeAI is a Kubernetes operator for teams serving AI models on their own infrastructure or cloud clusters. It manages model servers and adjusts capacity to demand, including starting models from zero running replicas.

Agent Console helps developers and teams monitor Claude Code and Codex sessions through a local dashboard. It reads the agents' transcripts on your machines so you can compare usage and spot sessions that need attention.

BeeAI Framework is for developers building AI agents and systems where several agents work together. It lets developers enforce rules on agent behavior while retaining the model's ability to reason.

AgentENV runs isolated environments for AI agents across your own servers. It's for teams building parallel agent workflows or reinforcement learning systems that need many separate sandboxes.

Label Studio is a platform for teams preparing training datasets or reviewing AI outputs. It brings human annotation and evaluation into a browser interface that can run locally or on your own server.

Promptfoo is an open source CLI and library for developers and security teams evaluating AI applications on their own machine. It supports local models through Ollama alongside cloud providers. Evaluations run locally, but tests against cloud models send requests to those providers, and most require an API key.

Pezzo helps developers and teams manage prompts and investigate problems in LLM applications. It brings prompt editing and monitoring into one place for projects where several people contribute to AI behavior.

RedAmon is an AI penetration testing framework for security teams testing systems they're authorized to assess. It connects attack-surface discovery, exploit testing and code fixes in one self-hosted workspace.

H2O LLM Studio helps teams adapt language models to their own datasets through a browser interface, without writing training code. It's aimed at organizations building models for specific tasks.

Hermes HUD is a browser dashboard for people who already run the Hermes AI agent and want to inspect its activity and persistent memory. It requires a running agent and reads its local data.

MartinLoop runs around coding agents to keep software jobs within agreed boundaries and check their results. It's for developers and engineering teams who want to delegate longer tasks while retaining limits, recovery information and a record of what happened.

Msty is a family of AI products for individuals, teams and organizations working with models and their own knowledge. Msty Studio is its workspace product, with separate products for agents and centralized model access.

Helicone helps developers inspect and debug LLM applications, including agents, chatbots, and document processing pipelines. It combines request monitoring with a gateway for routing calls across AI providers.

APIPark is an AI gateway and API developer portal for teams managing model services and business APIs. It gives applications a common request format and lets teams share APIs with controlled access.

DeepTeam is a Python framework that runs on your machine to test chatbots, AI agents and retrieval-augmented generation applications for security failures. It's for developers and security teams who want adversarial tests they can repeat as an application changes. Its model-powered attacks and scoring use the selected provider; calls to OpenAI or other cloud services send data off your machine.

Pi Extensible Workflows is an extension for developers who want repeatable processes involving several AI agents. It coordinates resumable workflows inside the Pi host environment and lets you inspect work while it runs.

Codenotch is a desktop monitor for people who use AI coding assistants and local LLMs. A small display at a screen edge keeps usage readings and session states visible while you work in other apps.

Agent Router is an open source AI gateway for teams connecting agents to model providers and MCP tools. It gives applications a shared entry point while platform teams manage access and routing.

Bifrost is an AI gateway for developers and teams building applications that use local LLMs or multiple cloud providers. It gives those applications a shared API for model access and a central place to manage usage.