LLMOps: Gateways, Observability and Guardrails

llmfit helps people choosing models for their own computers compare what will fit in memory and how fast it might run. It checks your hardware before recommending models, so you can assess candidates before downloading them.

ECC adds reusable engineering workflows to AI coding assistants. It's for developers who want editable plans, test-first development and separate review contexts across sessions.

FreeLLMAPI gives developers one API for experimenting with cloud models and local backends. The router runs locally, but cloud inference still sends requests to the providers you enable.

Free Claude Code lets developers use coding agents with local models or a choice of cloud providers. It connects existing clients to a shared model catalog rather than requiring a separate coding workflow.

OmniRoute is an AI gateway for developers who want their coding assistants and other AI clients to share one set of provider connections. It routes requests across models and accounts when a service fails, reaches a rate limit or runs out of quota.

Paperclip coordinates AI agents for people managing work across development, research, marketing and other business functions. It gives the team a shared structure and keeps its work tied to organization goals.

Cua supplies desktops and automation tools for developers building AI agents that operate native apps and browsers. You bring your own agent and model; Cua provides the computer environment and ways to interact with it.

LangWatch helps AI teams test agent behavior before release and investigate production failures. Teams can reproduce a failed conversation and check how an agent handles it again.

CC Switch is a desktop manager for developers who use multiple AI coding assistants or API providers. It brings their settings into one interface and preserves existing plugins, hooks and other custom settings when switching providers.

AIClient2API is a self-hosted AI API proxy for developers connecting chat apps and coding assistants to multiple model providers. It exposes services such as Antigravity, Codex, Grok, and Kiro through interfaces those apps already support.

Context Mode is a local MCP plugin for developers whose AI coding agents fill their context windows with logs, file reads and search results. It stores large tool outputs on your machine so the agent can retrieve relevant passages as needed.

HarnessX is a Python framework for developers building AI agents whose behavior they can reuse and adapt. It separates model choice from tool use, memory and context handling, so changing an agent's role doesn't require rewriting it.

MTEB helps researchers and developers compare embedding models and retrieval systems for the tasks they need. Its benchmarks cover multiple languages and modalities.

Garak is an LLM vulnerability scanner for developers and security teams assessing models or dialogue systems. It attempts to provoke unwanted responses so teams can identify weaknesses in the systems they test.

Higress is an AI gateway for teams building AI apps and agents that need shared controls across model requests and tool calls. It brings model APIs, MCP services and Kubernetes inference workloads behind one entry point.

Wordflow is an AI writing assistant for writers and researchers who want to reuse editing prompts across their work. Its browser editor lets you revise a paragraph or selected passage with a choice of local and external models.

Mini SWE Agent is a command-line AI coding assistant for developers tackling GitHub issues and everyday coding tasks. Its small Python core also suits researchers who want to inspect and modify an agent for model evaluations.

Future AGI helps teams test AI agents, investigate production failures and compare prompt changes. It brings evaluation and monitoring into the same platform so teams can connect poor responses to the steps that caused them.

Open Multi Agent is a TypeScript framework for developers building self-hosted AI agent teams. It lets you coordinate work across agents, require human decisions before consequential actions, and inspect what happened afterward.

AutoGen is a framework for cooperating AI agents, with local Playwright browser automation, code execution and an experimental OllamaChatCompletionClient for local models. Its orchestration can run on your machine or across servers. Standard OpenAI and Azure OpenAI service connections send model requests to remote services; compatible endpoint overrides are separate configurations.

BotSharp is a C# framework for .NET developers building AI assistants into business applications. It coordinates agents with different responsibilities and keeps track of conversation state.

OpenZiti LLM Gateway is a self-hosted API proxy for developers and teams connecting AI applications to inference servers across private networks and cloud services. Its optional zrok integration uses OpenZiti for identity-based access and end-to-end encryption, so clients and backends can connect across network boundaries without opening inbound ports.

OpenLLMetry helps developers trace LLM applications alongside their existing database and API activity. It extends OpenTelemetry, so teams can send AI application traces to the monitoring system they already use.

Unstract is a self-hosted, no-code platform for teams that need structured data from documents whose layouts vary. It uses LLMs to extract JSON from PDFs, images, scans and office files without building a template for each vendor or form. Its use cases include finance, insurance, healthcare and KYC document processing.

LiteLLM is an AI gateway for platform teams managing model access across applications and providers. It puts cloud and self-hosted models behind a shared interface so developers can switch providers without rewriting their integration.

RSI Harness builds on the Pi coding agent for developers and researchers who want to version and share tailored agent setups. It collects each setup into a local directory bundle called a Genome.

AIBrix provides infrastructure components for teams serving large language models on Kubernetes. It's aimed at organizations that need to manage inference capacity, request routing and models across a cluster.

NeMo Guardrails is a Python toolkit for developers building AI assistants that need rules for what they say and which actions they take. It adds checks around conversations, document-based answers and tool use.

Paritok is a self-hosted gateway for developers whose coding agents send large file reads, tool output and conversation histories to an LLM. It reduces that input before the request reaches the upstream model.

Strix is an AI penetration testing tool for developers and security teams assessing code, web applications and APIs. Its free, open source edition runs locally with Docker under the Apache 2.0 license. It supports local models through Ollama or LM Studio, as well as remote providers such as OpenAI and Anthropic; cloud-backed runs send model requests to the selected provider.

NullHub is a self-hosted management console for people running NullClaw and its supporting tools. It brings agent instances, workflow runs and task queues into one browser dashboard.

opencodex is a local AI gateway for developers who want to choose which models power their coding assistants. It connects Codex, Claude Code, Claude Desktop and Grok Build to alternative providers while keeping each tool's native interface.

ccusage is a local command-line tool for developers who want to see how many tokens their coding agents use and what that usage costs. It reads the agents' local usage records, so you can compare activity across tools in one place.

AgentArk is a self-hosted AI agent runtime for people who want a personal assistant that remembers preferences and carries out ongoing work under their control. It runs through Docker on macOS, Windows and Linux, with a browser interface for chat, tasks and action review. It's free and open source under Apache 2.0.