LLMOps: Gateways, Observability and Guardrails
Self-hosted tools for LLM apps in production, including gateways that route between models, tracing, evals and guardrails.
Subcategories
An open-source Python framework evaluates LLMs, RAG and predictive models with custom metrics and a self-hosted monitoring dashboard.
8.1KUpdated 2 months agoApache-2.0
Web
Evidently helps AI teams check output quality and catch regressions in chatbots, RAG applications, AI agents and predictive models. Its Python library handles text and tabular data for development evaluations and production monitoring.
LLM Evaluation and TestingLLM Observability and Tracing+0
A browser-based UI design tool that turns Material 3 Expressive layouts into prompts for AI coding assistants, with a free, MIT-licensed static app.
10.3KUpdated 3 weeks agoMIT
Web#Multilingual
M3E Canvas is a visual editor for people planning Material 3 Expressive apps with an AI coding assistant. It lets you sketch screens and test their flow before handing a text prompt to a tool such as Claude Code, Codex, Gemini CLI or Cursor.
Coding Assistants and AgentsPrompt Management and Playgrounds+0
A desktop coding agent workspace runs Claude Code, Codex and ACP agents side by side, with local session capture and commit-linked history.
9.4KUpdated 1 day agoApache-2.0
macOS · Windows · Linux#Agent Client Protocol#Agent Skills#Code execution
Atlas is a desktop workspace for developers who use coding agents across projects and want to keep the context behind their code changes. It brings agent conversations, code review and project history into one window.
Agent MemoryAI Code EditorsLLM Observability and Tracing+0
An open-source LLM inference stack for self-hosted Kubernetes clusters that coordinates vLLM and SGLang across GPUs, TPUs, XPUs and CPUs.
5.3KUpdated 2 days agoApache-2.0
#Batch processing#Distributed execution#OpenAI-compatible API
llm-d is a distributed LLM inference stack for infrastructure teams serving models across a Kubernetes cluster. It coordinates model servers such as vLLM and SGLang to handle production traffic across multiple machines.
Inference Servers and ClustersKubernetes and Cloud-Native AILLM Gateways and Routers+0
A coding agent workflow analyzer that turns project and session evidence into prioritized findings, with support for Claude Code, Codex and Qoder.
2.5KUpdated 3 days agoMIT
#Agent Skills#Git integration#Source citations
Better Harness helps developers and teams assess how coding agents carry out work. It examines the workflow around a code change to find unclear goals, missing validation and delivery checks that a diff review can miss.
Coding Assistants and AgentsLLM Evaluation and TestingLLM Observability and TracingMCP Servers and Agent Skills+0
A Python library that reduces LLM cache memory use with Hugging Face Transformers, multiple compression methods and support for cache quantization.
1.3KUpdated 4 days agoApache-2.0
#Context compression#Distributed execution#Hugging Face integration
kvpress is a Python library for developers and researchers reducing the memory needed for long-context LLM inference. It compresses the key-value cache that grows as a model processes longer inputs.
Inference Libraries and BindingsLLM Evaluation and Testing+0
AI content moderation models check LLM inputs and outputs for text and image risks, with multilingual support and MLCommons hazard categories.
4.5KUpdated 2 days ago
#Guardrails#Multilingual#Multimodal input
Llama Guard is a family of AI content moderation models for developers who need to check what users submit to an LLM and what the model returns. It supports text and image checks for applications that accept more than written prompts.
Guardrails, Security and SandboxingVision-Language Models+0
An AI plugin for self-hosted Discourse forums adds chatbots, semantic search and moderation, with a choice of providers and self-hostable AI API services.
48.1KUpdated 22 hours agoGPL-2.0
Linux · Web#LLM tracing#Multi-user access#Structured output
Discourse AI adds community assistants, search and moderation to a Discourse forum you can host on your own server. It's for forum operators who want AI features inside their community site. Forum hosting and AI processing are separate choices: the plugin supports a choice of providers, and its AI API services have a self-hosting option.
AI Bots for Chat AppsLLM Evaluation and TestingWebsite and Support Chatbots+0
A local Claude Code session manager for macOS and Linux with spending limits and optional LLM supervision through Ollama or compatible endpoints.
204Updated 2 weeks agoMIT
macOS · Linux#Distributed execution#Guardrails#Human approval
claudectl manages multiple Claude Code sessions for developers who need to track parallel work and intervene when an agent gets stuck. Its terminal dashboard brings session activity, approvals and spending into one view.
Agent FrameworksCoding Assistants and AgentsGuardrails, Security and SandboxingLLM Observability and TracingMCP Servers and Agent Skills+0
An open-source Python LLM evaluation framework with local test runs, explainable scores and custom criteria for agents, RAG pipelines and chatbots.
19.5KUpdated 2 days agoApache-2.0
#LLM tracing#Multimodal input
DeepEval is an LLM evaluation framework for developers testing AI agents, RAG pipelines and chatbots. It helps compare prompts and models and catch quality regressions within Python tests or CI/CD.
Datasets and Synthetic DataLLM Evaluation and TestingLLM Observability and Tracing+0
A self-hosted AI agent observability platform with OpenTelemetry tracing, evaluations and guardrails, free to run under Apache 2.0.
3.1KUpdated 4 days agoApache-2.0
Windows · Linux · Docker · Web#LLM tracing#Ollama integration#Prompt versioning
OpenLIT helps developers investigate failures and assess the quality of existing AI applications and coding agents. It connects model requests with the tools, retrieval and agent steps involved in producing an answer.
GPU Management and MonitoringLLM Evaluation and TestingLLM Observability and TracingPrompt Management and Playgrounds+0
A self-hostable LLM observability platform with tracing, evaluations and prompt management, available under MIT or as a hosted cloud service.
37.4KUpdated 1 day ago
Docker#Agent Skills#LLM tracing#MCP
Langfuse helps teams debug LLM applications and AI agents, then test changes against production examples. It brings traces, evaluations and prompt revisions into one place.
LLM Evaluation and TestingLLM Observability and TracingPrompt Management and Playgrounds+0
Self-hosted AI gateway routes requests to Ollama and cloud providers through an OpenAI-compatible API, with optional EU-hosted observability. MIT licensed.
25Updated 1 week agoMIT
macOS · Linux · Docker#Ollama integration#OpenAI-compatible API#Streaming inference
LunarGate is a self-hosted AI gateway for teams whose apps use multiple model providers. It gives those apps one OpenAI-compatible endpoint and keeps routing, retries and fallback rules in your infrastructure. The gateway is open source under the MIT license and runs on macOS and Linux, with a container image also available.
LLM Gateways and Routers+0
An open-source AI app framework connects to Ollama, Gemini and OpenAI, with self-hosted application logic, tool calling and structured output.
6.6KUpdated 1 day agoApache-2.0
Web#LLM tracing#Multimodal input#Ollama integration
Genkit is Google's framework for developers building AI agents and chatbots into web and mobile apps. It brings model connections, application workflows and testing tools into one framework.
Agent FrameworksLLM Evaluation and TestingLLM Libraries and SDKsPrompt Management and PlaygroundsRAG Frameworks and Pipelines+0
A self-hosted LLM gateway with Ollama support, response evaluation and adaptive A/B testing, licensed under Apache 2.0 and no longer maintained.
11.7KUpdated 5 months agoApache-2.0
Docker · Web#Batch processing#LLM tracing#Multimodal input
TensorZero is a self-hosted platform for developers who need to compare and improve models in LLM applications. The project is archived and no longer maintained.
LLM Evaluation and TestingLLM Fine-TuningLLM Gateways and RoutersLLM Observability and TracingPrompt Management and Playgrounds+0
A TypeScript test runner for AI apps lets you inspect results on localhost, compare LLM outputs, and enforce score thresholds in CI.
1.7KUpdated 5 days agoMIT
Web#LLM tracing
Evalite is a TypeScript evaluation tool for developers testing LLM-powered apps. It pairs code-based tests with a local interface for reviewing how an app responds.
LLM Evaluation and Testing+0
A Python LLM security toolkit you can deploy as an API to screen prompts and responses. MIT licensed; archived and no longer maintained.
3.2KUpdated 4 months agoMIT
#Guardrails#PII redaction
LLM Guard is a Python security toolkit developers can deploy as a self-hosted API to check prompts and model responses in their applications. It provides screening and sanitization rather than running the application's LLM. The project is archived and no longer maintained, including its associated Hugging Face models.
Guardrails, Security and SandboxingLLM Libraries and SDKs+0
LogotyperGot an idea for an AI tool? I'm Eugene, the designer and developer behind Localhosted. I'll help you build it, from the first sketch to launch.
Start your projectLogotyperGot an idea for an AI tool? I'm Eugene, the designer and developer behind Localhosted. I'll help you build it, from the first sketch to launch.
Start your project
