Tools tagged with "AI red teaming"

LangWatch helps AI teams test agent behavior before release and investigate production failures. Teams can reproduce a failed conversation and check how an agent handles it again.

Garak is an LLM vulnerability scanner for developers and security teams assessing models or dialogue systems. It attempts to provoke unwanted responses so teams can identify weaknesses in the systems they test.

Future AGI helps teams test AI agents, investigate production failures and compare prompt changes. It brings evaluation and monitoring into the same platform so teams can connect poor responses to the steps that caused them.

ChainForge helps developers test how reliably LLMs respond before choosing a prompt or model for an application. Its visual flows turn prompt variations into repeatable comparisons rather than isolated chat experiments.

ASSERT is a local-first Python framework for developers testing AI agents and LLM applications against their own requirements. It turns written expectations into evaluations for quality, safety, and regression testing.

PyRIT is an open-source Python framework for testing generative AI systems, with a self-hosted CoPyRIT browser interface. Its backend serves the interface locally, and Docker deployment is available. Targets and scorers use configured endpoints, including Ollama or cloud providers. The documented OpenAI configuration sends requests to OpenAI; hosting the testing interface locally does not make every model local.

AI Infra Guard is a self-hosted AI security testing platform for developers and security teams checking agent workflows, model responses and AI infrastructure. It combines vulnerability scans and adversarial prompt tests in a web interface.

Captum is a Python library for examining what contributes to a PyTorch model's predictions, including input features, learned concepts and training examples. It's for model developers investigating unexpected results, application engineers explaining predictions, and researchers comparing interpretability methods. It works with most PyTorch model types with little modification to the model itself.

Tracely helps developers trace AI agents and turn production failures into tests that catch the same problem before another release. It uses recorded runs as test cases, so teams can build regression coverage from failures their agents have actually encountered.

Giskard Hub is an AI agent testing platform available hosted or on-premise for organizations that need testing within their own environment. Developers can also run its open-source Python library locally, but the library's LLM judges and attack generators require a model provider and API key, with integrations for OpenAI and Anthropic. Local execution doesn't make those evaluations offline.

DarkMoon runs autonomous penetration tests for security teams, DevSecOps engineers and red teamers. It discovers a target's technology stack, assigns specialist AI agents and tests vulnerabilities across connected systems.

FederatedScope is an open-source federated learning platform for researchers and teams building training systems across separate data holders. It can simulate clients and servers on one device or run them as separate distributed participants. Each participant loads its own data, while models can pass between participants through messages.

Nox is a security scanner for developers building LLM applications, RAG systems and agents with tool access. It traces untrusted input into model calls and model output into dangerous operations, including paths across functions and files.

Promptfoo is an open source CLI and library for developers and security teams evaluating AI applications on their own machine. It supports local models through Ollama alongside cloud providers. Evaluations run locally, but tests against cloud models send requests to those providers, and most require an API key.

DeepTeam is a Python framework that runs on your machine to test chatbots, AI agents and retrieval-augmented generation applications for security failures. It's for developers and security teams who want adversarial tests they can repeat as an application changes. Its model-powered attacks and scoring use the selected provider; calls to OpenAI or other cloud services send data off your machine.

Agentic Security is an LLM vulnerability scanner for developers, researchers and security teams testing AI models and agent workflows. It probes how systems respond to hostile inputs and attempts to bypass their safety controls.

TrustLLM is a research toolkit for teams comparing language models beyond answer quality. It evaluates local LLMs and models accessed through APIs for behaviors such as hallucination, harmful responses, bias, and information leakage.