ASSERT

ASSERT is a local-first Python framework for developers testing AI agents and LLM applications against their own requirements. It turns written expectations into evaluations for quality, safety, and regression testing.

• Tests from requirements: Generate behavior categories and allowed or prohibited actions from product requirements, policies, or system prompts, with reusable presets for safety, fairness, and agent failures.
• Single-turn and multi-turn coverage: Create ordinary and adversarial prompts or simulated conversations, with coverage divided across dimensions relevant to your application.
• Evidence-based scoring: An LLM judge checks results against your policies and can cite captured tool calls, routing, model calls, and latency alongside the final response.
• Run comparisons: Use the local viewer to compare evaluations against a pinned baseline, inspect failures by behavior, and read judge explanations and traces.
• Sandboxed agent actions: Test agents in a Docker sandbox that can mock or block tool calls, deny direct internet access, and record attempted actions for review.

ASSERT is open source under the MIT license. It supports targets callable from Python, model endpoints through LiteLLM (including vLLM), and agent traces through OpenInference and OpenTelemetry. Results stay in local JSON and JSONL files. Configured external targets, judges, model providers, or trace collectors receive the evaluation data sent to them.