Inspect AI

Inspect AI is a Python framework for researchers and developers evaluating language models and AI agents. It measures coding, reasoning, knowledge, behavior and multimodal understanding, including tasks where agents must take actions to succeed.

• Reusable evaluations: Combine datasets, agents, tools and scorers into custom tests, or run ready-made benchmarks such as SimpleQA.
• Answer scoring: Assess responses with text comparisons, model-based grading or custom criteria, with support for multi-turn conversations.
• Agent testing: Evaluate built-in agents, multi-agent tasks and external agents such as Claude Code, Codex CLI and Gemini CLI using custom or MCP tools, bash, Python and browser interaction.
• Isolated execution: Run untrusted model code in sandboxes backed by Docker, Kubernetes, Modal, Proxmox or Vagrant.
• Results inspection: Monitor evaluations and examine individual transcripts in the browser-based Inspect View, with a VS Code extension for writing and debugging evaluations.

Developed by the UK AI Security Institute and Meridian Labs, Inspect AI is open source under the MIT license. It supports local inference on your hardware through Hugging Face, vLLM and SGLang, as well as cloud APIs from OpenAI, Anthropic and Google. Cloud API requests go to the selected provider. Python packages can extend scoring and elicitation techniques, while a sandbox extension API supports additional execution environments.