Promptfoo

Promptfoo is an open source CLI and library for developers and security teams evaluating AI applications on their own machine. It supports local models through Ollama alongside cloud providers. Evaluations run locally, but tests against cloud models send requests to those providers, and most require an API key.

Automated evaluations compare prompts and model responses side by side. A local results viewer and evaluation metrics help developers assess changes to an application. OpenAI, Anthropic, Azure and Bedrock support lets teams compare hosted models with an Ollama backend in the same testing workflow.

Security testing examines the application around the model, including agents, document retrieval and connected tools. It generates attacks tailored to business logic and covers prompt injection and jailbreak attempts. Vulnerability reports capture the findings, while code scanning checks pull requests for LLM-related security and compliance issues.

Teams can bring evaluations into CI/CD, with integrations for GitHub, GitLab and Jenkins, and share results with colleagues. The broader security platform also provides remediation guidance in pull requests and tracks fixes across teams.

The CLI and library use the MIT license. Live reload and caching support repeated evaluations during development.