Promptfoo
Open source CLI and library for local LLM evaluations and security testing, with Ollama and hosted model APIs plus CI/CD integration.
Promptfoo is an open source CLI and library for testing prompts, AI agents, and RAG applications. It runs evaluations locally and helps developers compare model responses while security teams look for weaknesses in the applications built around them. The project is MIT licensed.
It works with Ollama as well as hosted providers including OpenAI, Anthropic, Azure, and Bedrock. You can compare models side by side and assess responses using automated evaluations. Hosted providers generally require API keys. Promptfoo supports on-premise and cloud use, so teams can choose where to run the testing tool.
Its security testing generates attacks tailored to an application, including prompt injection and jailbreak attempts. Tests can cover RAG pipelines, agents, business logic, and connected tools. That scope matters for teams whose risks come from how an application uses a model, not only from the model's answers.
Promptfoo fits into CI/CD workflows with GitHub, GitLab, and Jenkins. It can scan code for LLM-related security and compliance issues, put findings and suggested fixes in pull requests, and track remediation across teams. Developers can share evaluation results with colleagues.
