Promptfoo tutorial: compare models and test chatbots

This Promptfoo tutorial demonstrates three QA workflows: comparing customer support emails, checking prompt regressions and testing a chatbot's boundaries. The description calls Promptfoo an open source LLM testing tool. The demo uses configured model providers; it does not show how to run models locally.

The email comparison tests Llama and Anthropic Claude Sonnet 4 across four tones. Keyword assertions check required details, such as a 10% discount or a $299 refund, while LLM rubrics judge tone. The demonstrated run passes six of eight combinations. Both models fail the empathetic retention rubric despite including the requested discount, according to the presenter.

The regression example compares two ticket summarization prompts against the same three cases. Both pass in this run. The presenter explains how assertions can accept alternative wording, such as "duplicate charge," rather than require one exact phrase.

The chatbot example checks store policies, an unrelated weather question, invented product specifications and a prompt injection asking the bot to act like a pirate. Two tests pass and three fail. One failure comes from an overly strict assertion: the bot correctly refuses the weather question but repeats the prohibited word. The other failures expose fabricated specifications and pirate-style responses. The presenter recommends combining string checks with LLM rubrics, while noting that rubric evaluation has a cost.