Player not loading? Watch on YouTube
This Promptfoo tutorial covers installation, YAML test configuration and results review in a local dashboard. The presenter works in VS Code and switches from Node 18 to Node 22 after an installation error. He then initializes a project and explains the generated prompts, model providers, test variables and assertions.
The example uses OpenAI's 4o and 4o Mini models to generate tweets about bananas, avocados and New York City. Two prompts, two providers and three test cases produce 12 results. The setup includes an OpenAI API key stored in a .env file. Although the testing workflow runs on the presenter's computer, the demonstrated model calls use a paid API; this is not an offline inference tutorial.
The recorded run has 10 passes and two failures. Both failures concern the New York City test: an assertion requires humor, but the basic prompt does not request it. The presenter uses this mismatch to explain an LLM rubric, where another model judges the response against a written requirement.
He also distinguishes a deterministic content check from a JavaScript length score. In this example, the score has no failure threshold, so it ranks outputs without rejecting them. The final section reviews token usage, latency and cost, with a warning from the presenter that prompt or model changes can increase spending.
