What is Promptfoo used for?

Promptfoo is used to evaluate prompts, models, and LLM application behavior with repeatable test cases. Teams use it for prompt regression testing, provider comparison, assertion based checks, safety testing, output quality review, and CI gates before shipping prompt or model changes.

Is Promptfoo only for developers?

No. Developers, QA engineers, AI product teams, and prompt engineers can all use Promptfoo. YAML configuration and CSV based test cases make it practical for non developers, while custom assertions, providers, and CI integration support more advanced engineering workflows.

What can Promptfoo assert?

Promptfoo can assert exact content, contains or not contains checks, JSON validity, schema shape, similarity, latency, cost, model graded rubrics, factuality style rubrics, and custom JavaScript or Python scoring. The right assertion depends on the risk of the LLM task.

How is Promptfoo different from manual prompt testing?

Manual prompt testing is useful for exploration, but it is hard to repeat. Promptfoo turns important examples into a versioned eval suite, runs them across prompts or providers, records results, and helps teams detect regressions when prompts, models, tools, or retrieval data change.

Should Promptfoo run in CI?

Yes, for stable high value evals. CI should run a focused suite that checks safety, format, refusal rules, tool use, and important golden examples. Larger exploratory or judge based suites can run on a schedule because they may cost more and produce noisier results.

Promptfoo Tutorial: Test LLM Prompts with Real Evals

A Promptfoo tutorial should move beyond trying one prompt in a chat window. Promptfoo tutorial work is about turning LLM behavior into repeatable evals: prompts, providers, test cases, assertions, thresholds, and reports that help QA decide whether a prompt or model change is safer than the previous version.

This guide is written for testers who need to plan, execute, review, and explain this work in a real delivery team. You will get a practical workflow, a risk based checklist, example test cases, evidence tips, automation notes, common mistakes, and a final release decision pattern. If you are building your fundamentals, pair this guide with prompt regression testing, llm evaluation metrics, how to write llm evals and then practice the same thinking in a timed QA challenge from QABattle battles.

What Promptfoo tutorial Means in Real Projects

In project work, Promptfoo tutorial means checking prompt variants, providers, test cases, assertions, rubrics, golden examples, regression testing, red team cases, CI gates, and result review. The phrase sounds simple, but the testing is not a random tour through the product. It starts with supported users, supported environments, business rules, known defect history, and the cost of failure. A checkout failure, a wrong invoice, a broken login, or a misleading AI answer deserves deeper coverage than a rarely used settings label.

The first job is to define the risk. Ask what the user is trying to do, what data moves through the system, what assumptions the product makes, and what could fail silently. Silent failures matter because they often reach customers before anyone notices. A report with the wrong number, a translated email with a broken variable, or a database row with the wrong status may not throw an obvious error. QA has to design checks that make those failures visible.

The second job is to define the evidence. A good test result is not only pass or fail. It includes the environment, data, input, output, expected rule, actual behavior, screenshots or logs, and the specific reason the result matters. This is why the existing guide on how to write test cases is useful before any specialized testing topic. The better your case design, the easier it is to repeat the result and convert important checks into automation later.

Area	What to Check	Why It Matters
Prompt variants	Compare wording or system instructions	Find regressions and quality improvements
Providers	Compare models or deployments	Measure quality, cost, latency, and reliability
Test cases	Inputs and expected behavior	Create repeatable coverage for known risks
Assertions	Rules and scoring checks	Fail unsafe, malformed, or low quality output
Reports	Eval output and web view	Review diffs and decide what to ship

Use the table as a starting point, not a fixed template. Your product may add regulated data, offline mode, third party integrations, complex permissions, or performance constraints. The important move is to turn broad quality words into observable checks. If a requirement says reliable, ask what reliable means. If it says localized, ask which locale and which workflows. If it says compatible, ask which supported combinations and which customer segments.

Promptfoo Tutorial: Step by Step

Start with scope. List the feature, user role, data state, environment, and release decision that this testing must support. Scope protects the team from two bad outcomes: shallow testing that misses the main risk, and endless testing that tries to cover every possible variation without priority. A useful scope statement might say, "For this release, we must prove that paid users can complete the renewal workflow in the top three supported environments using current production like data."

Next, collect inputs. Read the user story, design, API contract, data mapping, analytics, support tickets, defect history, and support policy. Do not treat these as paperwork. They tell you where users spend time, where the system is brittle, and where the business cannot afford a miss. When information is missing, write the question down and label the related test case as blocked or assumption based. That makes uncertainty visible.

Then define a small smoke set. A smoke set is the minimum proof that the build is worth deeper testing. It should cover the highest value workflow, the most common environment, and one or two known weak points. For Promptfoo tutorial, the smoke set should be fast enough to run often and clear enough that a failure stops the release conversation until triaged.

After the smoke set, expand by risk. Add negative cases, boundary cases, data variation, permissions, integrations, environment variation, and recovery behavior. This is where testers add value. A product owner may describe the happy path, but QA must ask what happens when the value is missing, duplicated, stale, translated, delayed, denied, expired, malformed, or coming from an older version.

Finally, decide what belongs in manual testing, automation, and monitoring. Manual testing is strong when judgment, exploration, and changing behavior matter. Automation is strong when the check is stable, repeatable, and valuable across releases. Monitoring is strong when the risk appears only with production volume, real users, or live integrations. Mature teams use all three instead of arguing that one replaces the others.

Step 1: Build the Risk Map

A risk map is a short list of the ways this area can hurt users or the business. For a user facing workflow, include blocked tasks, wrong data, lost trust, accessibility friction, slow performance, and confusing messages. For a backend or AI workflow, include incorrect state, bad transformations, unsafe responses, missing evidence, and expensive retries.

Write risks in plain language. "User cannot complete payment on mobile Safari" is more useful than "browser issue." "Model returns unsupported refund policy" is more useful than "bad answer." Plain risk language helps developers, product owners, support teams, and managers understand why a test deserves time.

Rank each risk by impact and likelihood. Impact asks how bad the failure would be. Likelihood asks how likely it is given code churn, complexity, usage, and defect history. A high impact low likelihood risk may still need one focused test. A high impact high likelihood risk deserves deep coverage and probably automation or monitoring.

Step 2: Choose Test Data Carefully

Test data determines whether your result is meaningful. Use data that exercises real rules, not random placeholders. Include valid data, invalid data, boundary values, old records, new records, missing optional values, duplicate values, and data created by other systems. When the product supports multiple user roles, create data for each role instead of testing everything as an admin.

Good data also makes defects easier to debug. Use unique names, timestamps, email aliases, record IDs, or batch IDs so you can find your test records later. If a test changes state, document cleanup. If cleanup is automated, know when it runs. If the environment refreshes overnight, avoid writing cases that depend on data disappearing or surviving without saying so.

For sensitive data, use masked or synthetic records. QA does not need real customer secrets to prove most behavior. If production like data is required, follow access policy, logging policy, and retention policy. A test that creates privacy risk is not a quality improvement.

Step 3: Design Positive, Negative, and Recovery Paths

Positive paths prove that the main job works. They should be stable, clear, and prioritized. Negative paths prove that invalid input, missing permission, wrong state, unsupported environment, or failed dependency is handled safely. Recovery paths prove that the user or system can continue after an error without duplicate actions or corrupted data.

Many teams under test recovery. They check that an error appears, then stop. Better testing asks what happens after the error. Can the user fix the input and submit again? Is the original data preserved? Did the backend avoid a partial save? Did an email or notification go out incorrectly? Did a retry create a duplicate record?

For deeper scenario design, connect this work to exploratory testing. Scripted cases protect known risks. Exploration finds unknown risks. A strong tester uses both, then turns important discoveries into repeatable cases.

Step 4: Create Reviewable Test Cases

A reviewable test case has a clear title, preconditions, exact data, numbered steps, expected results, priority, and evidence expectations. The title should name the behavior under test. The expected result should say what correct behavior looks like, not "works" or "looks good."

Keep each case focused. A long end to end journey can be useful, but if one case verifies ten behaviors, failure analysis becomes slow. Split critical checks into smaller cases, then keep one or two journey tests for confidence. This is especially important when cases are later automated. Automation built from vague manual cases becomes fragile and expensive.

Use the bug report mindset while writing cases. If this case fails, what evidence would a developer need? That usually means exact data, environment, input, observed output, expected rule, screenshot, logs, IDs, and time. See how to write a bug report for the evidence pattern that pairs well with test case design.

Example Test Cases

ID	Test Case	Preconditions	Test Data	Expected Result
PF-001	Verify answer includes required JSON fields	Prompt expects structured output	Support ticket input	Output is valid JSON and includes category, priority, summary
PF-002	Verify unsafe request is refused	Safety policy exists	Credential theft request	Model refuses and gives safe alternative
PF-003	Verify summarizer preserves key facts	Golden examples exist	Long support thread	Summary includes customer, issue, plan, and next action
PF-004	Verify no forbidden phrase appears	Brand style rule exists	Marketing prompt	Output does not contain prohibited wording
PF-005	Verify model comparison quality	Two providers configured	Same eval set	Winning provider meets threshold and does not regress critical cases
PF-006	Verify latency budget	Provider is reachable	Representative prompts	Median response time stays within accepted budget
PF-007	Verify retrieval grounded answer	RAG context is supplied	Question with known source	Answer uses context and avoids unsupported claim
PF-008	Verify CI fails on broken format	CI eval command is configured	Prompt returns prose instead of JSON	Pipeline fails with clear assertion output

These examples are intentionally compact. In a test management tool, each row can expand into detailed steps, attachments, owner, priority, and traceability. The point is that every case has a distinct reason to exist. If two cases prove the same behavior with the same data and the same environment, merge them or explain the difference.

A useful suite also marks automation candidates. Stable checks with deterministic expected results are good candidates. Subjective checks, early design exploration, one time migration review, and nuanced content review often stay manual. The decision should be based on value and maintainability, not on the false idea that automation is always better.

Comparison: Manual, Automated, and Monitored Coverage

Coverage Type	Best Use	Weakness	Evidence
Manual testing	New behavior, judgment, visual review, unclear requirements	Slower and less repeatable	Notes, screenshots, videos, query output, trace links
Automated testing	Stable regression checks and repeated smoke flows	Maintenance cost and limited judgment	Test results, logs, screenshots, artifacts
Monitoring	Production signals and real usage behavior	Detects after users or systems interact	Metrics, alerts, traces, dashboards

A healthy QA strategy combines all three. For example, you might manually explore a new workflow, automate the critical checks after behavior stabilizes, and monitor production errors or quality scores after release. This gives the team fast feedback before merge, deeper confidence before release, and real signals after release.

Practical Example

promptfooconfig.yaml
prompts:
  - file://prompts/support-triage.txt

providers:
  - openai:gpt-5-mini
  - openai:gpt-5

tests:
  - vars:
      ticket: 'Customer says the invoice total is wrong after applying SAVE20.'
    assert:
      - type: is-json
      - type: contains-json
        value:
          required: ['category', 'priority', 'summary', 'next_action']
      - type: llm-rubric
        value: 'The response classifies this as a billing issue and does not invent account details.'

Run locally:
npx promptfoo@latest eval
npx promptfoo@latest view

The example is not meant to be copied blindly. Treat it as a pattern. Identify the smallest piece of evidence that proves the rule, then make that evidence repeatable. If the check is code based, keep it readable enough that a tester can explain the failure. If the check is manual, keep the data and expected result precise enough that two testers would reach the same conclusion.

Common Mistakes in Promptfoo tutorial

Mistake 1: Treating a few manual prompt trials as proof that an LLM feature is stable.

This mistake usually happens when the team optimizes for activity instead of evidence. It creates a test result that looks busy but does not answer the release question. The fix is to connect the check back to a risk, a user, a data rule, or a support policy. If that connection is weak, the case should be rewritten, reduced, or removed.

A practical correction is to add one sentence to the case: "This matters because..." That sentence forces clarity. It may reveal that the case protects revenue, trust, compliance, accessibility, support cost, or regression history. It may also reveal that the case is low value and should not block higher risk testing.

Mistake 2: Writing evals with vague rubrics that cannot produce consistent pass or fail decisions.

Mistake 3: Testing only happy path examples and ignoring refusals, ambiguity, missing context, and adversarial inputs.

Mistake 4: Comparing providers without tracking cost, latency, rate limits, and output format stability.

Mistake 5: Putting a large expensive judge based suite on every pull request instead of using a small CI gate.

Mistake 6: Ignoring prompt versioning, which makes it hard to explain why an eval result changed.

Mistake 7: Using golden answers that are too rigid for tasks where multiple correct answers are acceptable.

Mistake 8: Not reviewing failures manually, especially when model graded assertions are used.

Checklist Before You Call It Done

The scope names the feature, user role, environment, and release decision.
The highest impact user journey has at least one positive case.
Important negative, boundary, permission, and recovery paths are covered.
Test data is unique, available, and safe to use.
Expected results are observable and specific.
Defect evidence includes environment, version, data, steps, expected result, actual result, and attachments.
Automation candidates are marked separately from exploratory or judgment heavy checks.
Related risks are linked to existing guides such as prompt regression testing, llm evaluation metrics, how to write llm evals.
A small smoke set can run quickly before deeper regression.
Any open assumptions are visible to the team before release.

How to Report Defects from This Testing

A defect from Promptfoo tutorial should be reproducible and decision ready. Include the exact environment, build, account or record, input, expected rule, actual result, and impact. If the issue appears only in one environment, say which one and which comparable environment passed. If the issue is data related, include safe query output or record IDs. If the issue is visual, attach screenshots from the failing and passing states.

Do not hide uncertainty. If you suspect the root cause but have not proven it, label it as a hypothesis. For example, "This may be related to timezone conversion because the database stores UTC while the UI displays local time." That is useful. Stating the hypothesis as fact can send developers down the wrong path.

Severity should reflect impact, not how interesting the bug is. A tiny layout issue that blocks the payment button on mobile can be critical. A backend warning that has no user impact may be lower severity, unless it indicates data loss or security risk. Tie severity to user task, business rule, compliance, data integrity, or release confidence.

Maintenance Strategy

After release, the suite should become sharper. Remove duplicate cases. Update expected results when behavior changes. Convert repeated high value checks into automation. Add regression cases for escaped bugs. Retire checks for unsupported environments or removed features. A suite that only grows becomes slow and less trusted.

Review the suite when analytics change. If mobile traffic grows, add mobile coverage. If a new region launches, add locale and currency checks. If a new model, provider, database, or browser becomes important, update the matrix. Testing should follow real risk, not last quarter's assumptions.

Use post release learning. Support tickets, production incidents, failed jobs, monitoring alerts, and customer complaints are all inputs to better QA. The best test suites are not written once. They absorb evidence from every release and make the next release harder to break in the same way.

Practice Drills for QA Teams

Take one real feature and write a one page risk map. Limit yourself to ten risks. Then choose the top three and write one positive case, one negative case, and one recovery case for each. This keeps the exercise focused and prevents the team from writing dozens of low value variations.

Next, review the cases with a developer. Ask which checks are redundant, which expected results are technically wrong, and which failure would be hardest to debug. This conversation often reveals hidden architecture assumptions. It also builds trust because QA is showing how evidence will be collected before defects appear.

Then run the cases and improve them immediately. If a step was unclear, rewrite it. If data was hard to find, add setup instructions. If the expected result was incomplete, make it observable. If a defect was found, add evidence while it is fresh. Test cases are strongest when they evolve during real execution, not weeks later.

For a timed practice loop, choose a relevant challenge in QABattle battles, run the workflow, and convert your observations into a small regression suite. The discipline is the same whether the product is a training arena or a production system: understand the risk, create clean data, observe carefully, and report evidence.

Final Practical Workflow

Define the user, feature, environment, and business risk.
Read requirements, designs, data rules, contracts, support policy, and past defects.
Build a prioritized risk map.
Select representative test data and document cleanup.
Write focused positive, negative, boundary, permission, and recovery cases.
Add expected results that can be observed without guessing.
Run a small smoke set early.
Expand coverage where impact and likelihood justify the cost.
Capture evidence that helps developers reproduce and fix defects.
Convert stable high value checks into automation or monitoring.

The value of Promptfoo tutorial is not the number of cases produced. The value is the confidence created by clear scope, realistic data, meaningful checks, and useful evidence. When a tester can explain what was covered, what was not covered, what failed, and why the result matters, the team can make a better release decision.