Promptfoo Tutorial: Test LLM Prompts with Real Evals

A prompt change made a ticket classifier sound more helpful, but it also changed its output from strict JSON to a paragraph followed by JSON. The demo still looked correct. The production parser failed every response and routed tickets to the manual queue. A five-case format suite would have stopped the release before anyone debated answer quality.

Promptfoo can turn those repeatable risks into a configuration-driven evaluation. The important work is choosing cases and assertions that represent the application contract, then reading failures rather than chasing a single score.

Freeze the application contract first

Write down what consumes the model output. For the ticket router, the parser expects one JSON object with category, priority, and needs_human. Allowed categories are billing, delivery, returns, and other. Priority is an integer from 1 to 3. Safety policy requires human review when the user threatens self-harm or legal action.

Separate three kinds of requirements:

Do not begin with an LLM judge for a requirement a parser can verify exactly. Deterministic checks are cheaper to diagnose and less ambiguous.

Create a minimal runnable evaluation

Keep secrets in environment variables and keep the configuration in version control. Provider identifiers and credentials depend on the service your team uses. This example uses a placeholder provider so it cannot be copied into a real account accidentally:

description: Ticket routing contract

prompts:
  - file://prompts/router-v3.txt

providers:
  - id: your-provider:your-model

tests:
  - description: Late parcel stays in delivery queue
    vars:
      ticket: "Order 8841 was due Tuesday and has not arrived."
    assert:
      - type: is-json
      - type: javascript
        value: |
          const value = JSON.parse(output);
          return value.category === 'delivery' &&
            Number.isInteger(value.priority) &&
            value.priority >= 1 && value.priority <= 3 &&
            typeof value.needs_human === 'boolean';

The yaml block should execute one prompt-provider-test combination after the placeholder provider is replaced with an approved provider ID. is-json proves parseability. The JavaScript assertion proves selected business fields and returns false with a generic reason, so a file-based assertion is better once the rule grows.

Run from a clean working directory and retain the exact Promptfoo and configuration versions in CI evidence. Caching can speed iteration, but deliberately bypass or clear it when confirming behavior after a provider-side or application change.

Design a dataset around failure modes

Five paraphrases of an easy delivery ticket create volume, not coverage. Build cases from routing boundaries, production errors, policy exceptions, multilingual input if supported, prompt injection, missing order numbers, contradictory statements, and input length limits.

A focused first set could include:

Give each case a stable ID, source, owner, risk tag, and expected fields. Remove customer data. Keep a holdout set for release comparison so repeated prompt tuning does not simply memorize the visible suite.

Layer assertions from hard to soft

Evaluate in the order the application depends on results. If JSON is invalid, semantic grading may add little value. Then check schema, forbidden leakage, exact decisions, and only afterward tone or explanation quality.

Move complex logic to a JavaScript file so failures explain themselves:

module.exports = (output, { vars }) => {
  let value;
  try {
    value = JSON.parse(output);
  } catch {
    return { pass: false, score: 0, reason: 'Output is not JSON' };
  }

  const allowed = ['billing', 'delivery', 'returns', 'other'];
  const pass = allowed.includes(value.category) &&
    Number.isInteger(value.priority) &&
    value.priority >= 1 && value.priority <= 3 &&
    typeof value.needs_human === 'boolean';

  return {
    pass,
    score: pass ? 1 : 0,
    reason: pass ? 'Contract satisfied' : `Invalid routing object for ${vars.case_id}`,
  };
};

The javascript block returns a grading result with pass, score, and reason. Reference it from an assertion using type: javascript and value: file://assertions/routing-contract.js. Test the assertion itself with malformed, partial, and valid outputs.

Use negative assertions for secrets, system-prompt fragments, and forbidden claims. Use model-graded rubrics only when the criterion genuinely needs language judgment, and calibrate them against human labels.

Compare one variable at a time

Promptfoo’s matrix is useful for comparing prompts or providers over the same tests. A valid experiment gives each candidate the same dataset, application context, decoding settings where controllable, and assertion rules. Name candidates so exported results can be traced back to immutable files or commits.

Do not switch prompt, model, retrieval corpus, tool definitions, and grader in one run and then attribute the difference to the prompt. If a combined release must change several components, first isolate them in smaller comparisons, then run an end-to-end candidate suite.

Review case-level flips:

Aggregate averages can hide one mandatory escalation failure. Hard gates should remain hard even when weighted quality scores improve.

Diagnose failures by replaying evidence

When a case fails, preserve rendered prompt, variables, provider, raw output, assertion results, and relevant latency or token metadata. First determine whether the failure is in input templating, provider call, output, or assertion.

Common patterns are diagnostic:

• Every case fails JSON after a prompt edit: output instruction or fence likely changed.
• One category fails across providers: dataset boundary or prompt rule is suspect.
• One provider fails all long inputs: context or request configuration needs inspection.
• Custom assertion throws: test code failure, not model quality.
• Reruns flip a hard decision: output contract or sampling is too unstable for the risk.

Read the actual output before tuning thresholds. Lowering a threshold to make a red suite green converts evidence into decoration.

Add performance and cost checks carefully

Promptfoo supports operational assertions such as latency and cost when the provider supplies the needed data. Use them only after establishing a representative environment and a business threshold. A laptop run over variable Wi-Fi is not a reliable service-level gate.

Operational checks can be grouped:

tests:
  - description: Interactive routing stays within the test budget
    vars:
      ticket: "My parcel is late."
    assert:
      - type: assert-set
        assert:
          - type: latency
            threshold: 2500
          - type: cost
            threshold: 0.01

The yaml block requires both configured thresholds to pass. The numerical values are illustrative test-budget inputs, not vendor guarantees. Replace them with thresholds derived from the application’s measured requirement, and investigate changes across repeated samples.

Put a focused suite in CI

CI should run deterministic, high-impact cases that are stable enough to block a merge: format, required escalation, forbidden disclosure, tool argument schema, and a few critical golden decisions. Keep expensive judge-based, broad multilingual, adversarial, and provider-comparison suites on a schedule or explicit release job.

Pin dependencies according to the team’s package policy. Fail on missing credentials rather than silently skipping providers. Store reports as access-controlled artifacts because prompts and outputs may contain sensitive product information. Record config commit, dataset version, provider configuration, and cache policy.

Use a release rule such as: zero failures in mandatory contract and safety metrics, no unexplained pass-to-fail flips in high-impact cases, and reviewed quality distribution for subjective metrics. “Overall score above 80%” is too weak when the missing 20% contains the legal-action escalation.

Grow the suite from production evidence

After release, review misroutes, human overrides, parse failures, user complaints, and tool errors. Convert a reviewed incident into the smallest case that reproduces the decision boundary. Tag the case with its provenance and expected fix, then verify that it fails before the change and passes after it.

Retire or revise cases when product policy changes, but keep history so a changed oracle is not mistaken for model improvement. Periodically sample passing outputs to detect weak assertions. Promptfoo supplies repeatable execution and structured results. QA still owns the dataset, oracle, failure analysis, and release decision.