Braintrust Evals Tutorial: Testing LLM Apps with Real Traces

A support assistant passed every demo, then told a customer that an expired refund exception was still available. The wording was polished and the answer cited a real policy page, but the cited section described a different product tier. The team had logged the conversation in Braintrust, yet had no experiment that could answer the release question: did the new prompt improve policy answers without increasing confident misapplication?

That is the useful starting point for Braintrust evals. An experiment is not a gallery of outputs. It is a controlled comparison between a candidate and a baseline, using named examples, explicit scorers, and traces that explain why scores moved.

Frame one decision before creating an experiment

Write the decision in operational terms. For this assistant, it might be: “Ship prompt version 18 if it reduces policy-selection errors, keeps required disclosures intact, and does not add unacceptable latency.” That statement determines the dataset and the scorers. “Make responses better” does not.

Record the evaluation contract beside the code change:

• Candidate: prompt v18, unchanged model and retrieval index.
• Baseline: production prompt v17, replayed on the same examples.
• Primary risk: applying a valid rule to the wrong plan, region, or date.
• Hard constraints: no fabricated URLs, required escalation language present, no unsupported refund promise.
• Guardrails: median and tail latency, token use, and error rate cannot regress beyond agreed budgets.

Freeze controllable inputs. Pin the model identifier, prompt revision, tool schemas, retrieval snapshot, and runtime settings for the comparison. The model can still vary, so use repeated runs only on borderline or high-risk cases rather than pretending one output is permanent truth.

Model the dataset around decisions and evidence

A useful Braintrust dataset contains more than a prompt and an ideal paragraph. Store what the evaluator needs to judge the behavior and what the investigator needs to reproduce it.

{
  "id": "refund-expired-enterprise-eu",
  "input": {
    "message": "Can you extend our refund window by two weeks?",
    "account": { "plan": "enterprise", "region": "EU" },
    "asOf": "2026-06-15"
  },
  "expected": {
    "decision": "escalate",
    "requiredFacts": ["standard window has expired"],
    "forbiddenClaims": ["extension is guaranteed"]
  },
  "metadata": {
    "risk": "high",
    "journey": "refunds",
    "sourceDoc": "refund-policy-2026-04",
    "caseOrigin": "production-review"
  }
}

Do not force every case into an exact reference answer. Structured expectations survive harmless wording changes and support deterministic checks. Keep a small set of canonical responses for tone comparison, but make policy decisions, required facts, prohibited claims, citations, and escalation outcomes first-class fields.

Balance ordinary traffic with decision boundaries. Include one day inside and outside a deadline, similar plan names, missing account data, conflicting user claims, terse messages, and follow-up turns. Tag every case so results can be sliced by risk, locale, journey, and origin. A single average can hide a complete collapse in a small but costly slice.

Separate the task from the scorers

In Braintrust terms, the task invokes the system under test and scorers judge its output. Keep those responsibilities separate. If a scorer quietly calls retrieval again or repairs malformed output, the experiment no longer measures the production path.

The following TypeScript is illustrative architecture, not a guaranteed copy-paste API for every Braintrust SDK version:

type EvalCase = {
  input: SupportInput;
  expected: {
    decision: "answer" | "escalate";
    requiredFacts: string[];
    forbiddenClaims: string[];
  };
};

async function task(testCase: EvalCase) {
  const result = await runSupportAssistant(testCase.input);
  return {
    text: result.text,
    decision: result.decision,
    citations: result.citations,
    traceId: result.traceId,
    usage: result.usage,
  };
}

function decisionMatch(output: Awaited<ReturnType<typeof task>>, expected: EvalCase["expected"]) {
  return Number(output.decision === expected.decision);
}

Have the task return structured evidence even if the user sees prose. A final answer alone cannot reveal whether the wrong document was retrieved, the right document was ignored, or a tool result was overwritten during synthesis.

Use a scorer stack, not one magic grade

Start with deterministic scorers because their failures are unambiguous. Check schema validity, decision equality, citation membership, required disclosure presence, forbidden claim absence, tool error count, and maximum token budget. These checks should explain exactly what failed.

Add model-graded scoring only where the requirement is semantic, such as whether the response distinguishes an exception from a guarantee. Give the grader the user input, relevant policy excerpt, candidate answer, and a narrow rubric. Ask for a category and rationale, not an unexplained number.

{
  "label": "unsupported_commitment",
  "allowed": ["pass", "minor", "major"],
  "majorWhen": "The answer promises an outcome not supported by the supplied policy.",
  "minorWhen": "The answer is cautious but omits a qualification that does not change the action.",
  "passWhen": "Every commitment is supported and uncertainty is explicit."
}

Calibrate this grader against a human-labeled set before trusting it in CI. Review disagreements, especially false passes. A model grader is another probabilistic component, not an oracle. For high-risk policy cases, a deterministic critical-failure flag should override a favorable style or helpfulness score.

Read score changes through traces

When a candidate loses three points, open the traces before changing the prompt. Braintrust tracing is valuable because it connects the experiment row to model calls, retrieval, tools, timing, and metadata. Use that evidence to classify the failure stage.

For the refund example, a bad answer can come from distinct defects:

• Retrieval selected the consumer policy instead of the enterprise policy.
• The prompt failed to pass the account region into the decision context.
• The model saw the correct excerpt but converted “may approve” into “will approve.”
• The final formatter dropped the escalation sentence.
• A retry produced a valid answer but doubled latency and cost.

These require different owners and fixes. Attach the trace ID, dataset case ID, candidate revision, retrieved document IDs, and scorer rationales to the defect. Screenshots of the final text are weak evidence because they discard the execution path.

Compare experiments by slices and paired outcomes

Run baseline and candidate over identical dataset versions. Compare each case as a pair: fixed, regressed, unchanged pass, or unchanged fail. Paired changes are more actionable than comparing two aggregate percentages from different samples.

Inspect at least these views:

• Critical cases where any major policy error blocks release.
• Decision-boundary cases by plan, region, and effective date.
• Production-derived cases versus synthetic coverage.
• Short single-turn questions versus follow-up conversations.
• Cases with retrieval, tool, or parsing failures.

Suppose overall policy accuracy rises from the baseline, but EU enterprise cases lose two critical examples. The average is not permission to ship. Conversely, a tiny aggregate decline caused by stricter wording on low-risk cases may be acceptable if critical outcomes improve. Thresholds should encode business impact, not worship a global score.

Rerun a sample of changed outcomes to estimate instability. If a case flips repeatedly, flag it as flaky and investigate sensitivity to retrieval ties, nondeterministic tools, or ambiguous labeling. Do not simply average away the uncertainty.

Turn production traces into a review loop

Offline datasets age. Sample production traces using risk-based rules: low-confidence retrieval, user corrections, escalations after an answer, tool retries, long conversations, and unusually expensive calls. Remove or protect personal data before adding examples to a durable dataset.

Route sampled traces to human reviewers with a compact rubric. Product specialists judge policy application, QA judges reproducibility and coverage, and engineering verifies execution evidence. Reviewers should be able to mark an example as ambiguous rather than forcing a pass or fail. Ambiguous cases belong in label cleanup, not the release denominator.

Promote confirmed failures into the regression dataset with the smallest context that preserves the defect. Record why the case exists and which release introduced it. This creates a useful loop: production trace, reviewed failure, curated example, experiment, fix, and permanent regression coverage.

Build a release gate that tolerates variance

Use two gates. A fast pull-request suite can run deterministic scorers on a risk-weighted subset. A pre-release experiment can run the complete dataset, semantic graders, repeated high-risk cases, and human review of new regressions.

An example release policy could be:

• Zero new major failures in the critical slice.
• No more than a two-case paired regression in any supported locale, with product sign-off required for exceptions.
• At least 90 percent of previously failing production cases fixed or explicitly waived.
• P95 end-to-end latency no more than 15 percent above baseline on the same environment.
• Mean input plus output tokens no more than 10 percent above baseline unless the quality gain has an approved cost case.

The numbers must come from your service objectives and traffic economics, not from this example. Store waivers with an owner and expiry date. A release gate becomes meaningless when exceptions live only in chat.

Report a decision, not an experiment screenshot

The final report should name the dataset version, baseline and candidate revisions, run conditions, paired outcome counts, critical failures, slice regressions, latency and usage changes, flaky cases, and reviewer decisions. Link each blocker to its trace.

End with one of three outcomes: ship, hold, or ship with a bounded waiver. For the refund assistant, “hold because two EU enterprise cases changed from correct escalation to unsupported commitment” is defensible. “The average score is 0.87” is not. Braintrust supplies the experiment and trace evidence; the team still has to define what evidence is sufficient for release.