How to Write Evals for an LLM

"The new prompt feels better" is not a release gate. How to write evals for an LLM turns fuzzy taste into measured comparison on a fixed task set so model upgrades, prompt edits, and retrieval tweaks become reviewable with numbers, not vibes.

This guide is a practical writing manual: define tasks, capture failure modes, design cases, choose scorers, set thresholds, wire CI, and avoid the traps that produce false confidence. It complements deeper metric theory in LLM evaluation metrics and dataset craft in building LLM eval datasets.

What "Writing an Eval" Actually Produces

When people say "we need evals," they often mean one of four artifacts:

1. A task definition: what success means for a product behavior.
2. A dataset: inputs (and often contexts or tools) with expectations.
3. A scorer: code, rubric, or judge that turns outputs into scores.
4. A report + gate: aggregates, slices, and pass/fail thresholds.

How to write evals for an LLM is the skill of building those four pieces so they stay honest as the system changes.

An eval is not:

• A single demo conversation you liked.
• A leaderboard number from an unrelated public benchmark alone.
• A unit test that only checks HTTP 200 from the model API.

An eval is closer to a test suite for behavior that is partly non-deterministic and often graded rather than binary.

Start From Product Behavior, Not From a Framework

Frameworks such as DeepEval, Ragas, or custom harnesses help, but they do not invent your product risks. Start with:

• Who uses the feature?
• What task are they trying to finish?
• What does "wrong" cost (money, trust, safety, support load)?
• Which failures already happened in staging or production?

Example product behaviors:

• Answer policy questions using only the help center.
• Classify support tickets into queue and priority.
• Extract fields from invoices into JSON.
• Draft a reply a human agent will edit.
• Call tools to look up orders and never invent order state.

Each behavior needs its own eval slice. A single global "quality" score usually hides important fails.

Step-by-Step: How to Write Evals for an LLM

Step 1: Write a task contract

In one paragraph, define:

• Input type.
• Allowed context (RAG docs, tools, memory).
• Required properties of a good output.
• Forbidden behaviors.
• Severity of common failure modes.

Example contract for a policy assistant:

Given a customer question and retrieved help-center chunks, the assistant answers in plain language, cites sources when making policy claims, refuses when context is insufficient, and never invents fee amounts. It does not provide legal advice beyond published policy text.

That contract becomes the source of rubric lines and dataset tags.

Step 2: Inventory failure modes

List how the system actually fails:

Failure mode	Example	Severity
Hallucinated fact	Invents a restocking fee	High
Unfaithful to context	Contradicts retrieved policy	High
Irrelevant answer	Talks about shipping when asked about billing	Medium
Over-refuse	Refuses a normal public policy question	Medium
Format break	Invalid JSON for extractor	High for API
Tool misuse	Calls refund without confirmation	Critical
Prompt injection	Follows hostile doc instruction	Critical
Latency / cost blowup	40 tool calls for a simple FAQ	Medium

Your first evals should target high severity modes first.

Step 3: Choose case types

A healthy suite mixes:

• Gold happy paths: known good answers or structured outputs.
• Edge cases: empty input, huge input, typos, mixed languages.
• Adversarial cases: injection, jailbreak, abuse of tools.
• Ambiguous cases: should ask clarifying question or refuse.
• Regression cases: every production incident becomes a row.
• Distribution slices: new users, power users, VIP tenants, locales.

Step 4: Write individual eval cases

Use a strict row schema.

{
  "id": "policy-refund-012",
  "feature": "policy_assistant",
  "input": {
    "question": "Can I return a used headset after 20 days?"
  },
  "context": {
    "docs": ["returns_policy_v3.md"]
  },
  "expect": {
    "must_include_concepts": ["20-day window", "unused or defective rules"],
    "must_not_claim": ["free returns forever", "restocking fee 50%"],
    "require_citations": true,
    "allow_refuse_if_missing_context": false
  },
  "rubric": {
    "correctness": "Answer matches policy for used item after 20 days",
    "faithfulness": "No fees or windows not present in docs",
    "helpfulness": "Clear next step for the customer"
  },
  "tags": ["returns", "boundary", "high_severity"],
  "source": "prod_ticket_88421"
}

Write the expectation so two reviewers would score the same way. Vague expect fields create judge noise.

Step 5: Pick scorers that match the output shape

Output shape	Prefer	Avoid as sole score
Enum / label	Exact match, confusion matrix	Open-ended judge only
JSON extract	Schema validation + field compares	BLEU
Short factual answer	Normalized match, numeric tolerance	Exact string only
Open chat answer	Rubric + faithfulness + human sample	Single vibe score
Tool-using agent	Trajectory + final state asserts	Final text only
RAG answer	Faithfulness, context precision/recall, relevancy	Generic helpfulness alone

For metric choices in depth, use LLM evaluation metrics. For RAG-specific scoring, see Ragas RAG evaluation. For a hands-on framework path, see the DeepEval tutorial.

Step 6: Write rubrics that are observable

Bad rubric:

Score 1-5 for quality.

Better rubric:

Correctness (0/1):
1 if the refund window and condition rules match the policy docs.
0 if any window, fee, or eligibility rule is wrong or invented.

Faithfulness (0/1):
1 if every policy claim is supported by provided context.
0 if any material claim lacks support.

Actionability (0-2):
2 clear next step and what user should prepare
1 partial next step
0 no usable guidance

Safety (0/1):
0 if the answer invents legal guarantees or overrides policy
1 otherwise

Release blocker if Correctness=0 or Faithfulness=0 or Safety=0.

Binary gates for blockers plus graded dimensions for product quality is a strong pattern.

Step 7: Calibrate judges and humans

If you use LLM-as-judge:

1. Label 30-50 cases with humans.
2. Run the judge.
3. Measure agreement on blocker dimensions.
4. Fix rubric ambiguity before trusting CI.
5. Re-calibrate when the judge model changes.

Never let an uncalibrated judge be the only release gate for high-risk domains.

Step 8: Set thresholds and slices

Examples:

• Overall correctness >= 0.90 on gold set.
• Faithfulness >= 0.95 on RAG set.
• JSON schema validity == 1.0 on extractor set.
• Critical safety cases == 100% pass.
• p95 latency <= product budget on smoke path.
• No regression worse than -2 points on any high-severity tag without review.

Always report by tag (feature, locale, severity), not only a single average.

Step 9: Version everything

Version:

• Dataset rows and tags.
• Prompt templates.
• Model name and pin.
• Tool schemas.
• Judge prompt and judge model.
• Threshold config.

When a score drops, you need to know what changed. Store eval run metadata next to the score.

Step 10: Wire into development workflow

Recommended cadence:

Trigger	Suite	Goal
PR changing prompts/tools	Smoke 30-50 cases	Catch obvious breaks
Model pin change	Full core suite	Compare candidates
Nightly	Full + adversarial	Drift and flaky fails
Incident	Add case + rerun	Prevent repeat

This is the operational heart of prompt regression testing.

Worked Example: Writing Evals for a Ticket Classifier

Task contract

Given a support ticket subject and body, return JSON {"queue": "...", "priority": "low|medium|high", "reason": "..."}. Queues are billing, shipping, product, account. Priority high only for fraud, outage, or data loss signals.

Seed cases

ID	Input signal	Expected queue	Expected priority
CL-01	"charged twice"	billing	medium
CL-02	"package never arrived 30 days"	shipping	medium
CL-03	"I see another user's invoices"	account	high
CL-04	"button color ugly"	product	low
CL-05	empty body	account or product per policy	low + needs clarification flag if designed

Scorers

def score_classifier(output, expected):
    errors = []
    if not is_valid_schema(output):
        return {"pass": False, "errors": ["invalid_json"]}
    if output["queue"] != expected["queue"]:
        errors.append("queue_mismatch")
    if output["priority"] != expected["priority"]:
        errors.append("priority_mismatch")
    # reason is free text: graded separately
    return {
        "pass": len(errors) == 0,
        "errors": errors,
        "queue_ok": output["queue"] == expected["queue"],
        "priority_ok": output["priority"] == expected["priority"],
    }

Rubric for reason text (optional)

Judge only whether the reason cites a real cue from the ticket, not prose beauty.

Threshold

• queue accuracy >= 0.93
• priority accuracy >= 0.90
• schema validity == 1.0
• high priority recall on safety-like tickets >= 0.98

This is a complete mini eval: contract, cases, scorer, thresholds.

Writing Evals for Open-Ended Chat

Open-ended systems need more structure, not less.

Pattern A: Reference answer + semantic check

Store a reference answer or bullet facts. Score coverage of required facts and absence of forbidden claims.

Pattern B: Checklist rubric

For each case, a checklist of 3-7 observable items.

Pattern C: Pairwise preference

Compare model A vs B on the same input with a frozen judge prompt. Useful for migrations, dangerous as the only gate if the judge is biased.

Pattern D: User outcome proxy

Where possible, score task success (form filled, issue resolved category) instead of eloquence.

Writing Evals for RAG and Tools

RAG

Each case should include:

• Question.
• Corpus version or doc IDs.
• Gold answer or gold facts.
• Whether the answer is answerable from corpus.

Score retrieval and generation separately when you can. A wrong answer with perfect retrieval is a generator bug. A right-looking answer with empty retrieval is a hallucination risk.

Tools / agents

Eval rows need environment setup and final state asserts:

• Correct tool chosen.
• Arguments valid.
• Side effect once.
• Final answer consistent with tool output.
• No invented tool results.

Text-only scoring is insufficient for agents.

How Detailed Should Expected Outputs Be?

Situation	Expectation style
Structured extraction	Exact fields + tolerances
Classification	Exact label
Math / IDs	Exact or numeric tolerance
Policy Q&A	Required facts + forbidden claims
Creative copy	Style rubric + brand constraints
Safety	Hard refuse / safe complete criteria

Over-specifying wording makes evals brittle. Under-specifying makes them meaningless. Prefer facts, constraints, and structure over full golden essays unless the product must match a template.

Human Review Still Belongs in the Loop

Write evals so humans can audit:

• Sample fails weekly.
• Review all critical safety fails.
• Double-label disputed rubrics.
• Track inter-rater agreement on graded dimensions.

Automation scales. Humans keep the suite honest.

Common Mistakes When Writing LLM Evals

Mistake 1: One average score for everything

A 4.2 "quality" average can hide a 20% hallucination rate on refunds. Slice by tag and severity.

Mistake 2: Exact match on free-form answers

You will flunk good paraphrases and pass lucky wording. Use constraints and rubrics.

Mistake 3: Dataset of only happy demos

Demos do not catch production edges. Add messy real tickets and adversarial rows.

Mistake 4: Judge prompt as vague as the product prompt

"Be strict but fair" is not a rubric. Write observable criteria.

Mistake 5: No versioning

You cannot explain a 5-point drop if prompts, models, and data all moved.

Mistake 6: Thresholds set by vibes on launch day

Set thresholds from baselines and risk tolerance, then revisit with data. Do not freeze an accidental number forever without review.

Mistake 7: Evaluating only the model, not the product

User-facing systems include retrieval, tools, memory, UI truncation, and post-filters. Eval the pipeline users get.

Mistake 8: Ignoring cost and latency

A slightly better answer at 8x cost may be a product failure. Include operational metrics in the eval report.

Mistake 9: Never deleting bad cases

Wrong gold labels poison trust. Have a process to fix or retire rows.

Mistake 10: Treating public benchmarks as product evals

MMLU-style scores do not prove your invoice extractor works. Use them as side context, not release gates for product behavior.

Checklist: Before You Call the Suite "Done"

• Task contracts exist for each major behavior.
• Failure modes are listed and tagged.
• Cases include happy, edge, adversarial, and regression.
• Expectations are specific enough for agreement.
• Scorers match output shape.
• Blocker dimensions are binary where needed.
• Slices and thresholds are defined.
• Dataset and prompts are versioned.
• Smoke suite runs on relevant PRs.
• Humans calibrate judges on a sample.
• Production incidents feed new cases.
• Reports are readable by eng and product.

Putting It Into Practice on a Team

A lightweight operating model:

1. Owner: one DRI for eval health per AI feature.
2. Budget: every feature PR that changes model behavior updates or adds cases.
3. Review: eval diffs reviewed like code.
4. Cadence: weekly fail triage meeting (30 minutes).
5. Visibility: dashboard with slices, not only green checks.

Practice writing tight cases under time pressure helps. Use QABattle battles or create an account to rehearse structured QA thinking, then apply the same clarity to eval rows.

Offline Evals, Online Evals, and When to Use Each

Writing offline evals is only half the system.

Offline evals

Run on curated datasets before release. Strengths: reproducible, comparable, good for CI. Weaknesses: distribution can drift from live traffic.

Online evals

Measure production with implicit signals (task completion, retries, escalations), explicit feedback (thumbs down), safety filters, latency, and cost. Strengths: real distribution. Weaknesses: noisier, slower feedback, privacy constraints.

A practical split:

Question	Prefer
Did this prompt regress gold cases?	Offline
Do users abandon after model upgrade?	Online
Are refund answers still faithful?	Offline + sampled online review
Is latency within SLO?	Online and offline smoke

When you write evals, reserve fields for later online joins: feature flag, model pin, prompt version. That makes production debugging possible.

Writing Evals for Safety and Policy

Safety cases deserve their own pack with zero-tolerance thresholds.

Categories:

• Disallowed assistance requests.
• Self-harm and violence policies per product rules.
• Prompt injection and secret exfiltration.
• Bias and harassment edge cases where relevant.
• Regulated claims (medical, financial) outside product scope.

Write these cases with legal/safety partner input when the domain requires it. Do not invent policy. Encode the policy your product already claims.

Scoring should be binary on blockers. A 4/5 "mostly safe" score is the wrong abstraction for a critical violation.

Example: Expanding a Weak Eval Into a Strong One

Weak eval row:

Question: Tell me about returns.
Expected: Good answer about returns.

Strong eval row:

ID: ret-017
Question: I bought wireless earbuds 25 days ago online and opened the box. Can I return them?
Context docs: returns_v4.md (30 days, unopened OR defective after open)
Gold facts:
  - Within 30-day window
  - Opened non-defective consumer electronics may be restocked or denied per policy text
Must not claim:
  - Free returns with no conditions
  - 14-day only window
Require:
  - Faithfulness to provided policy
  - Clear next step (portal or support)
Severity: high
Source: support_theme_returns_open_box

The strong version is reviewable, automatable, and tied to risk. That is the standard to aim for whenever you write evals for an LLM product surface.

Final Workflow You Can Reuse

When someone asks you how to write evals for an LLM, run this loop:

1. Name the user-visible behavior.
2. Write a task contract and forbidden behaviors.
3. List failure modes by severity.
4. Draft 20 seed cases with tags.
5. Choose deterministic scorers first, judges second.
6. Calibrate on a labeled sample.
7. Set blocker thresholds and slice reports.
8. Version dataset, prompt, model, judge.
9. Wire smoke CI + full release suite.
10. Grow the set from incidents and new risks.

Good evals are not a one-time science project. They are a living test suite for probabilistic products. Write them with the same seriousness you give API contract tests, and your prompt changes will finally become engineering instead of folklore.