DeepEval Tutorial: Unit Testing for LLM Applications

This DeepEval tutorial is for QA engineers and developers who want unit testing habits for LLM applications. Classical unit tests assert exact values. LLM features need a different shape of test: an input, an actual model output, optional retrieval context or expected behavior, and metrics that score quality with thresholds you can enforce in CI.

DeepEval is built around that workflow. It helps you express LLM quality checks in a pytest-style manner so prompt edits, model swaps, and chain changes get automatic feedback instead of hallway opinions.

What DeepEval Is and Why It Exists

LLM apps fail in ways that expect(text).toBe(gold) cannot handle well:

• wording changes while meaning stays correct
• answers that sound right but invent facts
• RAG systems that retrieve the wrong context
• agents that call the wrong tool
• subtle tone or policy violations

DeepEval approaches these as evaluation test cases with metrics. Instead of only comparing strings, you score behaviors such as:

• faithfulness to context
• answer relevancy
• similarity to expected output
• custom rubric judgments via G-Eval style metrics
• task-specific checks you compose into a suite

If you need the conceptual map of metric families first, read LLM evaluation metrics. This tutorial focuses on practice.

Core Concepts

Test case

A test case usually includes:

• input: the user prompt or request
• actual_output: what your app produced
• expected_output: optional reference answer
• retrieval_context or context list: optional documents for groundedness checks
• metadata such as scenario tags

Metric

A metric turns a test case into a score, often between 0 and 1, plus a reason or debug details.

Threshold

You decide what “pass” means. For example, faithfulness must be at least 0.7 on the critical policy set.

Suite

A suite is a collection of cases and metrics that protect a feature or risk area.

Mental Model: Pytest for LLM Outputs

Classic test:

def test_total():
    assert cart_total([10, 5]) == 15

LLM-oriented test:

def test_refund_policy_answer():
    output = answer_question("What is the refund window?")
    # evaluate output with metrics against policy context
    assert faithfulness_score(output, context) >= 0.7
    assert relevancy_score("What is the refund window?", output) >= 0.7

DeepEval formalizes that second pattern so you are not hand-rolling every judge prompt and score parser from scratch.

Setup Workflow

High-level setup steps:

1. Create or open your Python project for the LLM app or eval harness.
2. Install DeepEval in the environment your CI will use.
3. Configure any model provider keys needed for LLM-as-judge metrics.
4. Decide which app entrypoint the tests will call.
5. Build a small dataset of inputs and contexts.
6. Write one smoke metric test.
7. Add thresholds and suite structure.
8. Wire a CI job with secrets and cost controls.

Exact package versions change over time, so follow current DeepEval docs for install pins. The testing design in this tutorial stays stable even when APIs get minor renames.

Designing Your First DeepEval Cases

Start with five to fifteen cases, not five hundred.

Good first cases:

1. Simple known FAQ with clear source context.
2. Question that should refuse when context is insufficient.
3. Adversarial prompt that tries to override policy.
4. Verbose user question with the same intent as case 1.
5. Near-miss retrieval context that is related but wrong.
6. Multilingual or messy punctuation if your users write that way.
7. Structured output requirement if your app returns JSON.

Each case should have a name, intent, and owner. Treat the dataset as product code.

DeepEval Faithfulness Metric Example

Faithfulness is one of the highest value metrics for RAG and knowledge-grounded assistants.

Scenario:

• User asks about password reset timing.
• Retrieved context says reset links expire in 30 minutes.
• Model answers 24 hours.

A faithfulness metric should score that answer poorly because it contradicts context.

Case design

input: "How long is the password reset link valid?"
retrieval_context:
  - "Password reset links expire 30 minutes after they are issued."
actual_output: "Your reset link is valid for 24 hours."
expected metric: faithfulness below threshold

What you are asserting

Not “the wording matches.” You are asserting “claims are supported by context.”

Implementation pattern

# Pseudocode-style example for teaching the pattern
from my_app import answer_with_rag
from eval_harness import LLMTestCase, FaithfulnessMetric, assert_metric

def test_reset_link_faithfulness():
    question = "How long is the password reset link valid?"
    context = [
        "Password reset links expire 30 minutes after they are issued."
    ]
    output = answer_with_rag(question)

    case = LLMTestCase(
        input=question,
        actual_output=output,
        retrieval_context=context,
    )
    metric = FaithfulnessMetric(threshold=0.7)
    assert_metric(case, metric)

In real projects, prefer the official DeepEval class names and assert helpers from the version you installed. The pattern above is what matters: app output in, metric threshold enforced, context provided.

Answer Relevancy and Expected Output Checks

Faithfulness alone is not enough. An answer can stick to context and still miss the question.

Add relevancy when:

• users ask multi-intent questions
• models ramble into adjacent topics
• retrieval returns broad documents

Use expected output similarity when:

• there is a stable gold answer
• the task is constrained
• you are regression-testing a carefully edited response style

Avoid forcing exact string equality on open-ended answers unless the product truly requires fixed text.

DeepEval G-Eval Metric Tutorial

G-Eval style metrics shine when quality is rubric-based.

Example criteria for a support assistant:

Score the answer from 0 to 1.
1.0 = fully correct, policy-aligned, clear next step, no unsupported claims.
0.5 = partially helpful but missing a critical detail or mildly unclear.
0.0 = wrong, unsafe, or invents policy.

Deduct heavily if the answer invents deadlines or refund rules.

When to use G-Eval

• tone requirements
• pedagogy quality for tutoring apps
• custom domain correctness not captured by generic metrics
• “did the agent ask a clarifying question when needed?”

When not to use G-Eval alone

• simple schema validation (use deterministic checks)
• exact safety denylist (use explicit rules)
• cheap smoke tests where a smaller metric works

Making G-Eval reliable

1. Include few-shot graded examples in the rubric when possible.
2. Keep the judge model fixed in CI.
3. Store rationales for failed cases to speed debugging.
4. Calibrate thresholds against human labels.
5. Separate custom rubrics by feature so failures are actionable.

Pytest-Style LLM Evaluation With DeepEval

A maintainable layout:

tests/
  eval/
    test_support_faithfulness.py
    test_support_refusals.py
    test_sql_agent_smoke.py
    data/
      support_cases.jsonl
      sql_cases.jsonl

Tips for pytest discipline

• mark expensive evals with @pytest.mark.eval
• run smoke evals on every PR
• run full evals nightly
• fail fast on deterministic checks before calling judges
• cache fixtures where safe
• record model and prompt versions in reports

Example organization:

import pytest

pytestmark = pytest.mark.eval

@pytest.mark.parametrize("case", load_cases("support_cases.jsonl"))
def test_support_suite(case):
    output = run_support_bot(case["input"])
    evaluate_case(case, output)

Parametrization keeps the dataset data-driven while still producing readable failures.

Comparing Prompt or Model Versions

DeepEval suites are most valuable when they compare variants.

Workflow:

1. Freeze a dataset version.
2. Run suite on baseline prompt/model.
3. Run suite on candidate prompt/model.
4. Compare pass rates and score distributions by tag.
5. Inspect failures, not only averages.
6. Ship only if critical tags pass thresholds and overall deltas are acceptable.

Tags might include billing, auth, safety, multilingual, rag_hard.

DeepEval vs Ragas

Teams often ask which tool to pick.

Dimension	DeepEval	Ragas
Primary framing	Unit testing for LLM apps	RAG evaluation toolkit
Developer ergonomics	Strong pytest-style testing angle	Strong research/eval pipelines for RAG
Metric breadth	Broad app metrics including custom judges	Deep focus on RAG quality metrics
Best starting point	App-level regression suites	Retrieval and grounded answer diagnosis
Common pairing	Product CI gates	RAG improvement loops

Practical recommendation:

• Use DeepEval when you want engineer-owned tests around an application.
• Use Ragas when you are deeply iterating retrieval quality and need RAG metric focus.
• Many teams evaluate RAG with both perspectives over time.

For the RAG-centered walkthrough, see RAGAS: Evaluating RAG Pipelines.

Integrating DeepEval Into CI

PR pipeline

• deterministic tests: schema, tool-allow lists, unit logic
• small DeepEval smoke set: 10-30 critical cases
• budget guardrails for judge tokens

Nightly pipeline

• larger regression set
• multi-model comparisons if needed
• publish a simple HTML or markdown report
• open issues for new recurring failures

Secrets and cost control

• store provider keys in CI secrets
• cap max cases on forked PRs if needed
• skip judge metrics on docs-only changes
• pin model names used for judging

Debugging a Failed Metric

When faithfulness fails:

1. Read the actual output claims.
2. Read the retrieval context.
3. Check whether retrieval is wrong or generation is wrong.
4. Check if the threshold is unrealistically high for partial answers.
5. Check judge or metric configuration changes.

When G-Eval fails:

1. Read the rationale.
2. Confirm the rubric is unambiguous.
3. See if the judge is punishing style rather than substance.
4. Compare with a human rating on the same case.
5. Adjust rubric or app behavior deliberately, not both at once.

Combining Deterministic Checks With DeepEval Metrics

Hybrid tests are powerful.

def test_create_ticket_tool_agent():
    result = run_agent("Create a high priority ticket for login failures")
    assert result["tool_calls"][0]["name"] == "create_ticket"
    assert result["tool_calls"][0]["args"]["priority"] in {"high", "P1"}
    # then graded metric on user-facing confirmation message
    assert_relevancy(result["message"], "Create a high priority ticket for login failures")

Deterministic checks catch hard bugs cheaply. Metrics catch language quality and groundedness.

Dataset Hygiene for DeepEval Suites

Your suite is only as good as the cases.

Rules:

• prefer real anonymized traffic over synthetic-only sets
• include expected failure cases, not only happy paths
• refresh quarterly or after major product changes
• keep expected contexts current when docs change
• avoid training or few-shot contamination from test items
• track flaky cases separately

For dataset design depth, use building an LLM eval dataset.

Prompt Regression Testing With DeepEval

Every prompt edit should answer:

• What behavior are we trying to improve?
• Which suite tags should get better?
• Which tags must not get worse?

That is prompt regression testing in practice. DeepEval gives you the measurement harness; the discipline is not shipping prompt diffs without suite evidence. See also prompt regression testing.

Common Mistakes When Using DeepEval

Mistake 1: Thresholds Copied From Blog Posts

Your domain and risk tolerance define thresholds. Calibrate them.

Mistake 2: One Giant Suite With No Tags

When failures happen, you need to know whether billing, safety, or small talk regressed.

Mistake 3: Judging Without Fixed Model Settings

If the judge model or temperature drifts, scores are not comparable.

Mistake 4: No Deterministic Guards

Do not spend judge money to discover invalid JSON if a schema check would catch it instantly.

Mistake 5: Actual Output Not From the Real App Path

Testing a raw model call while production uses a different chain misses retrieval and tool bugs.

Mistake 6: Ignoring Cost and Runtime

A two-hour PR eval gate will be bypassed. Keep smoke suites lean.

Mistake 7: Treating Metrics as Product Truth Without Humans

Spot-check failures and successes. Metrics approximate quality; they do not replace responsibility.

Mistake 8: Freezing a Stale Dataset Forever

Old cases protect old product behavior. Update as docs, policies, and UX change.

Worked Mini-Project Plan

Build a first DeepEval harness in one day:

1. Morning: pick 12 support questions and policy contexts.
2. Midday: wire app entrypoint and one faithfulness test.
3. Afternoon: add relevancy and one G-Eval tone rubric.
4. End of day: run baseline scores and set provisional thresholds.
5. Next day: add CI smoke job and a failure triage doc.

By the end of two days, you have something far more useful than a slide saying “we will evaluate quality.”

Sample Score Review Table

Case tag	Baseline pass	Candidate pass	Decision
billing_policy	9/10	10/10	improved
auth_howto	8/10	8/10	stable
safety_refusal	10/10	9/10	investigate
rag_hard	6/10	7/10	improved
tone	8/10	7/10	product call

Never ship on an average alone when safety tags drop.

Release Checklist

• Critical smoke eval passes.
• No safety tag regressions.
• Faithfulness threshold holds on policy cases.
• Prompt and model versions recorded.
• Dataset version recorded.
• Judge configuration pinned.
• Cost and latency within budget.
• Failed cases triaged or accepted with rationale.
• Human reviewed a sample of changes for high-risk features.

Flaky Eval Quarantine Policy

Some graded metrics will flake under provider latency or rare judge variance. Use an explicit quarantine policy instead of silent deletions.

Rules that work well:

• A case may be quarantined only with an owner and a written reason.
• Quarantine expires in 14 days unless renewed.
• Safety and auth cases cannot be quarantined without lead approval.
• Quarantined cases still run nightly for visibility, but do not block PRs.
• Every quarantine ends in fix, rewrite, or permanent retirement with rationale.

This keeps CI trustworthy without pretending LLM evaluation is as deterministic as integer arithmetic.

Naming Conventions for Cases and Metrics

Clear names make failures actionable in CI logs.

Prefer:

• billing_refund_window_annual_faithfulness
• auth_reset_link_expiry_relevancy
• safety_prompt_injection_ignore_policies

Avoid:

• test1
• case_final_final
• misc_quality

Include the risk domain, scenario, and metric intent. When a nightly job fails, engineers should understand the business meaning before opening the artifact bundle.

Also version dataset files with dates or semantic versions, and record which dataset version produced a given CI report. Otherwise you cannot tell whether the product changed or the golden set changed.

Structured Output Testing Patterns

Many LLM apps must return JSON, markdown tables, or tool argument objects. DeepEval metrics should sit on top of strict structure checks.

Recommended order:

1. Parse JSON or schema. Fail immediately if invalid.
2. Check required keys and enums deterministically.
3. Run faithfulness/relevancy on user-visible fields.
4. Run G-Eval only for subjective residual quality.

Example hybrid case ideas:

• support bot returns { "reply": "...", "intent": "...", "needs_handoff": true/false }
• SQL copilot returns { "sql": "...", "warning": "..." }
• classifier returns label plus confidence band

This ordering keeps CI costs down and failure messages obvious.

Red-Team Slice Inside DeepEval

Reserve a tagged slice for abuse and policy tests:

• prompt injection attempting to ignore system rules
• requests for other users’ data
• jailbreak-style roleplay
• requests to reveal system prompts or hidden tools
• unsafe instructions relevant to your domain

These cases may use custom G-Eval rubrics or deterministic detectors. Keep them in PR smoke if the feature is user-facing and high risk. Do not bury all safety cases in a monthly report nobody reads.

Synthetic Data vs Production-Derived Cases

DeepEval suites often start synthetic because that is easy. Synthetic cases are useful for:

• bootstrap coverage
• rare safety scenarios
• controlled document contexts

They are weak when:

• real users use slang, typos, and multi-intent questions
• production retrieval corpora differ from toy contexts
• tool-argument quirks only appear in real workflows

A healthy mix:

Source	Share early	Share mature
Synthetic targeted cases	High	Medium
Anonymized production traces	Low	High
Incident regressions	Medium	Medium
Red-team adversarial prompts	Medium	Medium

Whenever a production failure escapes, add a minimized case to the suite within the same sprint if possible.

Local Dev Loop vs CI Loop

Developers need a fast local loop:

1. change prompt or chain
2. run a focused tag
3. inspect three failures
4. iterate

CI needs a trustworthy gate:

1. smoke tags on PR
2. full suite nightly
3. published comparison against main
4. owners notified on critical regressions

If local runs take as long as CI, people stop running them. Offer:

• make eval-smoke
• make eval-tag TAG=billing
• make eval-full

DeepEval is most effective when the commands are obvious and documented in the repo README.

Handling Non-Determinism Without Giving Up

LLM tests will not be as stable as pure function unit tests. That is acceptable if managed.

Tactics:

• lower temperature for eval runs when product allows
• average multiple runs only for research, not usually for PR gates
• keep critical assertions deterministic where possible
• quarantine known flaky cases with owners and expiry dates
• require re-run evidence before blaming infrastructure

Do not disable a safety case because it is inconvenient. Fix the product or improve the case design.

Team Roles Around DeepEval

Suggested ownership:

• Feature engineer: keeps cases for their feature green
• QA / eval owner: suite structure, thresholds, triage rituals
• Platform: CI secrets, cost budgets, report publishing
• Product: severity calls when tone metrics conflict with brevity goals
• Security / trust: adversarial and privacy cases

Without ownership, suites rot. With ownership, DeepEval becomes part of Definition of Done for AI changes.

Example Failure Triage Notes

Case: billing-refund-window-02
Metric: faithfulness 0.42 (threshold 0.70)
Actual: claimed 30-day refunds for monthly plans
Context: only annual plans have 14-day refunds; monthly non-refundable after use
Root cause: generator overgeneralized from adjacent annual policy chunk
Action: tighten prompt grounding; add explicit monthly non-refundable sentence to KB; keep case
Owner: payments-ai

Write triage notes like bug reports. Future you will need them.

Final Takeaways

DeepEval helps you bring software engineering discipline to LLM quality. The winning pattern is not “add a metric library.” The winning pattern is:

1. define risks
2. encode them as cases and metrics
3. threshold them in CI
4. debug failures like product defects
5. keep humans in the calibration loop
6. grow the dataset from production pain
7. assign owners so the suite cannot silently rot

Use faithfulness for grounded answers, relevancy for on-topic behavior, G-Eval for custom rubrics, and deterministic tests for hard constraints. Start small, tag everything, and treat eval failures with the same seriousness as failing unit tests, adjusted for probabilistic scoring.

If you want deliberate practice comparing outputs against rubrics, join QABattle and train the evaluation habit in the battle arena before you wire the same judgment style into automated suites.