Generate RAG Goldens with DeepEval Synthesizer

DeepEval Synthesizer RAG goldens are candidate evaluation records generated from documents or prepared context groups. They are not trusted test data at creation time. Preserve source provenance, review answerability and expected output, remove duplicates, version the approved set, and run every approved input through the RAG application before scoring its actual output.

That distinction separates dataset production from application evaluation. DeepEval's Synthesizer documentation describes several generation paths, while the repository challenges in src/db/seed-data/challenges/ai-evals.ts treat golden-set quality, freshness, coverage, and leakage as test risks. The retrieval cases in src/db/seed-data/challenges/ai-for-qa-wave2.ts add a second warning: a clean expected answer cannot compensate for context that was split, ranked, or associated incorrectly.

What are DeepEval Synthesizer RAG goldens?

A golden is an evaluation case that records the input and the reference information needed to judge an application response. For RAG, that reference usually includes one or more source contexts, an expected answer or expected behavior, and metadata that says where the evidence came from. DeepEval Synthesizer can create candidate goldens from supplied material, but generation does not make a candidate correct.

The word "golden" can sound final. Treat it as a role in a governed dataset, not a guarantee from the generator. A candidate becomes approved only after a reviewer can answer four questions:

1. Is the input meaningful for the application and intended users?
2. Can the expected behavior be supported by the preserved source context?
3. Does the case add a distinct risk, intent, boundary, or regression?
4. Can another engineer reproduce its origin and review decision?

This narrower topic differs from a general DeepEval tutorial, which introduces test cases, metrics, and framework use. It also differs from building LLM evaluation datasets, which covers tool-independent dataset design. Here the focus is the production line from DeepEval Synthesizer input to an approved RAG case.

For a single-turn case, generation and execution remain separate. Synthesizer does not call the RAG application to populate its actual_output. The application run must retrieve its own context and generate the response. That separation lets the evaluation compare a controlled reference boundary with what the current retriever and generator actually did.

When should you generate goldens from documents or contexts?

Generate from documents when the source collection is authoritative, available to the generation process, and needs to be transformed into context before questions can be proposed. This path is useful when the team wants DeepEval to handle document ingestion as part of candidate generation. The official generation from documents guide should be checked against the installed version before adopting its loader and configuration examples.

Generate from contexts when chunking and grouping are already controlled. Context input gives the team direct ownership of which passages belong together. That is valuable for table rows that depend on headers, rules split across neighboring paragraphs, or multi-hop evidence that must remain traceable. DeepEval documents context input as groups of strings in its generation from contexts guide.

Other paths serve different purposes:

The choice is not a contest between automation levels. Select the path that makes the intended evidence boundary inspectable. When chunk behavior itself is under test, do not generate only from final vector-store chunks and then claim the dataset covers chunking. Add dedicated chunk-boundary ablation tests that compare controlled representations of the same source.

Existing production failures also deserve direct human curation. Synthetic generation can add nearby variants, but the incident record should retain its own provenance and expected decision. The repository's golden-dataset challenges explicitly favor turning observed failures into permanent regressions instead of relying on occasional manual retesting.

What should a RAG golden contain?

Use a schema that supports diagnosis before choosing a metric. At minimum, an approved RAG golden needs a stable case ID, input, expected behavior, reference context, source identifiers, provenance, risk labels, review status, and dataset version. An expected output is useful when the acceptable answer is specific, but an expected behavior can be better for refusal, clarification, or policy routing.

A practical record can include:

• case_id: stable across result files and review tools.
• input: the exact user question or instruction.
• expected_output: a reviewed reference answer when appropriate.
• expected_behavior: answer, refuse, clarify, cite, or route.
• context: the evidence supplied during candidate construction.
• source_ids: document, version, page, section, and chunk identifiers.
• origin: document synthesis, context synthesis, human authoring, or incident.
• risk_slice: ordinary, boundary, adversarial, stale source, multi-hop, or another owned category.
• review: reviewer, status, reason, and review date.
• dataset_version: an immutable release identifier.

Do not substitute the reference context for the retrieval context captured during the application run. They answer different questions. Reference context describes what supports the case. Retrieval context shows what the current system found. Comparing them helps separate retrieval loss from generation failure, a distinction developed further in layered RAG evaluation architecture.

Multi-hop cases need explicit evidence relationships rather than a larger text blob. Record which source supports each step and which conclusion requires combining them. Evidence-chain labels are the appropriate next layer when a single context list cannot explain the expected reasoning path.

How do you prepare source documents and context groups?

Start with an inventory of approved sources. Give every document an owner and version, then remove unsupported formats, duplicate exports, secrets, personal data, and content that the application is not authorized to use. A synthetic question can expose source material in its wording, so generation deserves the same data-handling review as indexing.

Context groups should preserve meaning, not merely satisfy a character count. Keep a policy heading with its exceptions, a table header with relevant rows, and a definition with the clause that limits it. At the same time, do not place an entire manual into every group. Overloaded context can produce generic questions and hide whether a case depends on one specific source.

The preparation workflow is:

1. Inventory authoritative sources and assign stable versions.
2. Normalize text while preserving headings, tables, and source locations.
3. Create context groups around one answerable evidence unit.
4. Build one source_files label per context group and retain the detailed source-ID mapping beside it.
5. Mark groups that test multi-hop, ambiguity, refusal, or stale content.
6. Generate candidates into a staging dataset, never the approved dataset.
7. Preserve generation configuration and source versions with the run.
8. Reject any source group whose authorization or provenance is uncertain.

The retrieval seams in w2-chunking-retrieval-failures under src/db/seed-data/challenges/ai-for-qa-wave2.ts make the quality bar concrete: exact identifiers, table headers, and facts spanning boundaries need deliberate fixtures. Broader coverage guidance appears in test query decomposition in multi-hop RAG, but context preparation must come first.

How do you generate RAG goldens with DeepEval?

Install and configure the version used by the project, keep credentials outside the dataset, and start with a small source slice that reviewers understand. Current DeepEval separates generation controls across the Synthesizer constructor and the selected generation method. FiltrationConfig controls synthetic-input quality retries, EvolutionConfig controls how candidates are transformed, and method arguments control provenance, expected-output generation, and candidate volume.

from deepeval.synthesizer import Evolution, Synthesizer
from deepeval.synthesizer.config import EvolutionConfig, FiltrationConfig

context_groups = [
    [
        "Refund requests are accepted within 30 calendar days of delivery.",
        "Final-sale items are excluded unless they arrived damaged.",
    ],
    [
        "Account recovery requires a verified email address.",
        "Support must not disclose whether an unverified address has an account.",
    ],
]
source_files = [
    "refund-policy-v4.md",
    "account-recovery-v2.md",
]

synthesizer = Synthesizer(
    filtration_config=FiltrationConfig(
        synthetic_input_quality_threshold=0.7,
        max_quality_retries=3,
    ),
    evolution_config=EvolutionConfig(
        evolutions={
            Evolution.MULTICONTEXT: 0.25,
            Evolution.CONCRETIZING: 0.25,
            Evolution.CONSTRAINED: 0.25,
            Evolution.COMPARATIVE: 0.25,
        },
        num_evolutions=1,
    ),
)
goldens = synthesizer.generate_goldens_from_contexts(
    contexts=context_groups,
    source_files=source_files,
    include_expected_output=True,
    max_goldens_per_context=2,
)

for golden in goldens:
    print(
        golden.input,
        golden.expected_output,
        golden.context,
        golden.source_file,
    )

source_files must align one-to-one with contexts. It preserves a useful source label on each returned golden, but a filename alone is not complete provenance. Keep the document version, page or section, chunk IDs, and context group index in a sidecar manifest keyed by the generated batch and source-file label. Set include_expected_output=False only when the review workflow will author references separately. Set max_goldens_per_context from review capacity and coverage goals; its default is two, not a coverage guarantee.

For raw documents, configure DeepEval's context-construction stage explicitly:

from deepeval.synthesizer.config import ContextConstructionConfig

context_config = ContextConstructionConfig(
    max_contexts_per_document=4,
    min_contexts_per_document=2,
    max_context_length=3,
    min_context_length=1,
    chunk_size=512,
    chunk_overlap=64,
    context_quality_threshold=0.7,
    context_similarity_threshold=0.55,
    max_retries=3,
)

document_goldens = synthesizer.generate_goldens_from_docs(
    document_paths=["refund-policy-v4.md", "account-recovery-v2.md"],
    context_construction_config=context_config,
    include_expected_output=True,
    max_goldens_per_context=2,
)

ContextConstructionConfig is specific to document generation. It controls token-based chunking, selection, grouping, and the number and size of contexts constructed per document. Document generation also needs DeepEval's document processing dependencies, including ChromaDB and the relevant LangChain parsing packages. Prepared-context generation skips this stage, which is why it is the better choice when the production chunker and evidence groups are already owned by the test.

Evolution choice matters for RAG. DeepEval currently identifies MULTICONTEXT, CONCRETIZING, CONSTRAINED, and COMPARATIVE as evolutions that remain answerable from supplied context. Other evolution types can create useful general synthetic data but may move a RAG question outside its evidence boundary. Keep the allowed set explicit, store the realized evolution metadata, and reject candidates that still fail review. DeepEval's synthetic data introduction also recommends inspecting and editing generated data where possible.

How do you review grounding, coverage, and expected output?

Review one case at a time before looking at aggregate counts. First prove answerability: a reviewer should be able to mark the exact source passage that supports the expected answer. Then inspect whether the question accidentally reveals the answer, relies on outside knowledge, combines incompatible source versions, or asks for a conclusion the evidence cannot support.

Use a structured review result instead of a free-form approval checkbox:

from dataclasses import dataclass
from typing import Literal

@dataclass(frozen=True)
class HumanReview:
    reviewer: str
    reviewed_at: str
    decision: Literal["pending", "approved", "rejected"]
    answerable: bool
    expected_is_grounded: bool
    distinct_risk: bool
    provenance_complete: bool
    reason: str

@dataclass(frozen=True)
class ReviewResult:
    status: Literal["pending", "approved", "rejected"]
    reviewer: str
    reviewed_at: str
    checks: dict[str, bool]
    reason: str

def finalize_review(
    candidate,
    source_ids: list[str],
    human: HumanReview,
) -> ReviewResult:
    checks = {
        "has_input": bool(candidate.input and candidate.input.strip()),
        "has_context": bool(candidate.context),
        "has_expected_output": bool(
            candidate.expected_output and candidate.expected_output.strip()
        ),
        "source_ids_present": bool(source_ids) and all(source_ids),
        "reviewer_recorded": bool(human.reviewer.strip()),
        "review_date_recorded": bool(human.reviewed_at.strip()),
        "answerable": human.answerable,
        "expected_is_grounded": human.expected_is_grounded,
        "distinct_risk": human.distinct_risk,
        "provenance_complete": human.provenance_complete,
        "reason_recorded": bool(human.reason.strip()),
    }
    approved = (
        human.decision == "approved"
        and all(checks.values())
    )

    if approved:
        status = "approved"
        reason = human.reason
    elif human.decision == "rejected":
        status = "rejected"
        reason = human.reason or "Human reviewer rejected the candidate"
    else:
        status = "pending"
        failed = [name for name, passed in checks.items() if not passed]
        reason = "Approval blocked: " + ", ".join(failed)

    return ReviewResult(
        status=status,
        reviewer=human.reviewer,
        reviewed_at=human.reviewed_at,
        checks=checks,
        reason=reason,
    )

This helper fails closed. Structural checks cannot set status="approved" by themselves, and nonempty expected text is not treated as proof of grounding. A named reviewer must explicitly approve answerability, grounding, distinctness, and provenance, provide a review date and reason, and pass every deterministic check. Pending placeholders remain pending. A second reviewer is useful for high-impact policy, financial, health, or authorization cases, but that rule should come from the product's risk model rather than a generic number.

Coverage review should compare cases by intent and risk, not reward raw volume. Synthetic variants can cluster around source phrases that are easy to turn into questions. Add refusal, ambiguity, stale-policy, citation, multi-hop, and retrieval-failure slices intentionally. For metric-specific expected behavior, DeepEval G-Eval custom metrics explains how to express domain criteria after dataset quality is established.

How do you deduplicate and version synthetic goldens?

Exact duplicate removal is necessary but insufficient. Normalize stable fields for exact hashing, then review near-duplicates semantically. Two questions can use different words while testing the same evidence and behavior. Conversely, similar wording can represent different policies or source versions and should not be collapsed blindly.

import hashlib
import json

def golden_fingerprint(golden: dict) -> str:
    stable = {
        "input": " ".join(golden["input"].lower().split()),
        "expected_behavior": golden["expected_behavior"],
        "source_ids": sorted(golden["source_ids"]),
        "context": [" ".join(c.split()) for c in golden["context"]],
    }
    payload = json.dumps(stable, sort_keys=True, ensure_ascii=True)
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()

def remove_exact_duplicates(goldens: list[dict]) -> list[dict]:
    unique = {}
    for golden in goldens:
        unique.setdefault(golden_fingerprint(golden), golden)
    return list(unique.values())

Do not include mutable review timestamps in the content fingerprint. Keep them in the audit record. When an expected answer changes because a source policy changed, create a new dataset version and preserve the old decision for result reproducibility. Never silently rewrite the data beneath a historical score.

The repository's eval-dataset-design and golden hygiene challenges warn about near-duplicates, stale labels, imbalance, refusal gaps, and prompt leakage. Those are dataset defects even when the evaluation program runs correctly. When scores appear unstable after a data change, use the diagnosis steps in debugging NaN and unstable RAGAS scores without assuming the metric is the only cause.

How do you turn approved goldens into a repeatable eval?

Freeze an approved dataset manifest containing the case IDs, fingerprints, source versions, review decisions, and evaluation configuration. The application runner then reads each input, invokes the current RAG path, and captures actual retrieval and generation evidence. This is where actual_output is created.

A minimal gate can enforce record completeness before expensive evaluation:

import pytest

@pytest.mark.parametrize("golden", load_approved_goldens("rag-v7.jsonl"))
def test_approved_golden_is_executable(golden, rag_app):
    assert golden["review"]["status"] == "approved"
    assert golden["source_ids"]

    result = rag_app.answer(golden["input"])

    assert result.actual_output.strip()
    assert result.retrieval_context
    assert result.case_id == golden["case_id"]

Semantic metrics can run after these deterministic checks. Store the evaluator version, reasons, evidence spans, and application versions with each result. Do not gate on an unexplained average. Define critical cases that fail independently, then calibrate any thresholds against reviewed labels.

For prompt changes, create regression evals for prompt template migrations shows how to hold dataset identity stable while comparing behavior. Teams preparing for review discussions can use RAGAS interview questions for RAG evaluation to practice the retrieval, generation, and evaluator boundaries.

Which failures should block a RAG golden from release?

Block a candidate when provenance is missing, authorization is uncertain, the input is not answerable from the stated evidence, or the expected output adds unsupported facts. Also block stale source versions, unresolved reviewer disagreement, prompt leakage, exact or harmful near-duplicates, and cases whose expected behavior conflicts with current product policy.

A separate block applies to suite quality. A batch should not be released when one easy intent dominates because the generator repeated it, critical refusal or retrieval slices disappeared, or no owner can explain a changed label. These are not reasons to discard synthetic generation. They are reasons to keep generation inside a reviewed data pipeline.

Release evidence should identify the rejected case, source version, rule, reviewer, and remediation. Avoid turning every review issue into one generic "invalid golden" status. Useful categories let owners repair source parsing, context grouping, labels, duplication, or policy conflicts without guessing.

Version the Synthesizer recipe

Treat the current Synthesizer settings as part of the batch identity. Record the DeepEval package version, generation model, async_mode, max_concurrent, FiltrationConfig, EvolutionConfig, and method arguments. For document runs, also serialize ContextConstructionConfig. For context runs, preserve the ordered source_files list and a hash of every ordered context group. Without that recipe, two batches generated from the same documents are not meaningfully reproducible.

Store counts at each DeepEval stage: contexts submitted or constructed, candidate goldens returned, candidates rejected by human review, exact duplicates removed, and approved cases released. Include realized evolution metadata and source_file on the staging export. A changed distribution can then be traced to a source revision, context-construction setting, evolution mix, model change, or review decision instead of being described only as "generator drift."

Do not overwrite an earlier staging export after tuning settings. Generate a new batch ID and compare fingerprints, source labels, expected outputs, and review outcomes. This makes a DeepEval upgrade or configuration change reviewable as a data diff while leaving the broader dataset ownership guidance to building LLM evaluation datasets.

Diagnose the Synthesizer stage first

Classify a poor candidate at the earliest Synthesizer stage. In document mode, bad evidence may come from parsing, chunk size, overlap, random context selection, or semantic grouping. In either mode, the synthetic input may fail filtration, an evolution may move the question outside the evidence, styling may distort the requested task, or expected-output generation may add an unsupported claim.

DeepEval thresholds are not a substitute for this classification. The current generation pipeline can retain the highest-scoring candidate after configured quality retries even when it remains below the requested threshold. Context construction can likewise keep the best available context after retry exhaustion. Record the quality score and retry outcome where available, but keep human approval fail closed.

Only after a candidate is approved should an application failure be classified as retrieval, answer generation, or metric behavior. If the source context was wrong during synthesis, changing the RAG prompt does not repair the golden. If the Synthesizer context and expected output are sound but the application retrieves another chunk, the failure belongs to the application run rather than the synthetic-data stage.

Calibrate controls on a small corpus

Before a large run, use two or three short documents with known evidence boundaries. Run document generation once with an explicit ContextConstructionConfig, then feed reviewer-built groups from the same sources to generate_goldens_from_contexts. Compare context membership, source_file, candidate count, expected-output grounding, and review yield. This reveals whether DeepEval's construction stage or the team's prepared groups better represent the retrieval contract.

Exercise each control deliberately. Lower max_goldens_per_context and confirm the cap changes candidate volume. Disable include_expected_output and confirm references are absent rather than silently fabricated downstream. Supply a mismatched source_files list and require the run to fail. Replace the RAG-safe evolution set with one known not to guarantee contextual answerability in a quarantined run, then prove human review blocks unsupported candidates.

Keep one ambiguous context, one conflicting source version, one missing exception, and one duplicate export as negative controls. Keep a positive control with a known answer and source location. The goal is not one exact generated question; generation remains model-driven. The goal is evidence that the configured DeepEval pipeline preserves source identity, exposes its settings, and cannot promote a structurally incomplete or human-unreviewed candidate.

FAQ

Can Synthesizer create actual outputs?

For single-turn goldens, Synthesizer creates candidate inputs, expected outputs, contexts, and related metadata, but it does not run your RAG application to create actual outputs. Execute each approved input through the application under test, preserve the retrieved context and actual output, then apply the selected evaluation metrics.

Should synthetic goldens replace production examples?

No. Synthetic goldens broaden controlled coverage, while reviewed production examples represent language, failures, and distributions observed in use. Keep provenance labels for both sources, prevent prompt and evaluation leakage, and review performance by slice. A trusted suite combines curated synthetic cases, human-written cases, and sanitized regressions.

How many goldens should each context create?

There is no universal number that proves coverage. Choose counts from the decisions, intents, boundaries, and risks represented by each source group. Review whether extra candidates add a distinct behavior or merely paraphrase an existing case. Stop expanding a group when review finds repetition instead of useful coverage.

Can teams use prepared vector-store chunks?

Yes. Prepared chunks can be supplied as context groups when they preserve the evidence boundary the application will retrieve. Keep source IDs, document versions, chunk order, and neighboring context needed for interpretation. Also test chunk-boundary failures separately because generation from existing chunks cannot reveal evidence that the chunker already removed.

How should generated cases be reviewed?

Reviewers should verify that the input is answerable from supplied context, the expected output adds no unsupported claims, the case represents a useful risk slice, and provenance is complete. Reject ambiguous, duplicated, leaked, stale, or policy-conflicting candidates before any aggregate score can hide their defects.

How are synthetic and human-written goldens mixed?

Store both in one versioned schema with an explicit origin field, then report results by origin and risk slice as well as across the full suite. Human review applies to both. Keep a held-out set away from prompt authors, and promote redacted real incidents into permanent regression cases after adjudication.

Conclusion

DeepEval Synthesizer is most useful when it accelerates candidate creation inside a controlled dataset process. Documents and context groups establish the evidence boundary; review establishes answerability and policy fit; deduplication protects coverage; versioning protects reproducibility; and the application run supplies retrieval context and actual output.

Keep those stages separate and inspectable. The result is not merely a larger golden set. It is an evaluation asset whose sources, decisions, failures, and changes can be explained before a release gate acts on it.