Building an LLM Eval Dataset: Golden Sets and Rubrics

If you are focused on building an LLM eval dataset, you are building the foundation every other AI quality practice sits on. Metrics without data are slogans. Prompt experiments without a golden set are taste tests. Hallucination detectors without labeled examples cannot be calibrated. The dataset is the specification of what good looks like, written as cases a machine and a human can both score.

This guide walks through designing golden sets and rubrics, mixing human labels with synthetic data, choosing size, covering edge cases, versioning, avoiding leakage, and operating the dataset as a living product asset rather than a one-time spreadsheet.

What Is an LLM Eval Dataset?

An LLM eval dataset is a curated collection of inputs (and often context, tools, or multi-turn history) paired with expectations used to score model or system behavior. Expectations may be:

• Exact reference answers.
• Structured target fields.
• Allowed answer sets.
• Rubric criteria and score anchors.
• Pass/fail properties.
• Human preference rankings.

A golden set is the trusted core used for regression and release gates. It should be cleaner, better reviewed, and more stable than exploratory datasets used for research spikes.

Dataset Types You Will Likely Need

Type	Purpose	Stability	Typical size
Golden regression set	Release gates, prompt/model diffs	High	50-500
Development set	Prompt iteration during building	Medium	30-150
Challenge / adversarial set	Stress failures and safety	Medium	50-300
Production sample set	Drift and realism checks	Rotating	100-1000 per period
Calibration set	Human vs judge agreement	High review	50-200

Do not force one set to serve every purpose. Leakage and overfitting become likely when the same 40 cases guide daily prompt hacking and also gate production.

Principles of a Good Golden Set

1. Product aligned: cases map to real user jobs and risks.
2. Coverage aware: intents, roles, languages, and edge paths are deliberate.
3. Consistent labels: two raters apply the same standard.
4. Measurable: each case has a scoring method that can pass or fail.
5. Versioned: changes are reviewed like code.
6. Leak resistant: eval content is not pasted into prompts or training fine-tunes casually.
7. Maintainable: owners, tags, and refresh rules exist.
8. Severity conscious: not every miss is equal.

Step by Step: Building an LLM Eval Dataset

Step 1: Write the Evaluation Charter

Before collecting examples, write one page:

• Product promise to the user.
• Out of scope behaviors.
• Quality dimensions that matter (accuracy, groundedness, tone, tool success, latency, cost).
• Risks that are release blocking.
• Languages and regions in scope.
• Who owns labels and gates.

Without a charter, datasets become random bag collections.

Step 2: Inventory Real User Tasks

Mine:

• Support tickets and chat logs (with privacy controls).
• Search queries and click paths.
• Sales demos and failed demos.
• Bug reports and "AI said something weird" threads.
• Compliance scenarios from legal and security.

Cluster tasks into intents. Example for an HR assistant: leave policy, payroll timing, benefits eligibility, device requests, harassment reporting routing, password resets.

Step 3: Define Rubrics Before Mass Labeling

A rubric turns taste into a scorable standard.

Example rubric dimension: Groundedness

Score	Meaning
1	Major unsupported claims or contradictions
2	Some support but material invention or omission of key caveats
3	Mostly supported, minor unsupported flourish
4	Fully supported, precise, appropriate abstention when needed

Example dimension: Task success

Score	Meaning
0	User cannot complete the job
1	Partial completion with significant friction
2	Success with small issues
3	Full success, clear next step

Provide three anchor examples per score level. Anchors reduce rater drift more than long prose definitions alone.

Before scaling the dataset, align the case fields and scoring logic with how to write LLM evals.

Step 4: Design the Case Schema

Use a stable schema from day one.

id: HR-0142
title: Part-time parental leave eligibility
intent: leave_policy
tags: [policy, part_time, edge]
severity: high
locale: en-US
input:
  user: "I work 20 hours a week. How much parental leave do I get?"
context:
  - doc_id: handbook_v4_leave
    span: "Full-time employees receive 16 weeks..."
tools_expected: []
expectations:
  type: rubric
  must:
    - "Does not invent a numeric entitlement not in context"
    - "States part-time rule if present, else abstains"
  dimensions:
    groundedness: 4
    helpfulness: 3
    tone: 3
notes: "Common failure: inventing pro-rated weeks"
owner: qa-ai
version_added: 2026.07.01

Store as JSONL, YAML, or a spreadsheet that exports cleanly. Spreadsheets are fine early; structured files scale better for CI.

Step 5: Seed With Human-Labeled Gold

Human-labeled eval set best practices:

• Train raters on the charter and anchors.
• Double label at least 10-20% of cases.
• Measure agreement (exact match or adjacent agreement).
• Adjudicate disagreements with a senior owner.
• Separate authors of prompts from sole labelers when possible.
• Record why a score was given for hard cases.

Humans are expensive. Spend them on seed quality and critical dimensions, not on labeling every paraphrase forever.

Step 6: Expand With Synthetic Data Carefully

Synthetic data for LLM evaluation is useful when:

• You need rare edge cases that rarely appear in logs.
• You want paraphrases of the same intent.
• You stress long inputs, empty fields, or malformed tool JSON.
• You generate multi-turn scripts with controlled state.

Process:

1. Start from a human gold seed.
2. Generate variants with a strong model.
3. Filter for diversity and realism.
4. Re-label or verify expectations, especially answers and policy outcomes.
5. Tag synthetic origin so you can analyze score gaps vs real traffic.

Never let unchecked synthetic references become release truth for medical, legal, or financial claims.

Step 7: Cover Edge Cases for LLM Test Datasets on Purpose

Edge coverage checklist:

• Empty, null, and whitespace inputs.
• Extremely long inputs and context overflow.
• Conflicting retrieved documents.
• Out of policy user pressure.
• Multilingual and code-mixed text.
• Homoglyphs and lookalike product names.
• Time sensitive facts (prices, versions, deadlines).
• Tool timeouts and empty tool results.
• Prompt injection inside documents or user text.
• Ambiguous entities (multiple "Jordan" matches).
• Users who change goals mid conversation.

Allocate a fixed percentage of the golden set to edges, not leftovers.

Step 8: Choose Size With a Coverage Model

Ask:

• How many primary intents?
• How many critical risk patterns per intent?
• How many locales?
• How many output formats?

A simple planning table:

Intent	Happy	Edge	Safety	Locale variants	Total
Order status	8	6	2	4	20
Refunds	6	8	4	4	22
Product Q&A	10	8	2	4	24

If you have 12 intents at roughly 20 cases each, you are near 240 cases before special adversarial packs. That is a solid mid-stage golden set for many products.

Early stage can ship gates with 40-80 cases if they are ruthlessly prioritized.

Step 9: Version, Review, and Freeze for Releases

• Tag dataset versions: golden_support_v7.
• Freeze the golden set for a release evaluation window.
• Land new cases on next and promote after review.
• Changelog every removal or expectation change.

Changing labels under a model comparison without a version bump invalidates the comparison.

Step 10: Connect Dataset to Metrics and CI

Each case should declare how it is scored:

• Exact match.
• JSON schema.
• Rubric via human.
• Rubric via judge model.
• Tool trajectory assertions.

Then wire the set into prompt regression testing and metric dashboards from LLM evaluation metrics.

Rubric Design Deep Dive

Good Rubric Traits

• Observable: based on what is in the output and traces.
• Anchored: examples for low, mid, high scores.
• Orthogonal: dimensions do not secretly measure the same thing twice.
• Weighted: safety may veto even if tone is perfect.
• Short enough that raters finish without fatigue errors.

Bad Rubric Traits

• "Be excellent" without definition.
• Mixing accuracy and style into one score.
• Ten dimensions for every tiny case.
• Criteria that require secret product knowledge not in the case packet.

Preference Data vs Absolute Rubrics

Preference pairs ("A better than B") help ranking models and some offline analyses. Absolute rubrics help regression gates. Many teams need both, but release gates are clearer with absolute thresholds and critical must-pass properties.

Human Label Workflow

1. Case drafted by QA or eng.
2. Expectations proposed.
3. Domain review for factual and policy correctness.
4. Rater scoring on dimensions.
5. Second rater on sample.
6. Adjudication.
7. Import to versioned store.
8. Periodic relabel audits.

Track rater IDs and times. Sudden score shifts may be rater drift, not model change.

Avoiding Dataset Leakage and Overfit

• Do not paste golden answers into few-shot production prompts.
• Do not fine-tune on the golden set without a held-out slice.
• Keep a secret challenge set for major model upgrades.
• Rotate paraphrases so teams cannot memorize cases.
• Separate development cases from gate cases.

If your pass rate jumps after someone "improved prompts" by embedding eval answers, you did not improve the product. You improved the cheat sheet.

Worked Example: Building a Dataset for a RAG Help Center Bot

Charter Snippet

Answer only from retrieved help articles. Prefer abstention over invention. Be concise. Escalate account-specific billing disputes to human agents.

Intent Map

• Password reset
• Shipping times
• Return windows
• Subscription cancel
• Data export request
• Prompt injection via article text

First 40 Cases Plan

• 20 from anonymized real queries
• 10 synthetic edges (missing context, conflicts, injection)
• 5 multilingual
• 5 tool or escalation paths

Sample Case

ID: RET-009
Input: "I bought a keyboard 28 days ago. Can I still return it?"
Context: Return window is 30 days for keyboards, accessories 15 days.
Must:
  - Say yes if within 30 days
  - Mention condition requirements if present in context
  - Not invent restocking fee
Fail if:
  - Cites 15-day window for keyboards
  - Invents store credit only policy

Scoring

• Groundedness 1-4
• Correct decision pass/fail
• Escalation appropriateness pass/fail

After Two Sprints

Add cases from:

• Production thumbs-down.
• Support tickets where AI was wrong.
• New product category launches.
• Model upgrade diffs that failed.

Dataset becomes a memory of organizational learning.

Operating Model and Ownership

Role	Responsibility
QA / AI quality	Dataset ops, tags, CI gates, reporting
Product	Priority intents, acceptance bar
Domain experts	Policy and factual labels
Engineering	Logging, runners, schema, tooling
Security / legal	Sensitive data handling, high risk criteria

Hold a short eval review in each release train: what failed, what cases were added, what thresholds change.

Common Mistakes When Building Eval Datasets

Mistake 1: Only Happy Paths From Demos

Demo queries are polished. Real users are messy. Your set should look like traffic, not marketing.

Mistake 2: Gigantic Noisy Sets Without Labels

Ten thousand unlabeled prompts do not make an eval dataset. They make a pile.

Mistake 3: Unversioned Spreadsheet Chaos

If nobody can reproduce last month's score, the dataset is not operational.

Mistake 4: Rubrics Written After Labeling Starts

Retroactive standards create inconsistent history.

Mistake 5: Synthetic-Only Gold for High Risk Domains

Plausible fiction is still fiction. Humans must validate.

Mistake 6: Ignoring Locale and Accessibility of Language

English-only sets hide multilingual failure modes.

Mistake 7: No Edge or Adversarial Slice

Average accuracy can look fine while injection and conflict cases burn you.

Mistake 8: Never Deleting Obsolete Cases

Old product rules as "expected" answers become false failures and train the team to ignore the suite.

Practical Checklist

• Evaluation charter exists and is reviewed.
• Schema includes id, input, context, expectations, tags, severity, owner.
• Rubrics have anchors and severity vetoes.
• Seed set is human reviewed.
• Synthetic expansion is tagged and verified.
• Edge and safety slices have explicit quotas.
• Dataset version is pinned in CI reports.
• Agreement metrics for raters are tracked.
• Production failures flow back into the set on an SLA.
• Secret holdout exists for major model swaps.
• Privacy review completed for any real user data.

How This Connects to Hallucinations and Prompts

A dataset built only for style will not catch factual invention. Include accuracy-focused cases and link your scoring to hallucination detection. Once the set is solid, use it as the backbone of prompt and model regression. For framework wiring, see the DeepEval tutorial.

Sampling Production Traffic Ethically

Real traffic is the best source of realism, but it is also the easiest way to create privacy incidents.

Rules of thumb:

• Prefer opt-in feedback samples and synthetic clones of real patterns.
• Redact names, emails, account IDs, and secrets before labeling.
• Separate PII storage from the eval repository when possible.
• Document lawful basis and retention for any raw logs used.
• Exclude employee break-glass and security test traffic unless intentional.

A useful pattern is pattern extraction without content retention: analysts note "users often paste full stack traces" and write synthetic cases that look like that, without storing the original stack traces in git.

Inter-Rater Reliability Targets

If two trained raters cannot agree, your model scores are noise.

Practical targets for many QA teams:

• Exact agreement on binary pass/fail: >= 0.85 after training.
• Adjacent agreement on 4-point rubrics: >= 0.90.
• Critical safety labels: escalate disagreements to a senior adjudicator every time.

When agreement is low:

• Simplify the rubric.
• Add anchors.
• Split mixed dimensions.
• Improve rater training with counterexamples.
• Remove subjective style dimensions from release gates.

Dataset Slices and Reporting

A single average score hides pain. Always slice:

• By intent.
• By severity.
• By language.
• By context length bucket.
• By synthetic vs real origin.
• By presence of tools.

A model can improve on long English happy paths while collapsing on short multilingual edge cases. Slices make that visible.

Golden Set Hygiene Schedule

Activity	Frequency	Owner
Add incident cases	Within 5 business days of incident	QA
Remove obsolete product rules	Each release	Product + QA
Rater recalibration session	Monthly	Eval lead
Full label audit sample	Quarterly	Eval lead
Holdout reshuffle	On major model switch	Eval lead
Privacy review	When new log sources added	Security

Hygiene is part of building an LLM eval dataset as an ongoing system, not a launch task.

Example: From Zero to v1 in Two Weeks

Week 1

• Day 1-2: charter and intent map.
• Day 3: rubric anchors for three dimensions.
• Day 4-5: 40 seed cases from tickets and demos, human labeled.

Week 2

• Day 6: 15 adversarial and abstention cases.
• Day 7: double label sample, fix rubric wording.
• Day 8: wire runner and baseline scores on current prod.
• Day 9: set temporary gates with product.
• Day 10: document ownership and promotion rules for new cases.

You will not have perfect coverage in two weeks. You will have a living control system. That is the goal.

Tooling Options Without Overbuilding

Early stack that is enough:

• Git + JSONL for cases.
• Make/pytest or a simple Python runner.
• Spreadsheet export for non-technical reviewers.
• CI job that prints a markdown scorecard.

Later additions:

• Labeling UI.
• Judge model services.
• Experiment tracking.
• Production sampling pipelines.

Do not delay the first golden set until the perfect platform arrives.

Final Workflow

1. Write the charter.
2. Map intents and risks.
3. Draft rubrics and anchors.
4. Collect seed inputs from reality.
5. Label with review and agreement checks.
6. Add targeted synthetic edges.
7. Freeze a golden version.
8. Run baselines on current production.
9. Set gates.
10. Improve the product, then grow the set from failures.

Building an LLM eval dataset is product management for quality evidence. The teams that win are not the ones with the largest files. They are the ones with clear contracts, consistent labels, honest edges, and a feedback loop from production back into gold. Start with a ruthless small set this week, put it under version control, and let every incident make the next release harder to break. When you want to sharpen general testing judgment that transfers into eval design, practice under time pressure in QABattle battles or create an account at sign-up to track your progress across tracks.