AI & LLM Evals / Scenario
The Suspiciously Perfect Testset
A RAGAS-generated testset gives the pipeline 0.97 everywhere and leadership wants to ship. Read the generation code and the samples, then explain why the numbers are a mirage.
- Difficulty
- Hard
- Format
- Scenario
- Points
- 200
- Estimate
- 14 min
// MISSION BRIEF
Your Mission
The team used RAGAS testset generation on the product docs, produced 200 questions in an afternoon, and the RAG pipeline scores 0.97 across every metric. The dashboard has never looked better. Something smells wrong to you.
Inspect the generation code and sample rows, then decide what the testset actually measures and how to make it honest. Decisions are weighted.
// FIRST CONTACT
Battle teaser
First artifact
generate_testset.py
Why does the pipeline ace this testset while failing real users?
- AThe judge model is biased toward this pipeline
- BThe CSV format corrupts the questions
- C200 samples is too many; large testsets always inflate scores
- DEvery question is single-hop and specific, phrased close to the source sentence; retrieval only has to find one obvious chunk, which measures string-adjacent lookup, not real user behavior like comparisons and multi-step troubleshooting
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
ragastestset-generationsynthetic-datagolden-datasets