AI & LLM Evals / Scenario
Cleaning the Golden Set
The eval score is stable, and wrong. The golden dataset behind it is a mess. Audit it and decide what to fix before you trust another number.
- Difficulty
- Medium
- Format
- Scenario
- Points
- 150
- Estimate
- 14 min
// MISSION BRIEF
Your Mission
A support-bot eval reports a steady 0.82, but production complaints keep rising. The metric is not the problem, the golden dataset is. You have a sample of it and the maintenance history.
Audit the dataset hygiene and prescribe the fixes.
// FIRST CONTACT
Battle teaser
First artifact
Golden set sample (rows)
Which dataset-hygiene problems are present? (Select all that apply.)
- ANear-duplicate rows (1, 2, 3) over-weight one intent
- BSevere class imbalance (47/60 password-reset) skews the score
- CNo refusal / out-of-scope cases, so the eval never tests boundaries
- DA stale label (row 4 still says 14 days; policy is now 30)
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
ai-evalsgolden-datasetsleakagedata-hygiene