AI & LLM Evals / Scenario
The Eval Gate That Never Fails
A DeepEval CI gate has been green for six weeks, including the release that shipped an obvious hallucination regression. Read the test code and find why the gate is theater.
- Difficulty
- Hard
- Format
- Scenario
- Points
- 200
- Estimate
- 14 min
// MISSION BRIEF
Your Mission
The team wired DeepEval into CI months ago and the badge has been green ever since. Last week a release started inventing refund policies, support tickets exploded, and the gate never blinked.
You have the test file and a passing CI log. Work out why this gate physically cannot fail, and what a real one looks like. Decisions are weighted.
// FIRST CONTACT
Battle teaser
First artifact
test_rag_quality.py
Why is faithfulness pinned at 1.00, even while the bot hallucinates policies?
- AThe threshold of 0.2 forces the score to round up
- BThe judge model is too small to detect hallucinations
- CFaithfulness always returns 1.0 unless the answer is empty
- DCircular grounding: retrieval_context is filled with the model's own answer, so every claim is 'supported' by itself; the metric must receive the chunks the retriever actually returned
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
deepevalci-cdfaithfulnesseval-engineering