QABattle

AI & LLM Evals / Scenario

The Hallucination Hunt

A customer-support RAG bot is confidently wrong. Read the transcript, the retrieved context, and the eval scores, then diagnose where the pipeline failed.

Difficulty
Hard
Format
Scenario
Points
200
Estimate
14 min

// MISSION BRIEF

Your Mission

An insurance company's RAG support bot told a customer their policy covers flood damage. It does not. The customer filed a claim, it was denied, and now Legal is involved.

You have the conversation, the chunks the retriever pulled, and the eval run. Diagnose the failure precisely, the fix depends entirely on which stage broke.

// FIRST CONTACT

Battle teaser

First artifact

Support conversation

Where did the pipeline primarily fail?

  1. AThe user asked an ambiguous question
  2. BRetrieval, the right documents were never fetched
  3. CThe embedding model is broken
  4. DGeneration, the retriever surfaced the correct contradicting chunk (recall 1.0), but the LLM ignored it and hallucinated coverage (faithfulness 0.31)
Answers, scoring, hints, and the full battle stay sealed.

// SKILL TAGS

raghallucinationfaithfulnessdeepeval