Hallucination Detection: Testing LLMs for Accuracy

If you are building LLM hallucination detection into a QA process, you are protecting users from confident nonsense. Large language models can sound correct while inventing citations, policies, prices, medical details, or tool results. Accuracy testing is not a nice-to-have add-on for AI products. It is core product quality work, closer to data integrity testing than to spell checking.

This guide explains what hallucinations are, why they happen, how to detect them with groundedness scoring and factual consistency evaluation, how to measure rates, how to design test cases that invite invention, and how to wire detection into release gates without drowning in noise.

What Is LLM Hallucination Detection?

LLM hallucination detection is the practice of finding and measuring false or unsupported claims in model outputs before users trust them. An LLM hallucination is a generated statement that is presented as true but is false, unverifiable, or unsupported by the sources the system is supposed to use. In product QA, the practical definition depends on the contract:

• Open domain chat: claims that contradict known facts or trusted references.
• RAG systems: claims not entailed by retrieved context, even if they happen to be true in the world.
• Tool-using agents: claims that invent tool results, IDs, or side effects that never happened.
• Extraction tasks: fields filled with plausible values that were never in the source document.

Testers should prefer operational definitions over philosophical ones. If the product promises "answers only from the knowledge base," any unsupported claim is a defect even when the claim is accidentally correct.

Related Failure Types

Failure	Definition	Example
Hallucination	Unsupported or false assertion	Invents a court case citation
Fabrication of sources	Invents documents, URLs, papers	Fake DOI links
Confabulated tool output	Pretends a tool returned data	Fake order status
Inconsistency	Contradicts earlier turns or context	Changes the refund amount mid chat
Overconfidence	States uncertainty as certainty	"Definitely approved" without data
Omission	Leaves out required facts	Omits fee disclosure

Hallucination detection focuses on unsupported or false claims, but good accuracy suites also track related failures because users experience them as the same trust break.

Why Hallucinations Happen

Models predict likely next tokens. They are not databases with guaranteed lookup. Common drivers:

• Missing or partial knowledge in training.
• Retrieval that returns irrelevant or contradictory chunks.
• Prompts that demand an answer when abstention is safer.
• High temperature and long generations that drift.
• Few-shot examples that teach a style of confident answering.
• Tool failures the model "fills in" instead of reporting.
• User pressure: "just give me a number."

Product choices amplify or reduce risk. A medical triage bot that must always answer will hallucinate more harmfully than a bot allowed to say "I do not know, please see a clinician."

Hallucination Detection Strategies

There is no single metric that catches every hallucination. Use layered detection.

Strategy 1: Groundedness Scoring for RAG Answers

Groundedness asks: is the answer supported by the provided context?

Typical pipeline:

1. Fix or capture the context used for generation.
2. Split the answer into atomic claims.
3. For each claim, judge support: fully supported, partially supported, unsupported, or contradictory.
4. Aggregate into a groundedness or faithfulness score.

This is the backbone of RAG accuracy testing. Frameworks and metrics such as faithfulness in RAGAS-style evaluation follow this idea. For pipeline-level RAG metrics, see RAGAS RAG evaluation.

Strengths: scalable, product-aligned for citation systems.
Limits: depends on claim splitting quality and judge reliability. Cannot score world facts outside the context contract.

Strategy 2: Reference-Based Factual Consistency

When you have a trusted reference answer or structured record:

• Compare key facts: amounts, dates, names, statuses.
• Use entailment models or LLM judges against the reference.
• Extract slots and assert equality.

Example for invoice Q&A:

Reference facts:
  vendor = Acme Supplies
  total = 1842.50
  due_date = 2026-08-01
  currency = USD

Answer claims extraction:
  vendor, total, due_date, currency

Score:
  exact match per field + penalty for extra invented fees

Strategy 3: Claim Verification Against External Oracles

For open-domain accuracy, verify claims against:

• Internal product databases.
• Curated knowledge graphs.
• Search over approved corpora.
• Human SMEs for high risk claims.

Automate the easy claims. Send ambiguous or high severity claims to humans.

Strategy 4: Self-Consistency and Cross-Model Checks

Generate multiple answers or use a second model to critique the first. Agreement is not truth, but disagreement is a useful suspicion signal for sampling human review.

Strategy 5: Structured Output and Constrained Generation

Prevention is part of detection strategy. If the model must return JSON with IDs that are validated against a database, entire classes of free-text invention disappear. Test both the happy path and the model's behavior when the database returns nothing.

Strategy 6: User-Facing Abstention Tests

Include cases where the correct behavior is "I do not have enough information." A model that always answers will look productive and still fail accuracy.

Building a Hallucination Test Suite

Step 1: Map Where False Claims Hurt Most

Prioritize:

• Money, health, legal, safety, identity.
• Irreversible actions.
• Citations and policy explanations.
• Any UI that displays model text as authoritative.

Step 2: Design Cases That Invite Invention

Good accuracy cases are slightly adversarial:

• Ask for specifics not present in context.
• Provide incomplete tables and ask for totals.
• Ask for quotes from documents that do not contain them.
• Request URLs, phone numbers, or case law.
• Contradict earlier context and see if the model invents reconciliation.
• Use entities that look real but are not in the corpus.

Step 3: Define Labels and Severity

Severity	Example	Release impact
Critical	Invented drug dosage	Block release
High	Wrong refund policy amount	Block or hotfix gate
Medium	Harmless extra adjective presented as fact	Track, fix soon
Low	Stylistic flourish with no factual load	Monitor

Without severity, average hallucination rate becomes a vanity metric.

Step 4: Choose Automatic Scorers Where They Work

Use automatic groundedness for RAG with fixed context. Use schema and DB checks for tool results. Use human labels for open-world claims until your judges are calibrated.

Step 5: Measure Hallucination Rate Properly

Pick a unit of analysis:

• Answer-level rate: answers with any hallucination / total answers.
• Claim-level rate: unsupported claims / total claims.
• Critical claim rate: critical unsupported claims / total answers.
• User journey rate: sessions where a user would be misled.

Report confidence intervals when the set is small. Segment by intent, language, and context quality.

Example scorecard:

Dataset: accuracy_v12 (n=180)
Answer hallucination rate: 11.1%
Critical answer rate: 1.7%
Claim unsupported rate: 6.4%
RAG groundedness mean: 0.91
Abstention correct rate: 0.84
Cost per 100 cases: $4.20

Trends over releases matter as much as the absolute number.

Worked Example: Policy Assistant

Product promise: answers only from the employee handbook excerpts provided to the model.

Case Design

ID	Setup	User question	Correct behavior
H-01	Context has parental leave of 16 weeks	How many weeks of parental leave?	16 weeks, cite section
H-02	Context silent on sabbaticals	How long is sabbatical leave?	Abstain or say not covered
H-03	Context has conflicting drafts	What is the meal stipend?	Flag conflict, do not invent a blend
H-04	Context lists eligibility rules	Am I eligible if part time?	Apply stated rules only
H-05	User asks for a quote that is not present	Quote the CEO on remote work	Refuse to invent a quote

Claim Check Pseudocode

def score_groundedness(answer, context, judge):
    claims = split_claims(answer)
    results = []
    for claim in claims:
        label = judge.support_label(claim, context)
        # supported | partial | unsupported | contradictory
        results.append(label)
    unsupported = sum(1 for r in results if r in {"unsupported", "contradictory"})
    return {
        "claim_count": len(claims),
        "unsupported_count": unsupported,
        "groundedness": 1 - (unsupported / max(len(claims), 1)),
        "has_critical_hallucination": any_critical(claims, results),
    }

Release Gate Example

• Critical hallucinations: 0 on the critical subset.
• Answer hallucination rate: <= 5% on standard set.
• Correct abstention on "not in context" cases: >= 90%.
• Citation present when policy answer is given: >= 95%.

If a prompt change improves friendliness but drops abstention accuracy, block the change. Friendliness that invents policy is not quality.

Human Review Still Matters

Automatic detectors miss nuance:

• Partial truths with misleading framing.
• Correct numbers with wrong units.
• Soft claims that imply guarantees.
• Jurisdiction sensitive legal wording.

Use humans to:

• Label seed datasets.
• Calibrate LLM judges.
• Review critical failures before release.
• Audit a sample of production answers.

A practical pattern is dual control: automatic score for all cases, human review for all critical fails plus a random sample of passes.

Connecting Detection to the Rest of Eval

Hallucination detection sits beside other quality work:

• Prompt and model regression suites: prompt regression testing.
• Dataset design: building LLM eval datasets.
• Metric selection: LLM evaluation metrics.

Accuracy without regression discipline is a one-time audit. Regression without accuracy metrics is theater. You need both.

Production Monitoring for Hallucinations

Offline suites miss live distribution shift. Add:

• Sampling of production answers for periodic human or judge review.
• User feedback buttons for "this looks wrong."
• Retrieval quality monitors: empty context, low similarity, conflicting chunks.
• Tool error rate monitors: models that invent results after tool timeouts.
• Topic classifiers that escalate high risk domains to stricter review.

Respect privacy and retention policies when logging prompts and outputs.

Common Mistakes in Hallucination Detection

Mistake 1: Only Checking Fluency and Tone

Polished text can still be false. Score claims, not vibes.

Mistake 2: Using World Knowledge Judges on Strict RAG Contracts

If the product must stick to context, a judge that rewards "true in the world but not in docs" will hide product defects.

Mistake 3: One Global Rate Without Severity

A drop from 12% to 10% means little if critical medical hallucinations rose.

Mistake 4: No Abstention Cases

If every test expects an answer, you train and select systems that invent.

Mistake 5: Ignoring Citations and Traceability

Users and auditors need to see sources. Test citation presence, relevance, and correctness separately from answer prose.

Mistake 6: Treating Search Snippets as Ground Truth

Web snippets can be wrong. Curate oracles carefully.

Mistake 7: Never Testing Multilingual Accuracy

Hallucinations often increase in non-English or mixed language inputs.

Mistake 8: Fixing Only the Prompt When Retrieval Is Broken

Many "model hallucinations" are retrieval failures. Diagnose the full pipeline.

Practical Checklist for Testers

• Product accuracy contract is written in one paragraph.
• High risk claim types are listed and prioritized.
• Golden set includes invention-inviting and abstention cases.
• Groundedness scoring is in place for RAG paths.
• Structured fields are validated against systems of record.
• Severity labels drive release gates.
• Hallucination rates are reported with segments and trends.
• Judges are calibrated against humans on a gold subset.
• Production sampling exists for drift.
• Failures become new eval cases within a defined SLA.

Worked Mini Playbook for a New Feature

1. Write the accuracy promise users will believe.
2. List five ways the model could invent harmful content.
3. Create 20 cases for those risks, plus 20 core happy paths.
4. Choose scorers per output type.
5. Establish baseline rates on the current system.
6. Set gates for critical fails and overall rate budgets.
7. Run gates on every prompt, model, and retrieval change.
8. After launch, sample live traffic weekly.
9. Feed confirmed hallucinations back into the suite.
10. Revisit thresholds when the product expands domains.

Hallucination Detection in Multi-Turn Conversations

Single-turn tests miss a common failure: the model invents a fact in turn two to stay consistent with a wrong assumption in turn one. Multi-turn accuracy cases should include:

• User corrects the model, then checks whether the correction sticks.
• Model makes a soft claim, user asks for the source, model invents a citation.
• Long threads where early tool data is forgotten and replaced with guesses.
• Users who change entities mid conversation ("actually, order 99 not 98").

Score both the local turn and the session. A session-level hallucination rate often correlates better with user trust than isolated answer scores.

Groundedness Scoring for RAG Answers in Detail

A practical groundedness pipeline for testers:

1. Capture the exact context sent to the model, including order and truncation.
2. Normalize the answer (strip boilerplate disclaimers you do not want scored as claims).
3. Split claims into atomic factual statements.
4. Classify each claim against context: supported, partial, unsupported, contradictory, opinion, or abstention.
5. Weight claims by severity. A wrong dosage outweighs a harmless adjective.
6. Aggregate into answer-level pass/fail using product rules.

Example product rule:

Fail answer if:
  any critical claim is unsupported or contradictory
  OR unsupported_count >= 2
  OR groundedness < 0.85 on non-critical informational answers
Pass abstention if the context truly lacks the answer

Calibrate the claim splitter. Over-splitting turns style into fake factual claims. Under-splitting hides invented numbers inside long sentences.

Factual Consistency Evaluation Beyond RAG

Not every product is RAG. For open assistants:

• Maintain a fact bank of product truths (pricing tiers, feature flags, company policies).
• Run periodic probes: "How much is plan X?" and compare to the fact bank.
• Track consistency across paraphrases of the same fact.
• When the fact bank changes, version it and expect intentional answer changes.

For code assistants, consistency may mean the generated API usage matches the installed library version docs you provide. For analytics assistants, consistency means numbers match the query engine output, not a remembered training prior.

Measuring Hallucination Rate Without Self-Deception

Bad measurement patterns:

• Counting only obvious fabricated URLs while ignoring subtle policy invention.
• Averaging all domains so medical risk is diluted by chitchat.
• Changing the dataset on the same day you change the model and calling it improvement.
• Using a judge model from the same family without human calibration.

Good measurement patterns:

• Fixed dataset versions.
• Severity-aware rates.
• Segment reports: intent, language, context length, retrieval quality bucket.
• Human audit of all critical fails and a random sample of passes.
• Confidence intervals when n is small.

Report rates like reliability engineering, not like marketing percentages.

Tool and Agent Hallucinations

When tools exist, inventing a tool result is among the worst hallucination classes. Dedicated cases:

• Tool returns empty: model must not fabricate rows.
• Tool times out: model must not claim success.
• Tool returns error code: model must surface failure.
• Tool returns unexpected null fields: model must not fill them with guesses.

Log comparison between tool payload and final answer should be automated for high risk fields.

Team Playbook for a Suspected Hallucination Incident

1. Capture the prompt, context, tools, model version, and output.
2. Identify the false claim and user impact.
3. Classify: retrieval miss, prompt pressure, model prior, tool misuse, or UI presentation bug.
4. Patch the immediate product path (guardrail, post-check, or disable feature).
5. Add one or more eval cases that would have failed before the fix.
6. Run regression on related intents.
7. Decide whether communication to users is required.
8. Schedule a deeper systemic fix if the root cause is architectural.

Incidents that do not produce new tests will recur.

Additional Comparison: Detection Methods

Method	Best for	Weak against	Automation
Groundedness to context	RAG contracts	Open-world facts	High
Reference fact match	Known fields and answers	Novel phrasings if matcher is brittle	High
LLM judge	Nuanced claims	Biased or weak judges	High with calibration
Human SME review	High risk domains	Scale and cost	Low
Cross-model agreement	Suspicion sampling	Shared blind spots	Medium
Database post-check	Tool-backed fields	Free-text explanation errors	High

Layer methods. Do not bet the company on a single judge prompt.

Final Guidance

LLM hallucination detection is a testing discipline with the same seriousness as payment correctness or authz checks. The techniques differ: claims, groundedness, oracles, abstention, and severity. The mindset is familiar: define expected behavior, design cases that can fail, measure, gate, and learn from production.

Do not wait for a public incident to start. Build a small accuracy suite this week, focus on the highest risk claims, and expand with every defect you find. For broader metric literacy, continue with LLM evaluation metrics. For competitive practice that sharpens observation skills under pressure, try a challenge in the QABattle arena.