LLM Hallucination Detection: Methods, Metrics, and Tools

LLM hallucination detection finds claims that lack evidence required by the application. A creative assistant may invent fiction, while a support bot must not invent a refund rule. A RAG answer can be true in the wider world and still be unfaithful to supplied context.

Detection is not one magic score. Capture the answer and evidence, extract claims, apply complementary checks, calibrate with human labels, and gate meaningful failures.

For a focused implementation of the core RAG construct, read how to evaluate faithfulness and groundedness. This article builds the broader detection system around it.

Define Hallucination Against a Truth Standard

Define which evidence may support an answer:

• Provided context: Every factual claim must be entailed by the retrieved chunks.
• Reference answer: The output must agree with a curated expected answer while allowing valid paraphrases.
• Authoritative source: Claims must be verifiable against a trusted, versioned knowledge base or tool result.
• World knowledge: Claims may require external verification, which is harder to make complete and current.
• System state: Claims about bookings, balances, permissions, or actions must match the system of record.

Do not mix standards silently. Context faithfulness cannot prove the context is correct. A reference can penalize valid alternatives, and search can miss or retrieve stale evidence. Record the standard per case.

Failures include unsupported additions, contradictions, fabricated citations, wrong attribution, entities, numbers, dates, and overstated evidence. For agents, verify downstream actions too.

Faithfulness and Groundedness Mean the Same Construct

Faithfulness and groundedness are naming differences, not separate dimensions. RAGAS and DeepEval use faithfulness; TruLens and Azure use groundedness. Both ask whether output claims are supported by context.

Keep related metrics separate:

An answer can be relevant and unfaithful, or faithful to irrelevant context. Diagnose retrieval and generation separately.

Map the Hallucination Detection Pipeline

The pipeline below makes evidence provenance visible. Every verdict should link back to the exact claim, evidence span, evaluator, and version that produced it.

Method 1: LLM-as-a-Judge

An LLM judge can extract claims, compare evidence, classify support, and explain its verdict. Require structured claim, status, evidence ID, contradiction reason, and confidence. Include the evidence so the judge does not rely on hidden knowledge.

Calibrate the rubric, model, and prompt on blinded human labels. Measure false passes and failures for negation, numbers, dates, partial support, tables, and conflicts. Recalibrate after changes.

Use multiple judges only when disagreement has a defined action. Route severe claims and disagreements to a domain reviewer.

Method 2: NLI and Entailment Grounding Checks

NLI models classify whether evidence entails, contradicts, or is neutral toward a claim. Their narrow claim-level task can suit high-volume screening.

Long context may need chunking, and claims can span chunks. Neutral often means unsupported, not false. Domain terms, numbers, and tables can reduce reliability. Calibrate thresholds and preserve support and contradiction evidence.

Use deterministic checks for identifiers, citations, dates, amounts, schemas, and permissions. A refund claim should be verified in the authorized order system.

Method 3: Semantic Entropy

Semantic entropy samples answers, groups equivalent meanings, and measures probability across those groups. Meaningfully different answers indicate more uncertainty than varied wording with one meaning.

Use it to route unstable cases for review, not as proof of falsehood. A model can be consistently wrong, and some questions allow several answers.

Record model, prompt, sampling, sample count, clustering, and threshold. Apply repeated generation to uncertain or high-risk slices to control cost.

Method 4: Retrieval-Grounded Claim Verification

Retrieval-grounded verification splits an answer into claims, finds evidence, and assigns supported, contradicted, or insufficient-evidence status. It shows exactly what failed.

A practical verifier should:

1. Capture the exact chunks that the generator received.
2. Split compound statements so one supported clause cannot hide one unsupported clause.
3. Link each claim to evidence spans and source identifiers.
4. Check direct contradiction before accepting broad topical similarity.
5. distinguish insufficient evidence from proven falsehood.
6. Aggregate claim results with severity-aware rules.

Evaluate retrieval too. Missing sources indicate context-recall work; present but ignored evidence points to the generator, prompt, ordering, or answer policy.

Tools for LLM Hallucination Detection

Choose by application language, trace format, CI, and privacy. See the DeepEval tutorial, RAGAS guide, and promptfoo tutorial.

Build a Practical Detection Dataset

Use critical questions, redacted failures, expert examples, and reviewed synthetic edges. Include support, contradiction, unanswerable, stale, conflicting, distractor, numeric, multi-hop, and fabricated-citation cases.

Store case ID, input, context and source IDs, output, claims, human labels, severity, abstention, locale, and provenance. Separate development and holdout sets.

Weight risk. A rare unsupported dosage, amount, access decision, or irreversible action outweighs harmless wording errors. Gate critical slices separately.

Implement a Layered Detection Pipeline

Use a sequence that spends the cheapest reliable evidence first:

hallucination_gate:
  capture: [input, output, retrieved_context, tool_results, versions]
  deterministic_checks: [schema, citation_id, amount, permission, prohibited_claim]
  claim_verification: [nli_entailment, llm_judge]
  uncertainty_signal: semantic_entropy_on_selected_cases
  outcomes: [pass, fail, needs_human_review]
  block_on: [unsupported_critical_claim, contradiction, fabricated_citation]

Validate structure and exact facts, then verify claims with NLI or a judge. Apply entropy selectively. Route severe failures, missing evidence, and disagreements to human review.

Store claim verdicts, not only an answer score. An aggregate can hide one catastrophic unsupported statement, so add severity and fabricated-citation rules.

Calibrate Metrics and Release Thresholds

Do not copy default thresholds. Use labeled calibration data, price false passes and failures, and inspect confusion by risk slice.

Version application, retrieval, context, judge, NLI, sampling, and metric components. Compare candidates on frozen cases.

Monitor privacy-safe production samples and promote confirmed failures into regression with stable expected outcomes.

Hallucination Detection Review Checklist

• Define the permitted truth standard for every use case.
• State that faithfulness and groundedness are the same construct.
• Keep context precision, context recall, and answer relevancy separate.
• Capture exact retrieved context and tool results used by the application.
• Split outputs into atomic, reviewable factual claims.
• Preserve evidence spans and distinguish contradiction from insufficient evidence.
• Combine deterministic checks with calibrated NLI or LLM judges.
• Use semantic entropy as uncertainty evidence, not proof of falsehood.
• Label high-risk, unanswerable, stale, conflicting, and multi-hop cases.
• Version models, prompts, contexts, evaluators, data, and thresholds.
• Gate severe claims independently of aggregate scores.
• Route evaluator disagreement and missing evidence to human review.

Frequently Asked Questions

What is an LLM hallucination?

An LLM hallucination is an output claim that is unsupported by the evidence or truth standard required for the task. In RAG, the operational test usually asks whether each answer claim is supported by the retrieved context.

Are faithfulness and groundedness different metrics?

They are different names for the same construct: whether output claims are supported by the provided context. RAGAS and DeepEval commonly use faithfulness, while TruLens and Azure commonly use groundedness.

Can an LLM reliably detect another LLM's hallucinations?

An LLM judge can evaluate nuanced claims at scale, but it is not an unquestionable oracle. Calibrate it against expert labels, require evidence and reasons, control judge versions, and route severe or disputed cases to humans.

What is semantic entropy in hallucination detection?

Semantic entropy estimates uncertainty by sampling multiple answers, grouping answers with the same meaning, and measuring how dispersed the probability is across those meaning groups. High uncertainty is a review signal, not direct proof of falsehood.

Which tools support LLM hallucination evaluation?

DeepEval, RAGAS, promptfoo, and TruLens support useful evaluation workflows. LangSmith evals and OpenAI Evals are also current framework options. Choose by data model, evaluator control, tracing, CI, privacy, and reporting needs.

What should block a release?

Block unsupported high-risk claims, contradictions with controlling evidence, fabricated citations, unsafe actions based on unsupported claims, and severe regressions in critical slices. Calibrate numerical thresholds on labeled data rather than copying defaults.

Make Unsupported Claims Inspectable

Good LLM hallucination detection does not merely label an answer bad. It identifies the exact unsupported claim, shows the evidence considered, records evaluator uncertainty, and assigns an action based on severity. That evidence makes failures debuggable and release decisions defensible.

Use the existing LLM hallucination detection overview for additional context, then compare evaluation designs in the /battles arena.