How to Evaluate LLM Outputs for Faithfulness and Groundedness

To evaluate LLM outputs, ask whether each factual claim is supported by application evidence. RAGAS and DeepEval call this faithfulness; TruLens and Azure call it groundedness. They are the same construct.

This workflow creates claim-level evidence and a release rule. For NLI and semantic entropy, read LLM hallucination detection.

Define the Evidence Boundary

Write the contract first:

This prevents outside knowledge from entering the verdict. A true statement can still be unfaithful to supplied context.

Faithfulness cannot prove that context is correct, current, or complete, that retrieval found all evidence, or that the answer is relevant.

Separate the RAG Metrics

Do not average them opaquely. Relevant answers can be unfaithful; faithful answers can be irrelevant. Low context recall needs retrieval work.

Build the Evaluation Record

Each test case should preserve:

• Case ID, user input, and exact output.
• Retrieved chunks in order with source IDs.
• Expected claims, support labels, severity, and abstention.
• Model, prompt, retriever, index, evaluator, and dataset versions.
• Locale, product slice, and provenance.

Use redacted failures, expert cases, critical questions, unanswerable input, stale or conflicting sources, distractors, multi-hop questions, numbers, and fabricated citations. Keep a holdout.

Map the Faithfulness Evaluation Flow

1. 01 / record

   Evaluation Record

   Input, output, context, sources, and versions.

   extract

2. 02 / claims

   Atomic Claims

   Split compound factual statements.

   verify

3. 03 / support

   Support Check

   Find entailment or contradiction evidence.

   validate

4. 04 / calibration

   Calibration

   Compare verdicts with human labels.

   decide

5. 05 / gate

   Risk Gate

   Block severe unsupported claims.

Score Claims, Not Just Answers

Split compound sentences. Record:

• Supported: the supplied context entails the claim.
• Contradicted: the context provides incompatible evidence.
• Insufficient evidence: the context neither supports nor disproves the claim.
• Not factual: the text is a greeting, preference, or other non-verifiable content.

Track supported claims over factual claims, plus each verdict. Even 0.95 can hide one critical failure, so add severity rules.

Require source IDs or spans. For numbers, dates, policies, identities, and citations, add deterministic or system-of-record assertions.

Use DeepEval, RAGAS, or TruLens Carefully

DeepEval and RAGAS expose faithfulness workflows. TruLens presents groundedness with context and answer relevance. The support question is the same.

promptfoo supports application assertions and comparisons. LangSmith evals and OpenAI Evals are also current. Choose by data, traces, language, CI, privacy, and evaluator control.

See the DeepEval tutorial and RAGAS guide. Preserve raw cases and results.

Calibrate the Evaluator

Have qualified humans label cases blind to automated verdicts. Compare false passes, failures, and disagreements across negation, partial support, tables, multiple chunks, conflicts, and domain terms.

Version judge, rubric, examples, sampling, framework, and threshold. Recalibrate after changes and use independent review for possible correlated blind spots.

Do not copy default thresholds. Price false passes against false failures. High-risk claims need stricter rules and human escalation.

Diagnose a Failed Case

1. Confirm exact output and context reached the evaluator.
2. Inspect omitted or merged claims.
3. Find overlooked support, contradiction, staleness, or conflict.
4. Classify retrieval, generation, data, or evaluation ownership.
5. Reproduce with frozen versions before changes.

Missing evidence indicates retrieval work; ignored evidence indicates generator work. Fix evaluator calibration without weakening product requirements.

Faithfulness Release Checklist

• Treat faithfulness and groundedness as the same construct.
• Capture context, output, sources, and versions.
• Separate supported, contradicted, and insufficient claims.
• Pair with context precision, context recall, and answer relevancy.
• Include unanswerable, stale, conflicting, numeric, and citation cases.
• Calibrate against human labels and preserve evidence spans.
• Gate critical claims independently and route disputes to review.
• Add production failures to regression.

Frequently Asked Questions

What is faithfulness in LLM evaluation?

Faithfulness measures whether factual claims in an LLM output are supported by the context supplied to the application. It does not prove that the context itself is complete, current, or factually correct.

Is groundedness the same as faithfulness?

Yes. They are names for the same support construct. RAGAS and DeepEval commonly use faithfulness, while TruLens and Azure commonly use groundedness. State the evidence boundary instead of treating the names as separate metrics.

How is answer relevancy different from faithfulness?

Answer relevancy asks whether the response addresses the question. Faithfulness asks whether its claims are supported by context. An answer can be highly relevant and still contain unsupported claims.

What data is required for a faithfulness evaluation?

At minimum, preserve the user input, actual output, exact retrieved context, source identifiers, application versions, and a case ID. Human calibration also needs claim-level support labels and severity.

Can I use a default metric threshold as a release gate?

Do not copy a default blindly. Label representative cases, inspect false passes and false failures, choose a threshold by risk slice, and add hard rules for critical unsupported claims and fabricated citations.

Make Every Score Explainable

The deliverable is a claim, evidence, calibrated verdict, and owned action, not just a decimal.