Use Ragas Noise Sensitivity to Test Retrieval

Ragas noise sensitivity metric testing reveals whether a RAG answer turns retrieved distractors or weak evidence into incorrect claims. Build it with labeled questions, references, generated responses, and the exact retrieved contexts; compare clean and noisy controls per case; calibrate model judgments with humans; then block releases only on predeclared, cost-bounded evidence.

This is a retrieval-to-generation test, not a retriever score in isolation. A system can retrieve an irrelevant passage yet ignore it, or retrieve mostly relevant passages and still copy one plausible false statement. The useful assertion is therefore about the response produced from a known context set. The broader Ragas RAG evaluation guide explains the surrounding metric family, while this article stays focused on noise exposure and release evidence.

Define the Evaluation Question and Metric Contract

The current Ragas Noise Sensitivity documentation defines NoiseSensitivity on four inputs: user_input, response, reference, and retrieved_contexts. It examines response claims, their correctness against the reference, and their relationship to relevant or irrelevant retrieved material. The score is bounded from zero to one, and lower is better. Store that direction beside every threshold because a generic "higher passed" dashboard will invert the release decision.

A precise contract is: for a fixed user question and adjudicated reference, adding a controlled distractor must not cause the candidate RAG system to introduce an incorrect claim. That contract separates three events that teams often collapse:

• The retriever selected an irrelevant or misleading context.
• The generator used or ignored that context.
• The evaluator classified claims and context relationships correctly.

NoiseSensitivity observes the second event through a model-based analysis of all four fields. It does not directly prove that context ranking is optimal, that every reference fact was recalled, or that the final answer satisfies the user. Pair it with deterministic retrieval assertions and the wider outcome checks in measuring RAG accuracy. If the release question is "Did the correct document appear at the required rank?", start with RAG retrieval testing instead.

Write the decision before choosing a threshold. Name the protected slice, such as refund eligibility or medication warnings; the change under review, such as a reranker or prompt; the comparison baseline; and the owner who can explain a failing claim. Declare whether the run uses mode="relevant" or mode="irrelevant". Results without mode, evaluator model, metric version, and case provenance are not comparable evidence.

The first observable assertion should be deterministic: all required fields exist, the contexts are the ones actually sent to generation, the reference version is approved, and the candidate and baseline use the intended configuration. Only then should an evaluator-model score influence release. This ordering prevents a missing context trace from masquerading as an excellent low score.

Build a Representative Labeled Dataset and Evidence Record

A useful dataset starts with real retrieval risks rather than random questions. Sample ordinary traffic, known production failures, high-impact policy questions, multi-document answers, tempting near-duplicates, stale passages, and cases where the correct behavior is to abstain. If synthetic generation helps expand coverage, review it as described in Ragas testset generation from real documents and keep its provenance distinct from production-derived cases.

Each record needs more than the four scoring fields. Preserve a stable case ID, source-document IDs and versions, context rank, retrieval score as diagnostic metadata, query rewrite, corpus snapshot, retriever and reranker configuration, generator model, prompt hash, evaluator model, metric mode, run ID, and actual or estimated evaluator cost. The response must be the output generated from the recorded retrieved_contexts. Re-running retrieval later can silently substitute a different context set and destroy causality.

Use paired cases. The clean member contains enough relevant evidence and no deliberate distractor. The noisy member holds the question, reference, generator configuration, and relevant evidence constant while adding or replacing one context with a labeled distractor. A pair lets reviewers ask whether the response changed because of noise instead of comparing unrelated examples.

{
  "case_id": "refund-trial-window-pair",
  "slice": "billing-policy",
  "mode": "irrelevant",
  "reference_version": "policy-reviewed",
  "user_input": "Can a trial purchase be refunded?",
  "reference": "A trial purchase may be refunded when the request is submitted within seven calendar days.",
  "variants": [
    {
      "name": "positive_clean",
      "retrieved_contexts": [
        "Trial purchases are eligible for refund requests within seven calendar days."
      ],
      "response": "Yes. Submit the refund request within seven calendar days."
    },
    {
      "name": "negative_noise_adoption",
      "retrieved_contexts": [
        "Trial purchases are eligible for refund requests within seven calendar days.",
        "Archived draft: trial purchases can never be refunded."
      ],
      "response": "No. Trial purchases can never be refunded."
    }
  ],
  "expected_relation": "negative_noise_adoption must score worse than positive_clean"
}

That pair is intentionally bound to mode: "irrelevant": the archived draft does not support the reference and the negative response adopts its false claim. A relevant-mode control needs a different construction. The context must still contain evidence that makes it relevant to the reference while also carrying the incorrect statement adopted by the response:

{
  "case_id": "refund-fee-relevant-context-pair",
  "slice": "billing-policy",
  "mode": "relevant",
  "reference_version": "policy-reviewed",
  "user_input": "Can a trial purchase and its processing fee be refunded?",
  "reference": "A trial purchase may be refunded within seven days, but its processing fee is nonrefundable.",
  "variants": [
    {
      "name": "positive_clean",
      "retrieved_contexts": [
        "Trial purchases may be refunded within seven days. Processing fees are nonrefundable."
      ],
      "response": "The purchase may be refunded within seven days, but the processing fee is not refunded."
    },
    {
      "name": "negative_relevant_adoption",
      "retrieved_contexts": [
        "Trial purchases may be refunded within seven days. An obsolete sentence says processing fees are also refunded."
      ],
      "response": "The purchase and its processing fee are both refunded within seven days."
    }
  ],
  "expected_relation": "negative_relevant_adoption must score worse than positive_clean"
}

The examples declare mode-specific orderings, not fabricated benchmarks. Human reviewers should confirm that the reference is sufficient, that each context receives the intended relevant or irrelevant classification, and that the negative response adopts the controlled false claim. For real datasets, redact personal data and secrets while retaining identifiers that join a result to approved source evidence.

Split development, calibration, and held-out release sets by source family, not by individual row. Near-duplicate passages or paraphrased questions across partitions can make a prompt adjustment look general when it only memorized a document pattern. Track natural traffic prevalence for reporting, but oversample rare harmful noise cases for diagnosis and label that sampling choice.

Add an evidence manifest for every dataset release. It should list the inclusion window, source owners, document licenses or usage authority, redaction method, known coverage gaps, reviewer identities or controlled pseudonyms, and the hashes of the case and reference files. Record whether a response was captured from production, regenerated under a laboratory configuration, or written as a control. Those origins are not interchangeable. A production response carries realistic behavior but may contain hidden environment effects; a constructed control provides a clear assertion but says nothing about prevalence. Keeping both allows the suite to diagnose mechanisms without presenting its sampling mix as customer incidence.

Review references on a schedule tied to source change, not only article publication or test failure. Policy documents, catalogs, and support procedures can change while a frozen answer remains syntactically valid. Mark expired cases as retired rather than deleting them, preserve prior result lineage, and issue a new case version when the correct answer changes. Otherwise, a healthy application update can appear to regress against stale truth.

Implement the Current Collections Metric in Python

The stable documentation recommends the collections API for new projects. It uses an asynchronous client in the example, constructs an evaluator with llm_factory, and calls ascore directly with the four required fields. The legacy SingleTurnSample route remains documented as legacy, so pin the installed Ragas version and avoid mixing result shapes across APIs.

import asyncio
import json
import os
from pathlib import Path

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import NoiseSensitivity


async def score_case(case: dict) -> dict:
    mode = case["mode"]
    if mode not in {"relevant", "irrelevant"}:
        raise ValueError(f"invalid NoiseSensitivity mode: {mode}")
    client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
    evaluator = llm_factory(os.environ["RAGAS_EVALUATOR_MODEL"], client=client)
    scorer = NoiseSensitivity(llm=evaluator, mode=mode)

    result = await scorer.ascore(
        user_input=case["user_input"],
        response=case["response"],
        reference=case["reference"],
        retrieved_contexts=case["retrieved_contexts"],
    )
    return {
        "case_id": case["case_id"],
        "slice": case["slice"],
        "mode": mode,
        "score": float(result.value),
        "evaluator_model": os.environ["RAGAS_EVALUATOR_MODEL"],
    }


async def main() -> None:
    cases = json.loads(Path("noise-cases.json").read_text())
    results = [await score_case(case) for case in cases]
    print(json.dumps(results, indent=2, sort_keys=True))


if __name__ == "__main__":
    asyncio.run(main())

Do not print a claimed expected score into the fixture. The evaluator may segment claims differently after a model or prompt change. Put stable requirements in the fixture: required fields, reference provenance, metric mode, control role, expected ordering, and business severity. Store the measured value only in the run artifact with all evaluator metadata.

The script intentionally makes the evaluator model mandatory and reads the reviewed mode from each case. An implicit provider or mode default is a source of unreviewed drift. In a production harness, reuse the client, apply provider concurrency limits, capture token or billing usage through the supported provider instrumentation, and sanitize errors before attaching them to CI.

Run Positive, Negative, and Adversarial Controls in Order

Controls prove that the test can distinguish a meaningful failure before it evaluates a candidate. They are not decorative rows. Keep their source evidence frozen and run them whenever the metric prompt, evaluator model, Ragas version, or parsing code changes.

Positive control: Provide sufficient relevant evidence and a response containing only reference-supported claims. This checks that a grounded answer is not falsely penalized. Include a concise answer and a longer answer with multiple supported claims because claim segmentation can affect the denominator.

Negative control: Match the construction to the selected mode. For irrelevant, add a plausible context that does not support the reference and use a response that adopts its false statement. For relevant, use a context that supports part of the reference but also contains the controlled false statement adopted by the response. Each negative should be worse than its paired positive under the same mode and evaluator configuration. If a pair does not separate, stop the run before interpreting candidate scores.

Adversarial control: Insert a distractor that resembles the source style, contains a stale effective date, or includes instructions aimed at the generator. The expected product behavior is to ignore or explicitly reject it. NoiseSensitivity can expose incorrect claims, but a separate prompt-injection or policy check must judge whether the system followed hostile instructions.

Reference-defect control: Deliberately provide an incomplete reference for a response whose extra claim is supported by the source documents. A competent reviewer should flag the case as invalid rather than tuning the judge to call the answer wrong. This control protects the dataset from being treated as infallible.

Use this numbered workflow for every candidate run:

1. Freeze the corpus, query rewrite, retriever, reranker, generator prompt, generator model, evaluator model, Ragas version, and metric mode.
2. Validate required fields, non-empty contexts, approved reference versions, unique case IDs, and clean/noisy pair membership.
3. Generate or load responses from the exact recorded context list; never replace them with a later replay without a new run ID.
4. Run deterministic retrieval assertions, including document identity, tenant boundary, rank constraints, and forbidden-source checks.
5. Score positive, negative, adversarial, and reference-defect controls before scoring release candidates.
6. Reject the evaluation if controls fail to order correctly, results are missing or non-finite, or evidence cannot be joined to the originating case.
7. Review candidate changes per case and slice, then inspect the response claims and contexts at the first divergence.
8. Apply the versioned CI policy and cost boundary; route ambiguous evaluator disagreements to human review instead of automatic retry.

End-to-end behavior still matters. Testing a RAG chatbot adds conversation history, citations, refusal behavior, and user-visible outcomes that a single-turn claim metric cannot cover.

Interpret Per-Case Scores and Compare Diagnostic Metrics

Start with the pair, not the average. Read the clean response, noisy response, reference, and ranked contexts. Identify which claim changed and whether the change came from retrieval, generation, reference quality, or evaluator judgment. A lower candidate score is favorable only when the case remains valid and the answer still serves the user. A system that refuses every question may avoid incorrect claims while failing its product purpose.

The Ragas metric design overview recommends a small number of interpretable signals and distinguishes end-to-end, component, and business metrics. Use that principle to avoid a dashboard where several correlated model judgments vote on the same hidden assumption.

For related Ragas distinctions, the faithfulness and context recall scenarios help separate unsupported generation from missing evidence. Record all metrics at case level and join them by one run ID. Do not average away a critical false policy claim because many easy factual questions remained clean.

Compare baseline and candidate only when dataset version, metric mode, evaluator identity, and scoring prompt are compatible. If any changes, run both application versions under the new evaluator or treat the result as a calibration study. A historical number produced by a different judge is not a stable ruler.

Use slices that map to action. Product area, source authority, distractor type, language, answer length, and business severity are more useful than a single global category named "RAG." Report the number of valid evaluated cases beside every aggregate so missing rows cannot make a result look better. Preserve abstentions and evaluator errors as separate outcomes. Converting them to zero would falsely imply perfect noise resistance; dropping them would hide a reliability problem.

When a pair reverses unexpectedly, inspect both members before blaming the candidate. The clean response may contain more claims than the noisy response, changing the opportunity for error. The noisy system may abstain, reducing incorrect claims but harming task success. Keep paired NoiseSensitivity beside answer completeness and refusal appropriateness so an optimization cannot improve the metric by saying less than users need.

Calibrate Model Judgments with Human Claim Review

NoiseSensitivity includes model judgments about claims, relevance, attribution, and correctness. Calibration asks whether those judgments support the intended release decision. It is not an attempt to force perfect agreement on inherently ambiguous cases.

Create a calibration panel from positive controls, negative controls, borderline cases, critical slices, abstentions, long multi-claim answers, and known evaluator errors. Have at least two qualified reviewers independently mark claim boundaries, correctness against the approved reference, and which context supports or contradicts each claim. Hide candidate identity and prior score where practical. Adjudicate disagreements and preserve the rationale.

The official Ragas judge alignment quickstart demonstrates comparing human pass/fail labels with an LLM judge and iterating on misalignment patterns. Apply the same discipline here at claim level. The detailed LLM-as-a-judge guide explains why fluent reasons are not validation, and human-adjudicated gold sets shows how to preserve reviewer independence and label lineage.

Run these calibration tests whenever the evaluator model, Ragas package, prompt, language, reference policy, or traffic mix changes:

• Control separation: Known noise adoption remains distinguishable from its grounded pair.
• Direction test: Lower remains the favorable direction throughout reporting and alerting.
• Claim-boundary test: Reviewer and evaluator differences are inspected on long answers rather than summarized as one unexplained score.
• Slice test: Critical languages, document types, and answer forms remain within the human-approved error policy.
• Repeatability test: Repeated evaluations of a frozen sentinel set expose material judge variation instead of silently overwriting it.
• Reference challenge: Reviewers can mark a reference defective or insufficient without being forced to agree with the metric.

Reserve a held-out calibration set after tuning. If the same disagreements are used to rewrite the prompt and certify the rewrite, improvement can be simple memorization. Version the gold set and never edit a final label in place; supersede it with a new adjudication record.

Debug Failures from the First Trustworthy Boundary

Ragas noise sensitivity metric testing debugging should begin with evidence integrity. Confirm the intended case ran, the four required values are present, the response came from those contexts, and the mode and evaluator match the baseline. Then compare the first changed artifact in this order: corpus, retrieved IDs and ranks, response claims, reference, evaluator output, aggregation, gate.

Treat these conditions as deterministic blockers:

• Missing, empty, duplicated, or mismatched case identifiers.
• Empty required text or no recorded retrieved_contexts.
• A response hash that does not match the generation trace for those contexts.
• An unapproved or missing reference version.
• Unknown metric mode, evaluator model, package version, or prompt version.
• Missing, non-numeric, non-finite, or out-of-range metric values.
• Failed positive-to-negative control ordering.
• Cross-tenant retrieval, forbidden document IDs, or absent access-control evidence.
• Candidate cost or case count beyond the declared run budget.

If deterministic checks pass, inspect model judgment. A long compound sentence may be split differently; a reference may imply rather than state a fact; a distractor may actually be current policy; or the generated answer may contain a harmless explanation that the reference omits. Preserve the claim-level rationale if the API exposes it, but validate against source evidence rather than trusting the explanation's tone.

Do not fix an unstable run by retrying until it passes. Repetition is useful only as a measured calibration experiment with every attempt retained. For wider patterns involving missing and unstable metric outputs, revisit the Ragas evaluation architecture and classify provider, parsing, data, and evaluator failures separately.

Set a CI Gate with Version and Cost Boundaries

CI should have two lanes. A pull-request lane runs schema checks, deterministic retrieval controls, and a small frozen evaluator sentinel set. A scheduled or pre-release lane runs the representative distribution, expensive adversarial pairs, and human-review sampling. The fast lane blocks obvious contract breaks; the broad lane estimates release risk without making every code change wait for the full corpus.

Store threshold and budget values in reviewed policy, not article code. A team must derive them from adjudicated baseline behavior, risk tolerance, evaluator variability, and available spend. A missing policy should fail closed for a release gate rather than falling back to a convenient constant.

import math
from dataclasses import dataclass
from decimal import Decimal


@dataclass(frozen=True)
class GatePolicy:
    max_score_by_slice: dict[str, float]
    max_cases: int
    max_total_cost: Decimal


def evaluate_gate(rows: list[dict], policy: GatePolicy) -> list[str]:
    blockers: list[str] = []
    if len(rows) > policy.max_cases:
        blockers.append("case count exceeds the approved cost boundary")

    total_cost = sum(Decimal(str(row["evaluator_cost"])) for row in rows)
    if total_cost > policy.max_total_cost:
        blockers.append("evaluator cost exceeds the approved boundary")

    for row in rows:
        score = float(row["score"])
        if not math.isfinite(score) or not 0.0 <= score <= 1.0:
            blockers.append(f'{row["case_id"]}: invalid NoiseSensitivity score')
            continue
        limit = policy.max_score_by_slice.get(row["slice"])
        if limit is None:
            blockers.append(f'{row["case_id"]}: no reviewed slice policy')
        elif score > limit:
            blockers.append(f'{row["case_id"]}: exceeds its lower-is-better limit')

    return sorted(set(blockers))

Also record candidate-versus-baseline deltas per paired case, even when both remain inside policy. A warning trend can trigger investigation before it becomes a blocker. Permit overrides only with named owner, expiry, case-level rationale, and a follow-up issue. Never override a missing evidence join or cross-tenant retrieval as evaluator noise.

Bound cost by maximum eligible cases, concurrency, provider retries, evaluator model, token accounting method, and total currency budget. Stop scheduling new model calls when the boundary is reached, preserve completed evidence, and report the run as incomplete. An incomplete expensive run is not a pass. Revisit human-adjudicated calibration before promoting a new judge or threshold.

Separate budget exhaustion from quality failure in the CI status, but let both block a release when the policy requires complete evidence. A quality owner should see which cases never ran, what spend was recorded, whether provider retries consumed the allowance, and which lower-cost deterministic checks completed. This makes the next action explicit: fix product behavior, repair the harness, approve more evaluation capacity, or reduce scope through a reviewed risk selection. Quietly truncating the most expensive long-context cases biases the run toward easier examples.

Frequently Asked Questions

What does Ragas NoiseSensitivity measure?

NoiseSensitivity estimates how often a RAG response contains incorrect claims associated with the retrieved evidence. It evaluates the question, generated response, reference answer, and retrieved contexts together. The result ranges from zero to one, and lower values indicate that the system is less affected by retrieval noise.

Which fields are required for Ragas noise sensitivity metric testing?

Each scored case needs user_input, response, reference, and retrieved_contexts. Keep the exact contexts seen by the generator, not a later retrieval replay. Also retain case ID, corpus version, retriever configuration, model and prompt versions, and context ranks so a changed score can be traced to its first differing input.

What is a useful positive control for NoiseSensitivity?

Use a question with an adjudicated reference, sufficient relevant context, and a response containing only claims supported by that reference. Run it beside a paired noisy variant. The positive control should preserve the expected claim set and gives the evaluator a known grounded case before it judges a release candidate.

Should a team gate CI on the average NoiseSensitivity score?

Do not use the average alone. A release gate should protect critical cases and declared slices, reject missing or non-finite results, compare against a versioned baseline, and enforce a cost ceiling. Aggregate movement can warn, while a severe case-level regression should remain visible instead of being diluted by easy examples.

Why can NoiseSensitivity results change between repeated runs?

NoiseSensitivity uses an evaluator model to identify claims, relevance, attribution, and correctness, so model sampling, provider changes, prompt changes, and ambiguous references can alter judgments. Pin the evaluator configuration, cache only version-matched inputs, repeat a calibration subset, and send material disagreements to human claim-level review rather than hiding them with retries.

How should relevant and irrelevant modes be interpreted?

The modes answer different diagnostic questions about incorrect response claims and their relationship to retrieved material. Record the selected mode with every result and avoid comparing unlabeled mixtures. Neither mode replaces context recall, context precision, answer correctness, security, or business-outcome tests, so use NoiseSensitivity as one diagnostic signal in a wider evaluation design.

Practice the Release Decision in QABattle

A good metric run ends with an engineering decision, not a colorful average. Take one clean/noisy pair, trace the first incorrect claim to its context and source version, classify the failure boundary, and write the smallest deterministic blocker that would have caught it. Then practice the same evidence-first reasoning in the QABattle battles arena.