Stop trusting a custom Ragas score you cannot explain

Your RAG release check turns red after a parser change, but nobody can explain which evidence went missing. The dashboard shows one decimal score, the raw evaluation row is gone, and the team starts debating the model instead of the metric. That is a metric design failure before it is a model failure.

A useful custom score leaves a trail from input to decision. A reviewer should be able to reproduce the number, see why a row was rejected, and tell whether the failing owner is retrieval, attribution, test data, or the release rule. Ragas can carry that score, but the framework cannot choose the contract for you.

Know what the number actually means

Ragas documents numeric metrics as metrics that return an integer or float within a declared range. The numeric_metric decorator is the shortest path when the calculation is ordinary Python. It wraps a function with scoring methods and validates the produced value against allowed_values.

That last phrase causes a surprising number of bad gates. An allowed range is not a pass threshold. With allowed_values=(0.0, 1.0), both 0.2 and 0.95 are structurally valid results. Whether either score should block a release belongs to a separate policy. Mixing those two jobs makes later debugging almost impossible: a calculation change, a framework validation error, and a policy change all look like “the metric failed.”

The documented scoring interface returns a MetricResult. Its value property carries the raw result. Its reason field can carry the explanation supplied by your metric or the wrapper's explanation for an invalid result. Use .score(...) with named arguments in evaluation code. That is also the pattern used by the current Ragas experiment quickstart, and it gives the caller a consistent object to inspect.

Named arguments matter here. A custom metric often grows from two inputs to five as the evaluation contract matures. Calls such as score(answer, reference, claims) silently become ambiguous in a code review. Calls such as score(required_claim_ids=..., supported_claim_ids=...) identify the role of each value and follow the decorator's scoring contract.

Before writing the function, settle four questions in plain language:

1. What single property does the number represent?
2. Which inputs are reviewed facts, and which are produced by the system under test?
3. What makes an evaluation row invalid rather than bad?
4. Which policy consumes the valid score?

For evidence coverage, the numerator can be the number of required claim identifiers supported by reviewed evidence. The denominator can be the number of required claim identifiers in that case. An empty denominator is not “perfect coverage.” It means the fixture supplied no requirement, so the evaluation cannot answer its own question.

The word “supported” also needs an owner. A deterministic metric can compare identifiers, but it cannot decide that a paragraph semantically supports a claim unless something upstream has made that judgment. That upstream component might be a human label, a rules engine, or an LLM grader with its own calibration tests. The numeric function should receive the decision, not pretend that set arithmetic proved the underlying semantics.

Keep the score narrow. Do not combine citation coverage, writing style, latency, factuality, and refusal quality into one weighted number because a dashboard has room for only one column. A fall in that composite cannot tell an engineer what broke. Separate metrics can still be reported together, but each needs a stable meaning.

This distinction is why a Ragas numeric custom metric is useful for a reviewed ratio and dangerous as a generic “quality score.” The decorator can enforce a numeric shape. It cannot make unrelated dimensions commensurable, establish a threshold, or validate the labels that fed the calculation.

Build a deterministic evidence coverage metric

Suppose a support answer must cover three reviewed claims: the refund window, the refund method, and the support channel. Another evaluation stage has already mapped citations to those claim IDs. The custom metric's job is only to calculate coverage and expose missing IDs.

The implementation below rejects ambiguous fixtures instead of converting them into misleading zeros. It also returns a reason that survives beside the score. Save it as metrics/evidence_coverage.py.

from ragas.metrics import MetricResult, numeric_metric


def _clean_ids(values: list[str], field_name: str) -> list[str]:
    cleaned = [value.strip() for value in values]

    if any(not value for value in cleaned):
        raise ValueError(f"{field_name} contains a blank claim ID")

    if len(cleaned) != len(set(cleaned)):
        raise ValueError(f"{field_name} contains duplicate claim IDs")

    return cleaned


@numeric_metric(name="evidence_coverage", allowed_values=(0.0, 1.0))
def evidence_coverage(
    required_claim_ids: list[str],
    supported_claim_ids: list[str],
) -> MetricResult:
    required = _clean_ids(required_claim_ids, "required_claim_ids")
    supported = _clean_ids(supported_claim_ids, "supported_claim_ids")

    if not required:
        raise ValueError("required_claim_ids must not be empty")

    required_set = set(required)
    supported_set = set(supported)
    unknown = sorted(supported_set - required_set)

    if unknown:
        raise ValueError(
            "supported_claim_ids contains unknown IDs: " + ",".join(unknown)
        )

    missing = sorted(required_set - supported_set)
    score = len(supported_set) / len(required_set)
    missing_text = ",".join(missing) if missing else "none"
    reason = (
        f"supported={len(supported_set)}; "
        f"required={len(required_set)}; "
        f"missing={missing_text}"
    )

    return MetricResult(value=score, reason=reason)

There are deliberate opinions in this code.

First, identifiers are case-sensitive. Silently applying lower() or casefold() could merge two IDs that the dataset owner intended to keep distinct. If your schema says IDs are case-insensitive, normalize them when the dataset is authored and validate that schema once. Do not hide normalization inside a metric where it is hard to audit.

Second, duplicates are invalid. Set conversion alone would remove them and change the apparent denominator. A duplicated required claim usually means a fixture generator appended the same requirement twice. A duplicated supported claim can make a list-based implementation overcount. Rejecting both states keeps the formula honest.

Third, a supported ID that is absent from the required set is not free extra credit. It suggests that the attribution output and the fixture refer to different case versions. Ignoring it would let a stale or incorrectly joined artifact pass through the metric.

Run a single worked example before connecting any experiment runner:

from metrics.evidence_coverage import evidence_coverage


result = evidence_coverage.score(
    required_claim_ids=[
        "refund-window",
        "refund-method",
        "support-channel",
    ],
    supported_claim_ids=[
        "refund-window",
        "support-channel",
    ],
)

assert result.value == 2 / 3
assert result.reason == (
    "supported=2; required=3; missing=refund-method"
)
print(result)

The important diagnostic is not the decimal representation of two thirds. It is missing=refund-method. That text points to the next artifact to inspect: the answer span and citation expected to support the refund method. If the answer never mentioned the method, the product response is incomplete. If the answer mentioned it and the citation points to a valid source, the attribution stage or its labels are wrong. The same numeric result can therefore lead to different owners, and the retained claim ID tells you which path to take.

Consider a second worked example. A dataset migration changes refund-method to refund-destination, but only the required list is updated. The supported list still contains the old ID. The function raises because the lists no longer share a vocabulary. Through the scoring wrapper, this is an invalid evaluation result, not zero coverage. That distinction stops a data migration defect from being reported as a model regression.

There is a cost to this strictness. Producers and consumers must agree on stable identifiers, and a schema migration needs an explicit mapping or a new fixture version. That work is worth doing for release gates. If you accept unknown IDs and deduplicate silently, the score stays green by discarding precisely the evidence you need during an incident.

Reasons also take storage. A CSV containing only the final float is smaller than one containing case ID, inputs, score, and reason. Storage is rarely the expensive part of an evaluation incident. Reconstructing an overwritten fixture or rerunning a non-reproducible attribution step is.

Prove the metric fails for the right reasons

A happy-path unit test proves only that division works. The risky behavior lives at the boundaries: no supported claims, all claims supported, malformed identifiers, an empty requirement set, and an output outside the declared range.

Parametrize the valid rows so every case exercises the same contract. Keep invalid rows in separate tests because they mean something different operationally. The following test module is runnable with pytest and the metric above.

import pytest
from ragas.metrics import numeric_metric

from metrics.evidence_coverage import evidence_coverage


@pytest.mark.parametrize(
    ("supported", "expected_value", "expected_reason"),
    [
        ([], 0.0, "supported=0; required=2; missing=c1,c2"),
        (["c1"], 0.5, "supported=1; required=2; missing=c2"),
        (["c1", "c2"], 1.0, "supported=2; required=2; missing=none"),
    ],
)
def test_evidence_coverage_boundaries(
    supported: list[str],
    expected_value: float,
    expected_reason: str,
) -> None:
    result = evidence_coverage.score(
        required_claim_ids=["c1", "c2"],
        supported_claim_ids=supported,
    )

    assert result.value == pytest.approx(expected_value)
    assert result.reason == expected_reason


def test_empty_required_set_is_invalid() -> None:
    result = evidence_coverage.score(
        required_claim_ids=[],
        supported_claim_ids=[],
    )

    assert result.value is None
    assert "required_claim_ids must not be empty" in (result.reason or "")


def test_unknown_supported_id_is_invalid() -> None:
    result = evidence_coverage.score(
        required_claim_ids=["c1"],
        supported_claim_ids=["old-c1"],
    )

    assert result.value is None
    assert "unknown IDs: old-c1" in (result.reason or "")


def test_missing_named_argument_is_a_call_error() -> None:
    with pytest.raises(TypeError, match="required_claim_ids"):
        evidence_coverage.score(supported_claim_ids=[])


@numeric_metric(name="broken_ratio", allowed_values=(0.0, 1.0))
def broken_ratio() -> float:
    return 1.2


def test_allowed_range_rejects_bad_metric_output() -> None:
    result = broken_ratio.score()

    assert result.value is None
    assert "expected value in range" in (result.reason or "")

The final test is a diagnostic for the metric implementation itself. It does not simulate a poor RAG answer. It proves that a function returning 1.2 cannot be mistaken for exceptionally good coverage when the valid domain is zero through one.

Current Ragas validation treats both ends of a tuple range as valid. That is why zero and one belong in the parameterized cases. If your business rule is “strictly greater than zero,” put that comparison in the release policy. Do not distort the metric range to encode it.

Notice the two failure shapes in the test module. A missing required argument fails input validation and raises TypeError. An exception raised inside the metric function, or a numeric output rejected by the range validator, can produce a MetricResult whose value is None and whose reason explains the problem. Callers must handle both. Catching every exception and substituting 0.0 would merge invalid execution with a valid score showing no evidence.

Do not assert the complete framework-generated error sentence unless you own the Ragas version and want the test to flag wording changes. The examples check the stable diagnostic fragment and the semantic state, value is None. By contrast, the reason produced by your own function is part of your contract, so an exact assertion is appropriate.

Mutation tests are useful even if you do not use a mutation-testing tool. Temporarily change the numerator to len(required_set). The partial and empty-supported cases must fail. Temporarily remove the duplicate check. A dedicated duplicate fixture must fail. Temporarily change the denominator to the supported count. The zero-supported case should expose the division error. Each mutation asks whether the suite can catch a plausible implementation mistake, not whether it can execute the function.

Avoid random claim lists in these unit tests. Property-based testing can add value later, but a release metric needs named examples that a reviewer can reason about without replaying a seed. Keep at least one case for each meaningful boundary and one case copied from a real, redacted incident. The incident row protects the shape that previously escaped.

There is a maintenance cost. Exact reasons and incident fixtures need updates when the contract changes. Treat that friction as a design signal. If a harmless refactor requires rewriting every expected reason, the diagnostic contains too much implementation detail. If a policy change requires no fixture or test update, the policy may not be represented anywhere executable.

Separate metric bugs from retrieval bugs

The same low coverage score can originate in four places. Debugging should move backward through artifacts instead of jumping straight to prompt changes.

Start with the metric result. If value is None, stop treating the row as a product score. Read reason and inspect the call. Missing arguments, an empty required set, duplicate IDs, unknown IDs, and an out-of-range result all point at the metric or evaluation data.

If the value is valid and low, inspect the normalized ID lists stored for that exact case. The reason from the example tells you which required IDs are missing. Confirm that the denominator matches the fixture version reviewed for this run. A changed denominator is a contract change, even when the human-readable question did not change.

Next, inspect the attribution artifact that produced supported_claim_ids. It should retain each claim ID, the cited document ID, and the answer span used for support. The numeric metric should not have to reopen documents or rerun a judge. When that artifact is absent, you cannot distinguish “the answer lacked evidence” from “the support mapper dropped an ID.”

Finally, inspect retrieval. Search the raw retrieved contexts for the evidence expected by the missing claim. Three outcomes look similar in a summary report but require different fixes:

This order prevents a common near-miss. A parser rollout changes document IDs but leaves text retrieval healthy. The generated answer still contains the right fact. Citations now refer to new IDs, while the support mapper looks for the old IDs and emits an empty supported set. The dashboard resembles a retrieval collapse, yet the raw contexts prove retrieval succeeded. Re-indexing the corpus would add cost and risk without repairing the broken join.

Another near-miss comes from denominator drift. A content owner adds two optional claims to the required list without updating the contract version. Coverage falls even though the answer and supported IDs are byte-for-byte unchanged. That is not a model regression. It is a policy change presented as historical comparison. Store the requirement-set version beside every score and compare only rows that share the same contract.

Threshold drift creates a third look-alike. The score remains unchanged, but CI starts failing because the minimum accepted value moved. The evaluation artifact should record both the raw value and the policy version that interpreted it. If only a Boolean passed field survives, engineers may waste hours trying to reproduce a score change that never occurred.

Aggregation can erase these clues. Suppose a batch contains valid low values and one invalid row. Converting None to zero lowers the average and blames the application. Dropping the row raises the average and hides missing evaluation coverage. Report invalid rows as a separate count, fail the evaluation lane if critical rows are invalid, and compute aggregates only over results that passed metric validation.

Do not let retries rewrite evidence. A deterministic metric over the same normalized lists should return the same value. If a rerun changes, an input changed or the execution path is not the function you think it is. Preserve both attempts with case IDs and input hashes. If the upstream support mapper is probabilistic, rerun and calibrate that component separately; averaging its decisions inside the deterministic ratio conceals instability.

The trade-off is artifact volume and some privacy work. Answer spans and contexts can contain sensitive data. Store the minimum excerpt needed to inspect support, apply the same access controls as the source corpus, and redact unrelated personal data before persistence. Do not solve privacy by deleting every diagnostic and keeping only the score. A release gate that cannot be audited is not safer.

Wire the score into CI without hiding bad rows

Roll out the gate in stages. First, run the metric in report-only mode against a small reviewed dataset. Check every invalid row manually. Then freeze the metric code, requirement-set version, and dependency version. Only after the team agrees on the score's meaning should a policy threshold block a merge.

The checker below reads JSON Lines so each input row remains independently inspectable. It gives invalid evaluations a different exit code from valid rows below policy. Save it as scripts/check_evidence_coverage.py.

from __future__ import annotations

import json
import math
import os
import sys
from pathlib import Path
from typing import Any

from metrics.evidence_coverage import evidence_coverage


def emit(record: dict[str, Any]) -> None:
    print(json.dumps(record, sort_keys=True))


def load_threshold() -> float:
    raw = os.environ.get("MIN_EVIDENCE_COVERAGE")
    if raw is None:
        raise SystemExit("MIN_EVIDENCE_COVERAGE is required")

    value = float(raw)
    if not math.isfinite(value) or not 0.0 <= value <= 1.0:
        raise SystemExit(
            "MIN_EVIDENCE_COVERAGE must be between 0.0 and 1.0"
        )
    return value


def check(path: Path, minimum: float) -> int:
    invalid_rows = 0
    failed_rows = 0

    with path.open(encoding="utf-8") as source:
        for line_number, line in enumerate(source, start=1):
            if not line.strip():
                continue

            try:
                row = json.loads(line)
                case_id = str(row["case_id"])
                result = evidence_coverage.score(
                    required_claim_ids=row["required_claim_ids"],
                    supported_claim_ids=row["supported_claim_ids"],
                )
            except (json.JSONDecodeError, KeyError, TypeError) as error:
                invalid_rows += 1
                emit(
                    {
                        "line": line_number,
                        "status": "invalid",
                        "reason": str(error),
                    }
                )
                continue

            if result.value is None:
                invalid_rows += 1
                emit(
                    {
                        "case_id": case_id,
                        "status": "invalid",
                        "reason": result.reason,
                    }
                )
                continue

            status = "pass" if result.value >= minimum else "fail"
            failed_rows += status == "fail"
            emit(
                {
                    "case_id": case_id,
                    "status": status,
                    "score": result.value,
                    "minimum": minimum,
                    "reason": result.reason,
                }
            )

    if invalid_rows:
        return 2
    if failed_rows:
        return 1
    return 0


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit(
            "usage: check_evidence_coverage.py PATH_TO_JSONL"
        )

    raise SystemExit(
        check(Path(sys.argv[1]), load_threshold())
    )

Exit code 2 means the evaluation itself is incomplete or malformed. Exit code 1 means the metric ran and at least one valid score missed the configured minimum. That split pays off during triage. A test-data owner can take the first class, while a product or retrieval owner investigates the second.

The script gates each row. It does not average them. That is intentional for a small release set where every case represents an accepted requirement. If your dataset contains exploratory or non-blocking rows, add an explicit blocking field reviewed in the dataset. Do not infer importance from a case name or quietly exclude low scores after the run.

Wire unit tests and release fixtures as separate CI steps. The threshold below is an illustrative policy value, not a measured recommendation. Replace it with a value justified by your reviewed cases.

name: evidence-coverage-contract

on:
  pull_request:

jobs:
  metric-contract:
    runs-on: ubuntu-latest
    env:
      MIN_EVIDENCE_COVERAGE: "0.85"
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - name: Install locked evaluation dependencies
        run: python -m pip install --requirement requirements-evals.txt
      - name: Test the metric contract
        run: python -m pytest -q tests/evals/test_evidence_coverage.py
      - name: Check reviewed release rows
        run: python scripts/check_evidence_coverage.py eval-data/release.jsonl

Keep the Ragas dependency locked in the evaluation environment. The public API and validation behavior can evolve, and a floating dependency makes a wrapper change look like an application regression. When upgrading, run the unit contract first, inspect invalid-result behavior, and only then compare application scores.

Do not generate the release fixture during the same CI job that evaluates it. A live model or current production corpus can change the requirement or support labels under test. Commit or otherwise version the reviewed fixture, refresh it through a separate approval path, and record why each change was made.

For an existing suite, a safe migration has five concrete checkpoints:

1. Add reason and contract-version fields without changing the old gate.
2. Run the new metric beside the old calculation on the same stored inputs.
3. Investigate every disagreement at row level instead of comparing only means.
4. Classify invalid rows separately and repair the dataset before choosing a threshold.
5. Switch the blocking rule, then retain the old result for a short, named observation window.

Dual-running costs CI time and produces extra columns. It also gives you a reversible comparison while owners learn the new diagnostics. Remove the old calculation after the observation window; maintaining two permanent definitions invites teams to cite whichever score supports their preferred conclusion.

Threshold calibration has its own cost. Review examples just below and just above each candidate boundary. Ask whether two engineers would make the same release decision from the underlying evidence. If not, adding decimal precision to the score will not resolve the disagreement. Improve the claim contract, split the population, or route borderline cases to review.

Watch runtime growth when an upstream judge produces supported IDs. The set calculation is cheap, but semantic attribution can require model calls. Cache only against complete, versioned inputs, including the answer, context, claim set, grader prompt, and model configuration. A cache keyed only by case ID can serve stale support labels after any of those values change.

Know when a numeric metric is the wrong tool

Do not use a ratio when one missing item is categorically unacceptable. A regulated disclosure may require every mandated claim. In that case, report the missing claim IDs and make the policy Boolean. A score near the top of the range should not make a prohibited omission look nearly good enough.

Avoid a deterministic evidence-coverage metric when “support” has not been resolved upstream. Comparing strings or IDs can verify label coverage, not semantic entailment. If the application paraphrases evidence, you may need a calibrated semantic grader or an existing Ragas faithfulness-style metric. Keep that judgment separate from the ratio so grader disagreement remains visible.

Do not collapse different user populations into one score if their requirement sets have different consequences. An answer for a casual FAQ and an answer used during a payment dispute may share a topic but not a release threshold. Separate suites or explicit policy groups make the decision legible.

A numeric metric is also the wrong output when the task is to choose the best of several responses. Absolute scores can be poorly calibrated across prompts even when pairwise preferences are stable. Use a ranking or comparative contract for that question, and preserve the candidate order rather than forcing it into a zero-to-one interpretation.

Sparse fixtures create another trap. With one required claim, the only possible coverage values are zero and one. Calling the result continuous adds no information. A discrete pass or fail result with the missing claim reason is clearer. With two claims, the score still has only three possible values, so a decimal-heavy chart exaggerates its resolution.

Never use the metric to reward verbosity. An answer can mention every required claim and still be wrong, unsafe, or buried under irrelevant text. Coverage answers whether reviewed requirements are represented. Pair it with separate factuality, safety, and usability checks when those properties matter, and never imply that coverage certifies them.

Finally, decline a release threshold when the dataset is not representative of the decision. A perfectly implemented metric over stale, duplicated, or convenience-sampled cases can provide a precise answer to the wrong question. Fix case selection first. The code can tell you that two of three reviewed IDs were supported; only your evaluation design can justify why those three IDs and that case deserve release authority.

The practical test is simple: hand one failed row to an engineer who did not write the metric. If the case ID, inputs, score, and reason lead that person to the same missing artifact and the same owner, the metric is doing useful QA work. If the conversation begins with “what does this number mean here?”, the next task is not threshold tuning. It is repairing the contract.