When your Ragas ranking metric picks the wrong response

Two candidate answers both score well, yet one cites the wrong policy and still wins the release comparison. The dashboard looks precise, but the scalar has erased the decision your reviewer actually cares about. A ranking test forces the evaluator to choose an order and gives you something concrete to investigate when that order moves.

Why a ranked list can reveal what a score hides

The difficult part is not calling a metric. It is proving that the candidates were comparable, the criterion meant one thing, and the returned order can survive changes in input position, model output, and test data. Without those controls, a list such as ["answer-b", "answer-a", "answer-c"] looks decisive while saying very little.

A scalar score answers, "How much of this quality did the evaluator see?" A ranking answers, "Which candidate better satisfies this criterion relative to the others?" Those are different test questions. If the release decision is whether a new response beats the current response, relative order is usually closer to the real decision.

Consider a support assistant that answers a password-reset question. Candidate A gives the correct steps but omits the identity check. Candidate B includes the identity check and the reset steps. Candidate C invents a menu item. Three independent 1-to-5 scores may put A and B close together because both are readable and mostly useful. A ranking criterion such as "Order by policy compliance, then task completion" makes the missing identity check decisive.

Ragas currently exposes RankingMetric for evaluations that produce ordered lists. Its reference documents allowed_values as the expected number of items in that list. The value is a count, not a three-point rating and not a release threshold. The metric result exposes the ordered output through result.value. That small distinction matters because an assertion such as result.value <= 3 is not merely weak. It is checking the wrong type of thing.

The order also needs a declared direction. Does the first item mean best or worst? Does the function return candidate text, indices, or IDs? The test fixture must settle those questions. Never infer direction from one happy-path example. Store an explicit convention such as "best candidate first" beside the oracle, then validate it in code.

Ranking does not remove subjectivity. It makes the subjectivity easier to locate. A useful comparison names one criterion and gives reviewers a reason they can challenge. "Best response" is too broad because one judge may favor completeness while another favors brevity. "Most faithful to the supplied refund policy, with unsupported claims ranked last" creates a testable ordering.

A relative result has one more trap: adding a weak candidate can change the order of the strong candidates. The judge may use the extra answer as an anchor, or the prompt may become long enough to dilute an important detail. That is why a good suite includes both pairwise and three-way fixtures. If A beats B when they are alone but loses to B when C is added, you have evidence of context sensitivity rather than a clean application regression.

Treat three layers as separate observations:

• The response generator produced specific candidate artifacts.
• The ranking metric returned a well-formed permutation of those artifacts.
• The returned order agreed with a reviewed expectation closely enough for the decision being made.

A malformed list belongs to the metric integration. A stable but wrong order points to the criterion, prompt, judge, or gold label. Different candidate text points upstream to the application. Combining those layers under one "eval failed" label wastes the best diagnostic information the run produced.

Build a comparison fixture a reviewer can audit

Use stable candidate IDs and keep the full text in the fixture. IDs make diffs readable and prevent whitespace or punctuation changes from breaking list comparisons. The human-readable reason belongs beside the expected order, not in a wiki that will drift away from the test.

The following fixture uses a deterministic Ragas ranking metric to demonstrate the current input and output contract. The rule is intentionally simple. It proves the harness before an LLM judge is introduced.

from ragas.metrics import ranking_metric


RESPONSE_TEXT = {
    "current": "Send the reset link after an identity check.",
    "challenger": "Send the reset link immediately.",
    "unsafe": "Read the old password back to the user.",
}


@ranking_metric(name="policy_evidence_order", allowed_values=3)
def rank_by_policy_evidence(
    user_input: str,
    responses: list[str],
) -> list[str]:
    required_phrases = ("identity check", "reset link")
    scored = [
        (
            sum(
                phrase in RESPONSE_TEXT[candidate_id].lower()
                for phrase in required_phrases
            ),
            candidate_id,
        )
        for candidate_id in responses
    ]
    return [
        candidate_id
        for _, candidate_id in sorted(
            scored,
            key=lambda item: (-item[0], item[1]),
        )
    ]

result = rank_by_policy_evidence(
    user_input="I cannot sign in. Reset my password.",
    responses=list(RESPONSE_TEXT),
)

assert result.value == ["current", "challenger", "unsafe"]

The tie-breaker on candidate_id is not a quality rule. It only makes this deterministic demonstration reproducible when two candidates match the same number of phrases. In a production rubric, tie behavior should be deliberate. You can allow ties in a separate representation, ask the judge for a strict order, or route close cases to review. Do not let an incidental input order act as the tie-breaker without documenting that choice.

A fixture needs more than an expected list. Record the user question, any retrieved context, candidate version, criterion version, and the reviewer rationale. If the ranking uses an LLM, record the judge model identifier and metric configuration available to your application. Avoid assuming a provider's default model or prompt stays fixed.

Here is a compact file format for reviewed cases. The values are examples of a test contract, not measurements from an experiment.

case_id: password-reset-policy-001
criterion_version: policy-faithfulness-v2
direction: best-first
user_input: "I cannot sign in. Reset my password."
context:
  - "Verify the account holder before issuing a reset link."
candidates:
  - id: current
    text: "I will verify your identity, then send a reset link."
  - id: challenger
    text: "I will send a reset link now."
  - id: unsafe
    text: "I can show you the password currently on the account."
expected_order:
  - current
  - challenger
  - unsafe
rationale:
  - "current follows both required steps"
  - "challenger omits identity verification"
  - "unsafe claims an unsupported and unsafe capability"
gate: must-win

The gate field prevents every example from carrying the same release weight. A must-win fixture has an obvious, reviewed distinction. A monitor-only fixture represents a close editorial preference or a disputed criterion. Mixing those classes produces a gate that is either noisy enough to be ignored or weak enough to miss a serious reversal.

Before trusting an automated ranker, ask two reviewers to order a calibration set independently. Resolve cases where they interpreted the criterion differently. This is not an exercise in forcing agreement. It finds ambiguous instructions and under-specified examples before model variation gets blamed for them.

Keep generated responses immutable during comparison. If the suite regenerates candidates on every metric run, a changed order could come from different answers or a different judge. Save the candidate artifacts first, then rank that frozen set. Test generation drift and evaluator drift in separate jobs.

Exercise three failures that look different once you inspect the evidence

A strong suite does more than confirm one expected order. It attacks the properties that make the order trustworthy.

Failure one: the result is not a permutation of the candidates.

A ranker can duplicate an item, omit an item, or return an unknown label. This is a contract failure even if the preferred candidate appears first. Check shape before checking quality.

def assert_valid_ranking(
    candidate_ids: list[str],
    ranked_ids: list[str],
) -> None:
    if len(ranked_ids) != len(candidate_ids):
        raise AssertionError(
            f"expected {len(candidate_ids)} ranked items, got {len(ranked_ids)}"
        )

    expected = set(candidate_ids)
    actual = set(ranked_ids)

    missing = sorted(expected - actual)
    unknown = sorted(actual - expected)
    duplicates = sorted(
        candidate_id
        for candidate_id in actual
        if ranked_ids.count(candidate_id) > 1
    )

    if missing or unknown or duplicates:
        raise AssertionError(
            f"invalid ranking: missing={missing}, "
            f"unknown={unknown}, duplicates={duplicates}"
        )


assert_valid_ranking(
    ["current", "challenger", "unsafe"],
    ["current", "challenger", "unsafe"],
)

With ["current", "current", "unsafe"], this helper reports the missing challenger and duplicated current candidate. That message is more useful than a generic equality diff because it identifies an integration defect. It also stops a malformed result from entering agreement calculations that assume a true permutation.

Check the raw metric result when this fails. If result.value already contains a duplicate, investigate the metric function or judge output. If the raw result is valid but the saved artifact is not, investigate serialization and ID mapping. If the candidates in the artifact differ from the candidates sent to the metric, the fixture assembly is at fault.

Failure two: candidate position changes the winner.

Position bias can look like ordinary nondeterminism if the suite only repeats one ordering. Run the same content under multiple permutations, map outputs back to stable IDs, and count the first-place choice. For three candidates, all six permutations are cheap enough for a focused diagnostic. For larger lists, use a fixed sample of permutations so the job stays bounded.

from collections import Counter
from itertools import permutations
from typing import Callable


Ranker = Callable[[list[str]], list[str]]


def first_place_by_permutation(
    candidate_ids: list[str],
    ranker: Ranker,
) -> Counter[str]:
    winners: Counter[str] = Counter()

    for presented_order in permutations(candidate_ids):
        ranked = ranker(list(presented_order))
        assert_valid_ranking(candidate_ids, ranked)
        winners[ranked[0]] += 1

    return winners


def suspicious_first_item_ranker(presented: list[str]) -> list[str]:
    return presented.copy()


winners = first_place_by_permutation(
    ["current", "challenger", "unsafe"],
    suspicious_first_item_ranker,
)

assert winners == Counter(
    {"current": 2, "challenger": 2, "unsafe": 2}
)

The demonstration ranker simply echoes the input. The even winner count does not make it good. It proves that content has no influence at all. Pair this diagnostic with the reviewed expected winner. A healthy must-win case should keep the correct candidate first across presentation orders, subject to the repeatability policy your team has calibrated.

Evidence for position sensitivity is specific: the candidate placed first in the prompt keeps winning, while candidate text and criterion remain unchanged. That differs from temperature or provider variation, where the winner may move without a consistent relationship to input position. Preserve the presented order and returned order for every attempt. A final majority winner alone hides the pattern.

Failure three: a near-tie is treated as a catastrophic reversal.

Exact list equality gives the top and bottom swap the same failure weight as a swap between adjacent middle items. Sometimes that is correct. Often it is not. If only the winner matters, assert the winner. If the whole order matters, calculate agreement and set the threshold from reviewed examples.

The next utility measures the fraction of candidate pairs that appear in the same relative order. It is deliberately implemented in plain Python so the meaning of the number stays visible.

from itertools import combinations


def pairwise_agreement(
    expected: list[str],
    actual: list[str],
) -> float:
    assert_valid_ranking(expected, actual)
    expected_position = {
        candidate_id: index
        for index, candidate_id in enumerate(expected)
    }
    actual_position = {
        candidate_id: index
        for index, candidate_id in enumerate(actual)
    }

    pairs = list(combinations(expected, 2))
    agreed = sum(
        (
            expected_position[left] < expected_position[right]
        )
        == (
            actual_position[left] < actual_position[right]
        )
        for left, right in pairs
    )
    return agreed / len(pairs) if pairs else 1.0


assert pairwise_agreement(
    ["a", "b", "c"],
    ["a", "c", "b"],
) == 2 / 3

assert pairwise_agreement(
    ["a", "b", "c"],
    ["c", "b", "a"],
) == 0.0

These values come from the shown lists, not from a production evaluation. For three candidates there are three pairs. Swapping the bottom two preserves two of them, while reversing the list preserves none.

Do not pick an agreement threshold because 0.8 sounds strict. Label a calibration set, run the evaluator repeatedly, and inspect which disagreements would have changed a release decision. A threshold earns authority only after it separates acceptable variation from failures the team agrees matter.

Tell metric drift from an application regression

The fastest diagnosis starts by freezing the candidate payload. Hashing or versioning the artifact is useful, but the readable text must remain available. When a run fails, compare the exact user input, context, and candidate texts with the previous passing run.

If those inputs changed, the ranking failure may be valid. A new response may have omitted evidence, added an unsupported claim, or changed its refusal behavior. Send that case to the application owner with the candidate diff and the evaluator rationale.

If the inputs are byte-for-byte identical, inspect the evaluator layer. Check the criterion version, Ragas version, judge configuration, and presented order. A dependency upgrade can change a metric contract. A prompt edit can reverse the meaning of best-first. A model alias can point at a different provider snapshot. Report those facts rather than calling all of them "model drift."

A useful failure artifact looks like this:

case: password-reset-policy-001
candidate_digest: 6f57c2f...
criterion_version: policy-faithfulness-v2
presented_order: [unsafe, current, challenger]
expected_order: [current, challenger, unsafe]
actual_order: [challenger, current, unsafe]
pairwise_agreement: 0.6666666667
must_win_expected: current
must_win_actual: challenger

This is illustrative diagnostic output from the worked fixture. It is not a claimed measurement. The important clue is that the top two candidates swapped while the unsafe answer stayed last. That points toward the evaluator's handling of identity verification, not a total loss of ranking structure.

A different artifact might show unknown=["candidate-3"] after candidates were renamed to semantic IDs. That is mapping drift. Another might show the expected winner losing only when it appears last in presented_order. That is order sensitivity. The same red CI badge can represent three different owners.

Repeatability also needs two baselines. First, rerun one frozen prompt several times to observe judge variation. Second, rerun several input permutations to observe position sensitivity. Do not combine them into one unlabeled batch. Otherwise you cannot tell whether a changed winner tracks sampling or position.

Human labels can be wrong. When the evaluator repeatedly produces a defensible order that conflicts with the fixture, reopen the rationale. Keep the old label in history, record who approved the change, and increment the criterion or fixture version. Quietly editing expected_order to make CI green destroys the evidence needed to judge future changes.

A near-miss deserves special care. Suppose the trace shows the correct candidates and a valid three-item result, yet the metric's explanation mentions "friendly tone" while the criterion says "policy faithfulness." The list may even match by chance. Treat criterion leakage as a metric defect. Passing order alone is not enough when the reason proves the judge answered a different question.

Roll the checks into CI without turning variation into noise

Keep deterministic contract tests on every change. They should validate fixture parsing, candidate ID uniqueness, ranking shape, direction, and agreement math without making a network call. These tests are fast and should fail hard.

Put judge-backed comparisons in a separate job. Pin the package version and the judge configuration your integration supports. Limit the default pull-request set to reviewed must-win cases. A broader set can run on schedule or when evaluation code changes.

name: response-ranking-evals

on:
  pull_request:
    paths:
      - "evals/**"
      - "src/assistant/**"
  workflow_dispatch:

jobs:
  ranking-contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install -r requirements-evals.txt
      - run: python -m pytest tests/evals/test_ranking_contract.py -q

  reviewed-comparisons:
    needs: ranking-contract
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install -r requirements-evals.txt
      - run: python -m pytest tests/evals/test_must_win_cases.py -q
        env:
          JUDGE_API_KEY: ${{ secrets.JUDGE_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: ranking-evidence
          path: artifacts/ranking/

The YAML assumes the named requirement files and tests exist in your repository. Adapt paths to the suite you actually run. The important wiring is the separation between deterministic contract checks and judge-backed comparisons, plus artifact retention on failure.

A release rule should state what blocks and what creates review. Good blocking candidates include an invalid result shape, an unknown candidate ID, a must-win reversal confirmed on repeated runs, or a large agreement drop across a stable reviewed set. Close editorial cases should not make every pull request hostage to one model response.

Budget is a real trade-off. Permuting three candidates creates six evaluations per case. Repeating each permutation multiplies cost and latency again. Use exhaustive permutations for a small diagnostic set, not for every sample. For the main suite, rotate candidates using a deterministic seed and run the full position-bias battery on schedule.

Versioned artifacts make rollout safer. Start by collecting results without blocking. Review reversals for a week or a fixed number of changes, refine ambiguous criteria, and mark a small group as must-win. Turn on the gate only after the team can explain failures without rerunning until green.

When replacing scalar gates, run both systems side by side. Do not pretend their numbers are interchangeable. Record cases where the scalar passes but the ranking fails, and ask which decision matches reviewer judgment. Retire the old gate once the new contract has owners, calibration data, and a documented escape path for a genuine evaluator incident.

Diagnose a valid wrong order that the judge never actually saw

A valid but reversed list has a second cause that looks almost identical to judge disagreement in the ordinary failure log: the evaluator request omitted the evidence that separates the candidates. The source fixture can be complete while prompt assembly drops a context item, clips the end of a long candidate, or selects an older rendered artifact. In both cases actual_order puts the challenger first, the list is a valid permutation, and another run may repeat the same result. Changing the rubric or blaming model variation is the wrong response when the judge never received the decisive sentence.

Separate those causes with the final evaluator-visible request, not only the fixture checked into the repository. Preserve a sanitized rendering of the criterion, context, candidate IDs, and candidate text in the exact order supplied to the metric. Check that each candidate occurs once and that the policy passage cited by the reviewer is present. A judge defect has a complete rendered request and an indefensible order. An assembly defect has a reasonable order for the incomplete material it received. That distinction assigns the repair before anyone spends judge calls on repeated runs.

Read the existing failure artifact from identity outward. candidate_digest should match the digest produced when the reviewed candidates were frozen. presented_order should contain the same IDs as expected_order, even though their positions may differ. actual_order should also contain exactly that set. On a healthy must-win case, must_win_actual equals must_win_expected, and the agreement value is interpreted only after those identity checks pass. On the broken example, the challenger wins while the unsafe answer remains last, so the failure is narrow rather than a total ordering collapse.

One apparently healthy value is misleading. pairwise_agreement can be perfect for the candidates the judge saw even when one candidate was clipped before evaluation. It measures the returned order against the supplied IDs, not completeness of the associated text. A stable candidate_digest can also mislead if it describes the stored fixture rather than the rendered request. Name digests by boundary in the artifact or retain enough sanitized text to prove which boundary each digest represents. A single unlabeled hash invites false confidence.

An established suite usually breaks first in its result wrapper. Older wrappers often retain only result.value, because that was enough for a pass or fail. Land evaluator-input capture before strengthening any ranking gate. Next, replay saved must-win fixtures through the current assembly path and verify that reviewer-cited evidence survives rendering. Only then add completeness checks to the deterministic job. Let those checks report without blocking until the team has separated real defects from fixtures that cannot be safely retained.

After the capture path is trusted, enable the new evidence on a few existing must-win cases, then on the broader scheduled suite. Do not change the judge, rubric, candidate generator, and artifact format in the same release. If the order moves, simultaneous changes erase the comparison needed to tell which layer moved. The rollout is working when a failed case can be assigned from its first artifact, and when rerunning an incomplete prompt is no longer the default diagnostic step.

This evidence has a concrete cost. Storing every rendered request for every permutation multiplies artifact volume, and support examples may contain customer text that should not live in a general CI artifact. Redaction can remove the very phrase needed to diagnose the result. Keep a small, reviewed fixture set free of live customer data, limit retention, and store a digest plus approved excerpts when the full prompt is too sensitive. The team accepts less forensic detail in exchange for a smaller privacy and storage surface.

Ownership crosses four boundaries. The application team owns the frozen candidate and retrieved context. The evaluation team owns rubric rendering, candidate mapping, and the ranking assertion. The platform team owns evidence from the model request boundary when that boundary is shared infrastructure. The domain reviewer owns the policy rationale and the decision about whether a must-win label is still valid. A handoff should include the case ID, source and rendered candidate digests, presented and returned orders, the missing or disputed passage, the criterion version, and one reproduction using frozen inputs. It should not contain a vague screenshot of a red aggregate score.

This technique does not catch a common-mode factual error. If the source context, every candidate, the rubric, and the human rationale all repeat the same false policy, the ranking can be perfectly stable and still reward the wrong answer. Ranking evidence shows how a supplied comparison was decided. It does not establish that the underlying policy source is authoritative or current.

Know when a ranking metric is the wrong tool

Do not rank when the requirement is absolute. If every medical answer must include an emergency warning, test for that warning directly. A bad answer beating a worse answer is still bad. Relative order cannot prove a minimum safety bar.

Avoid ranking candidates that answer different questions or use different context. The evaluator may prefer one because its task is easier. Normalize the input first, or evaluate each response against its own reference with an appropriate metric.

Skip an LLM judge when a deterministic oracle exists. Citation URLs can be checked against an allowed source set. JSON can be schema-validated. A calculation can be compared with an exact expected value. Use ranking for the semantic distinction that remains, not as a replacement for cheaper facts.

A leaderboard is also a poor substitute for error analysis. Winning 60 reviewed comparisons might hide a consistent failure for one language, product area, or refusal type. Slice results by meaningful scenario and inspect the reversals. Do not invent confidence from the total count.

Do not use strict ordering for cases where reviewers genuinely accept a tie. Forcing a first-place answer creates label noise and trains the team to dismiss failures. Represent acceptable sets, compare only the critical pair, or keep the case out of the gate.

Finally, do not compare live generated responses and call every moved rank evaluator drift. Freeze the candidate set when testing the metric. If the purpose is an end-to-end release eval, preserve both the generation artifact and ranking artifact so the failure can be assigned to the right layer.

The cost of a defensible ranking suite is more fixture work, more judge calls, and more review than a single average score. In return, the team gets a result that matches a real product choice and evidence precise enough to challenge that choice when it is wrong.