Find the entities your RAG retriever keeps dropping

A support bot gives the right refund window but leaves out the payment provider that must process the refund. The generator looks guilty, yet its retrieved passages never contained that provider. Fixing the prompt will not recover evidence that retrieval dropped.

This is where entity recall earns its place. It asks a narrow question: did the retrieved text cover the people, products, places, dates, identifiers, and other entities present in the reviewed reference? That narrowness is useful, but only if the test preserves enough evidence to explain a low score.

What the score actually tells you

Ragas documents ContextEntityRecall as the fraction of reference entities also found in the retrieved contexts. In set notation, let RE be the entities extracted from the reference and RCE the entities extracted from all retrieved contexts. The metric evaluates the size of RE intersected with RCE, divided by the size of RE.

The current collections API accepts two inputs: a reference string and a list named retrieved_contexts. It returns a result object whose value holds the score. The metric uses an evaluator model to identify entities, so the calculation is simple after extraction, but the extracted sets are model judgments rather than a parser contract.

That distinction matters during triage. A low result can mean at least four different things:

1. The retriever did not return a passage containing a required entity.
2. The corpus did not contain the required evidence in retrievable form.
3. The reference was wrong, stale, or overloaded with details the product never promised to answer.
4. The evaluator extracted or matched entities differently from the reviewer.

Only the first item is an uncomplicated retrieval defect. The second belongs to ingestion or content ownership. The third is a dataset defect. The fourth calls for evaluator review, not an immediate production rollback.

The metric also says nothing about relationships between entities. A passage containing "AcmePay", "order 8841", and "14 days" covers those strings even if it says AcmePay rejected order 8841 after 14 days. The correct reference might say AcmePay approved the refund within 14 days. Entity overlap is high while the proposition is wrong. Faithfulness and correctness need their own checks.

Nor does the metric punish retrieval noise directly. Ten irrelevant passages plus one passage containing every reference entity can score well on entity recall. That run may still waste context tokens, distract the generator, and increase latency. Recall is the coverage side of retrieval. It is not precision in disguise.

The reference controls the denominator, which makes reference quality a testing concern. A terse reference such as "Contact AcmePay" may yield one important entity. A detailed reference copied from an internal runbook may include employee names, ticket numbers, office locations, and dates that users never need. The second case is harder by construction, even if both represent the same expected answer. Review references as product requirements, not prose samples.

Here is the smallest current API call worth keeping as a smoke test. It uses the collections import shown in the official documentation. The model name is an example configuration, not a claim that one model is universally best.

import asyncio
import os

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextEntityRecall


async def main() -> None:
    client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
    evaluator = llm_factory("gpt-4o-mini", client=client)
    metric = ContextEntityRecall(llm=evaluator)

    result = await metric.ascore(
        reference=(
            "AcmePay must refund order 8841 to Priya Rao within 14 days."
        ),
        retrieved_contexts=[
            "Refunds for order 8841 are handled within 14 days.",
            "The customer on the request is Priya Rao.",
        ],
    )
    print({"metric": "context_entity_recall", "score": result.value})


if __name__ == "__main__":
    asyncio.run(main())

Do not turn the printed value into an unexplained pass or fail. Store the case ID, reference revision, retrieved text or immutable document IDs, corpus version, evaluator model, metric package version, and score. Without those fields, a later investigator cannot tell retrieval drift from evaluation drift.

Which worked examples expose real retrieval failures

The fastest way to understand this metric is to hold most of the pipeline still and change one failure source at a time. The examples below use compact cases, but each represents a defect pattern that appears in production RAG systems.

A policy answer loses the responsible provider

Suppose the approved answer says: "AcmePay must refund order 8841 to Priya Rao within 14 days." Retrieval returns the order, customer, and time limit, but not AcmePay. The response model may produce a fluent generic instruction such as "The refund will arrive within 14 days." A general correctness judge might accept it if the provider is not emphasized. Entity coverage makes the missing operational owner visible.

Before adjusting embeddings or reranking, confirm the corpus. Search the indexed source for "AcmePay" and the canonical order policy. If the text is absent, the retriever cannot return it. The defect belongs to ingestion, permissions, source freshness, or the reference. If the text exists and a direct lexical query finds it while the application query does not, then query construction, filtering, embedding retrieval, or reranking becomes plausible.

The diagnostic below does not imitate Ragas entity extraction. It creates a reviewer-owned ledger for this one case. That is intentional. It gives the engineer a deterministic way to see which required facts appear literally in the captured retrieval, while the Ragas score remains the semantic evaluation signal.

from dataclasses import dataclass


@dataclass(frozen=True)
class RequiredEntity:
    label: str
    accepted_forms: tuple[str, ...]


required = (
    RequiredEntity("payment_provider", ("AcmePay",)),
    RequiredEntity("order_id", ("8841", "order 8841")),
    RequiredEntity("customer", ("Priya Rao",)),
    RequiredEntity("refund_window", ("14 days", "fourteen days")),
)

retrieved_contexts = [
    "Refunds for order 8841 are handled within 14 days.",
    "The customer on the request is Priya Rao.",
]
haystack = "\n".join(retrieved_contexts).casefold()

ledger = {
    item.label: any(form.casefold() in haystack for form in item.accepted_forms)
    for item in required
}
missing = [label for label, present in ledger.items() if not present]

print({"entity_ledger": ledger})
print({"missing_reviewed_entities": missing})

assert missing == ["payment_provider"]

This assertion is valid because the accepted forms are reviewed fixture data. It is not a general named-entity recognizer. Maintaining the aliases costs time, but that cost buys reproducibility for release-critical cases.

A filter removes the right regional document

Consider a travel assistant asked about a train from Paris to Zurich on 18 October. The reference mentions Paris, Zurich, SNCF, and the date. The correct document exists, but the request carries a region filter set to ch. A metadata rule removes the French operator page before semantic ranking runs. Retrieval still returns Swiss travel advice containing Zurich and the date, so the answer looks plausible.

An entity score may fall because Paris or SNCF is missing. The fix is not necessarily a lower similarity threshold. Capture candidates at three boundaries: before metadata filtering, after filtering, and after reranking. If the relevant document appears in the first list and disappears in the second, the filter is responsible. If it survives filtering and loses at reranking, inspect the reranker. If it never appears, inspect query formation, indexing, or access control.

This example shows why a score without retrieval stages is weak evidence. Two defects can produce the same final contexts. The first needs a metadata rule change. The second may need corpus or ranking work. Save document IDs and rejection reasons rather than logging only passage text.

For sensitive systems, raw passage logging may be prohibited. Store stable document IDs, content hashes, rank, score, filter decisions, and a secured trace reference. The evaluator still needs text to score, but the general CI artifact does not need to expose it to every job viewer.

A rename creates an evaluator disagreement

A product catalog changes "Northwind Identity Cloud" to "Northwind Access". The reference is updated immediately, while indexed documents and accepted aliases still use the old name. Reviewers know the names refer to one product. An evaluator may treat them as distinct entities, depending on the wording and model.

That result is not automatically a retriever regression. Look for a controlled alias case containing both names in a single sentence, then score it with the same evaluator configuration. Check the raw retrieved source and the reference revision. If evidence is semantically present under the former name, decide whether the product requires the new name in sources. The right response may be a corpus migration, an accepted-reference update, or a manual review lane.

Do not silently add both names to every reference. That changes the denominator and can hide incomplete migrations. Keep a separate alias map with an owner and effective date. References should say what a correct answer must communicate, while the alias fixture documents what the evaluation system may consider equivalent.

These three cases need different fixes even though each can produce a lower score. The provider case lacks required evidence. The regional case loses evidence during filtering. The rename case may contain the evidence under an equivalent surface form. A useful test report names that distinction.

How to prove retrieval caused the failure

Start triage with artifacts, not the aggregate. For each failed row, retrieve the exact inputs used by the metric. Then walk backward through the application pipeline until the required evidence reappears.

A practical case record contains:

• A stable case ID and dataset revision.
• The reviewed reference and its source or approval record.
• The user query exactly as sent to retrieval.
• Retrieved document IDs in rank order, plus the text given to the evaluator.
• Filters, tenant or permission scope, and index or corpus version.
• The evaluator model and metric package version.
• The raw per-case result, not just a suite mean.
• A human disposition such as retrieval defect, corpus gap, reference defect, evaluator dispute, or expected change.

Run a direct corpus lookup for each missing reviewed entity. A lexical search is often enough for identifiers and proper nouns. When the term exists, compare the application query against a query that names the entity. This is a diagnostic probe, not the product fix. If only the probe retrieves the document, the original query did not carry enough discriminating information or later ranking removed it.

Next, replay the same case against the same corpus snapshot. A replay that passes today does not erase yesterday's failure. It indicates nondeterminism, mutable data, changed evaluator behavior, or an incomplete capture. Compare corpus version, model, package version, and retrieved IDs before calling the first run flaky.

The following script validates the shape of a captured result and prints a useful failure line. It is deliberately independent of Ragas so malformed runs fail before quality thresholds are considered.

import json
import sys
from pathlib import Path


def validate_row(row: dict) -> list[str]:
    errors: list[str] = []
    required = (
        "case_id",
        "dataset_revision",
        "corpus_version",
        "reference",
        "retrieved_contexts",
        "evaluator_model",
        "score",
    )
    for field in required:
        if field not in row:
            errors.append(f"missing field: {field}")
    if "retrieved_contexts" in row and not isinstance(row["retrieved_contexts"], list):
        errors.append("retrieved_contexts must be a list")
    if "score" in row and not isinstance(row["score"], (int, float)):
        errors.append("score must be numeric")
    return errors


path = Path(sys.argv[1])
rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
failed = False

for row in rows:
    errors = validate_row(row)
    if errors:
        failed = True
        print(f"INVALID {row.get('case_id', '<unknown>')}: {', '.join(errors)}")
    else:
        print(
            f"CASE {row['case_id']} score={row['score']:.4f} "
            f"contexts={len(row['retrieved_contexts'])} "
            f"corpus={row['corpus_version']}"
        )

raise SystemExit(1 if failed else 0)

If a report says only context_entity_recall=0.61, you cannot distinguish one catastrophic case from many small misses. Inspect the score distribution and the actual failed case IDs. Segment by intent, language, tenant, content type, and query difficulty only when those labels were defined before looking at the results. Ad hoc slicing can turn any outcome into a story.

Watch for NaN, missing values, and evaluator errors. Do not coerce them to zero, because zero means a completed evaluation found no covered reference entities. Do not coerce them to one, because completion is unknown. Treat incomplete evaluation as its own operational failure and rerun only after preserving the error.

A capture defect can look identical to a retrieval miss in the metric log. The runtime selects the document containing AcmePay, but the adapter that builds retrieved_contexts drops that document, sends an earlier candidate list, or replaces its text during serialization. Ragas correctly scores the input it received. The low value is real for the evaluation payload but false as a description of production retrieval. Tuning embeddings in response would leave the broken measurement path untouched.

Compare two boundaries using stable document identity and content hashes. The runtime retrieval record should name the final documents supplied to prompt construction. The metric input record should name the documents whose text became retrieved_contexts. If the required document is absent from both, retrieval or an upstream filter remains responsible. If it appears in the runtime final list but not in the metric input, evaluation capture is responsible. If the ID appears in both but hashes differ, inspect truncation, normalization, version lookup, and serialization before judging retrieval.

Read a case line in this order: case ID, dataset revision, corpus version, runtime final document IDs, evaluator input IDs, the count at each boundary, content hashes, evaluation status, and score. A healthy case has comparable versions and matching identities at both boundaries before its value is interpreted. A broken retrieval case lacks the required ID at the runtime boundary and faithfully carries that absence into evaluation. A broken capture case contains the ID at runtime and loses it later. A misleading line can show four contexts at both boundaries while one required context was replaced by a duplicate, so equal counts do not prove equal evidence.

Prompt packing creates another boundary. If entity recall evaluates the full pre-budget retrieval list while the generator receives a shortened list, a high score does not show that the model saw the entities. Preserve the exact generator-visible context bundle or label the metric as pre-packing retrieval coverage. Context entity recall does not catch token-budget truncation that occurs after its recorded input. That failure needs a prompt-assembly contract test.

Which lookalike failures need a different fix

Several failures resemble missing retrieval evidence in the final answer but sit outside retrieval.

The first near-miss is a generation omission. The contexts contain AcmePay, order 8841, Priya Rao, and 14 days, yet the response leaves out AcmePay. Context entity recall should remain high because it examines the reference and contexts, not the response. A response-level correctness or completeness check should fail. Changing retrieval to return more passages adds noise without addressing the generator's omission.

The second is a wrong relationship. The retrieved passage contains all entities but says AcmePay rejected the request. The response repeats that claim even though the reference expects approval. Entity recall can still look healthy. Inspect the source's proposition and run a faithfulness or correctness evaluation appropriate to the answer. If the source itself is wrong, content quality owns the defect.

The third is over-retrieval. Every required entity appears somewhere in a long context bundle, but most passages are irrelevant. Entity recall rewards coverage and cannot tell that the useful evidence was buried. Check context precision, rank position, context size, latency, and answer quality. A higher top_k often raises recall while increasing noise and cost.

The fourth is reference leakage. Someone generated references from the same retrieval system being evaluated. Those references naturally repeat entities the retriever already favors, so coverage looks strong. A new corpus slice or user intent may fail badly. Trace each reference to an independent reviewed source. If references came from production outputs, label them and keep them out of the primary release set until reviewed.

The fifth is access-control behavior. A document containing an entity exists, but the test identity must not retrieve it. A low score is correct if the reference assumes broader permissions than the fixture user has. Record identity and authorization scope, then repair the reference or test persona. Never loosen production authorization to satisfy an evaluation reference.

Use a simple decision sequence:

1. Is the reference approved for this user intent and permission scope?
2. Does the eligible corpus contain the required evidence?
3. Did the document enter the candidate set?
4. Did filtering or reranking remove it?
5. Did the evaluator identify the same entities a reviewer identified?
6. If retrieval coverage is good, did generation still omit or distort the evidence?

This order stops teams from tuning similarity thresholds when the defect is actually a stale reference or forbidden document.

How to introduce a CI gate without hiding drift

Begin with observation. Run the metric on a small reviewed set and publish per-case artifacts without blocking merges. Include obvious controls: full entity coverage, one missing entity, irrelevant passages containing none of the required entities, an alias case, and a malformed row. Controls tell you whether the evaluation harness changed before you interpret product results.

After reviewers agree on dispositions, assign policies by slice. A billing identifier case may require all critical entities. A broad discovery question may tolerate lower entity coverage because no finite reference enumerates every useful answer. One global threshold turns unlike risks into a misleading number.

The CI skeleton below deliberately supplies no universal threshold. Put reviewed case and slice policy in the tests, where a pull request can show who changed it and why. A copied value such as 0.80 is not evidence that the same cutoff fits your product.

name: rag-entity-evaluation

on:
  pull_request:
    paths:
      - ".github/workflows/rag-entity-evaluation.yml"
      - "src/retrieval/**"
      - "evals/context_entities/**"
      - "evals/requirements.lock"

jobs:
  context-entity-recall:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: python -m pip install -r evals/requirements.lock
      - run: mkdir -p artifacts
      - run: python -m pytest evals/context_entities -q --junitxml=artifacts/context-entities.xml
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: context-entity-recall-results
          path: artifacts/context-entities.xml

Pin the evaluation dependencies in the lock file and record their versions in the artifact. Pinning reduces unreviewed evaluator drift, but it also delays improvements and security fixes. Schedule an explicit dependency-update job that runs old and new versions on the same frozen cases, then reviews differences before changing the release lane.

Separate three outcomes. A valid score below a calibrated floor is a quality failure. A missing field, timeout, authentication error, or evaluator exception is an evaluation-infrastructure failure. A known alias dispute or changed reference is a review result. Collapsing all three into "failed eval" makes ownership impossible.

Do not rely only on the mean. Gate named critical cases individually, then apply slice-level policy to the rest. A high mean can hide one missing legal entity, account identifier, contraindication, or payment provider. Conversely, one ambiguous low-risk case should not necessarily stop every deployment. Criticality belongs in dataset metadata reviewed before execution.

A safe rollout usually follows four stages. First, run locally on five to ten reviewed cases and verify artifacts. Second, add a non-blocking CI job and compare its classifications with human review. Third, block only structural failures and named critical cases. Fourth, add slice thresholds once you have enough reviewed history to defend them. Every expansion should have an owner and a rollback path for the gate, not for the product behavior it detects.

In an existing suite, add boundary provenance before enforcing a score. Legacy cases often have answer text and a decimal but no corpus version, runtime document IDs, or permission scope. Do not fill those fields with today's values. Mark the old row non-comparable, preserve it for history, and create a new reviewed revision. Land report readers that understand valid, infrastructure-failed, review, and non-comparable states before producers begin emitting them. Otherwise the first schema change can silently drop rows from the dashboard and make coverage appear better.

The change is working when known controls land in their expected categories, runtime and evaluator evidence match for comparable cases, and repeated runs preserve disposition even if a model-backed score moves slightly. Watch the number of evaluable cases as closely as the score distribution. A rising average paired with fewer comparable rows is a rollout failure, not an improvement.

Retrieval engineering owns candidate generation, filters, ranking, and the runtime final list. The evaluation owner owns serialization into metric inputs, evaluator configuration, and incomplete-run handling. Content or ingestion owners answer whether the eligible corpus contains the approved fact. Dataset reviewers own the reference and permission scope. A useful handoff includes the case ID, query, user scope, dataset and corpus versions, required entity ledger, candidate and final document IDs, evaluator input IDs, relevant hashes, per-stage rejection reason, raw status, and score. Name the first boundary where the entity or document disappears so the receiving team gets a falsifiable defect rather than a low decimal.

The costs are concrete. Evaluator calls add latency and provider spend. Capturing contexts increases storage and privacy obligations. Pinning models improves comparison but may retain an older evaluator. Human review slows ambiguous releases. A deterministic alias ledger is cheap to run but expensive to maintain. Choose the expense that matches the risk, and make it visible in the test plan.

When this metric is the wrong tool

Skip context entity recall when the answer has no meaningful entity requirement. Tone rewrites, creative summaries, generic explanations, and open-ended brainstorming often have no stable entity denominator. Forcing a reference full of nouns does not make the evaluation objective.

Do not use it as the sole check for numerical or relational correctness. A context can contain a drug name, dosage, patient, and date while connecting them incorrectly. A contract can mention both parties and the renewal date while reversing who may terminate. Test propositions, calculations, and constraints directly.

Avoid it when stable document IDs provide a stronger oracle. If the requirement is "retrieve policy revision P-104 and addendum A-7," ID-based recall is more direct and reproducible than asking a model to infer entities from prose. Entity recall is helpful when evidence is expressed across natural language and exact IDs are unavailable or insufficient.

It is also a poor release gate for unreviewed synthetic references. Synthetic cases can broaden coverage, but their entities may reflect generation quirks rather than user needs. Use them for exploration until a reviewer confirms the intent, reference, permission scope, and critical entities.

Finally, do not chase a perfect score by increasing top_k without limits. Returning the entire corpus would maximize the chance of entity coverage and destroy the purpose of retrieval. The trade-off is recall against noise, tokens, latency, and data exposure. Pair this metric with precision-oriented retrieval checks and response evaluation, then debug each lane on its own evidence.

A good context entity test ends with a specific statement: which approved reference entity was absent from which eligible retrieval, at what pipeline boundary, under which corpus and evaluator versions. Anything less is a score looking for a cause.