Build retrieval tests that expose acronym and synonym blind spots

Search for “RCA” in an engineering assistant and it finds the incident-review template. Ask for “root cause analysis,” and the same assistant surfaces a manufacturing glossary. The answer generator is not the first suspect. Retrieval has split two labels for one concept, or joined two concepts that happen to share a label.

That distinction matters in QA. A single expected answer cannot tell you whether the query was normalized incorrectly, the right document never entered the candidate set, a reranker moved it down, or the dataset described the acronym too broadly. A useful retrieval suite records enough evidence to make those failures different on purpose.

Model the concept before you collect queries

Most weak datasets start with a spreadsheet column called query and another called expected_answer. That shape is convenient for a demo, but it hides the part an evaluator must control. Acronyms and synonyms are relationships between labels and a concept. They are not universal string substitutions.

Consider “RCA.” An incident responder may mean root cause analysis. An electronics engineer may mean an RCA connector. A finance team may use the same letters for a ratio or an internal process. Expanding every occurrence to one phrase would make the first query easier and the others wrong. The test row therefore needs a domain, collection, tenant, locale, or enough words around the acronym to establish the intended sense.

A practical concept record has one stable concept ID and several reviewed labels. The labels should have roles, because an acronym exercises a different risk from a spelling variant:

• A preferred label is the wording your documentation uses most often.
• An acronym or initialism compresses the preferred label, such as “RCA” for “root cause analysis.”
• An expansion spells out an abbreviation found in queries or source documents.
• An operational paraphrase describes the task, such as “template for finding why an outage happened.”
• A product alias is language specific to your organization, not a dictionary synonym.
• A wrong-sense label is a deliberate distractor, such as “RCA cable” in an incident-management collection.

The W3C SKOS reference is useful as a vocabulary model even if your system does not store RDF. It distinguishes preferred, alternative, and hidden lexical labels, and it allows language tags. The testing lesson is simple: keep label identity separate from concept identity. Do not treat a flat bag of strings as proof that every pair is interchangeable.

Query normalization is another layer, not a substitute for that model. Case folding, whitespace handling, punctuation removal, stemming, and Unicode normalization can all change the text before retrieval. The exact operations depend on your index and query pipeline. Capture what your application sends to the retriever rather than assuming the UI text reaches it unchanged.

Unicode is a good example because identical-looking text can have different code-point sequences. JavaScript's String.prototype.normalize() returns a string in the requested Unicode normalization form. NFC combines canonically equivalent sequences into a composed form where one exists. NFKC also performs compatibility normalization, which can erase distinctions your product may care about. Use the form named by the product contract, and test it at the same boundary where indexing and querying normalize text.

The following diagnostic is intentionally small. It does not claim that NFC fixes retrieval. It tells you whether two labels that look alike arrive as the same sequence after the normalization your team chose.

type Probe = {
  label: string;
  value: string;
};

const probes: Probe[] = [
  { label: "composed", value: "café" },
  { label: "decomposed", value: "cafe\u0301" },
  { label: "full-width acronym", value: "ＲＣＡ" },
  { label: "ASCII acronym", value: "RCA" },
];

function codePoints(value: string): string {
  return Array.from(value)
    .map((character) => `U+${character.codePointAt(0)!.toString(16).toUpperCase().padStart(4, "0")}`)
    .join(" ");
}

for (const probe of probes) {
  const nfc = probe.value.normalize("NFC");
  const nfkc = probe.value.normalize("NFKC");
  console.log(JSON.stringify({
    label: probe.label,
    raw: codePoints(probe.value),
    nfc: codePoints(nfc),
    nfkc: codePoints(nfkc),
  }));
}

Run that probe against values copied from the request trace and the indexed source, not values retyped by hand. Retyping can silently produce a different character sequence. If NFC makes the accented forms equal but retrieval still differs, move on to tokenization, filters, candidate generation, and ranking. If only NFKC makes the full-width and ASCII forms equal, adopting NFKC is a product decision with a compatibility cost, not a test-side cleanup.

Keep the answer model out of this first investigation. A language model can answer from prior knowledge even when retrieval returns nothing useful. It can also write a poor answer after receiving the correct chunks. Store ordered document or chunk IDs before generation so retrieval quality remains observable.

Make every row prove one retrieval promise

A case should be reviewable without opening the implementation. The reviewer needs to know which concept the query targets, why the wording belongs to that concept, which corpus version was searched, what filters applied, which documents count as relevant, and which documents would demonstrate a wrong sense.

Stable document IDs are better oracles than expected prose. Prose changes when prompts and models change. A document ID says whether the evidence entered the context window. If your chunk IDs change every time content is rebuilt, retain a stable source-document ID alongside the chunk ID. Otherwise a harmless rechunk can make the entire gold set look broken.

Use an ordered result list, not an unordered set, for observed output. The order lets you distinguish a candidate-generation miss from a ranking movement. A relevant document at rank six is absent when the application only sends five chunks to generation. The same document at rank one satisfies a very different user experience even though both runs contain it somewhere in a large export.

Recall at k is straightforward when the fixture has a reviewed set of relevant document IDs:

recall@k = relevant document IDs found in the first k results / all relevant document IDs in the fixture

The denominator is a judgment, not a fact discovered by the metric. If reviewers know only one acceptable document, a perfect score means the system found that one document, not that the corpus contains no other relevant material. Record who reviewed the set and which corpus snapshot they saw.

Exact rank assertions are usually too brittle for a general relevance set. They make a harmless swap between two equally useful documents fail the build. Reserve an exact-rank rule for a product promise that truly depends on position, such as a safety procedure that must be the first result. For most cases, use a top-k window, a minimum recall policy, and explicit forbidden IDs for known wrong senses.

The fixture below is illustrative. Its document IDs and thresholds are contract examples, not measurements from a production system. Each row represents a different search intent, even when two rows refer to the same concept.

# retrieval_cases.py
CASES = [
    {
        "case_id": "incident-rca-acronym",
        "concept_id": "incident-root-cause-analysis",
        "query": "RCA template after an API outage",
        "variant": "acronym_with_context",
        "locale": "en",
        "corpus_version": "support-corpus-fixture-v3",
        "filters": {"collection": "engineering-operations"},
        "top_k": 5,
        "expected_document_ids": [
            "incident-analysis-runbook",
            "post-incident-review-template",
        ],
        "forbidden_document_ids": ["analog-rca-connector-guide"],
        "minimum_recall_at_k": 0.5,
    },
    {
        "case_id": "incident-root-cause-expanded",
        "concept_id": "incident-root-cause-analysis",
        "query": "root cause analysis template after an API outage",
        "variant": "expanded_label_with_context",
        "locale": "en",
        "corpus_version": "support-corpus-fixture-v3",
        "filters": {"collection": "engineering-operations"},
        "top_k": 5,
        "expected_document_ids": [
            "incident-analysis-runbook",
            "post-incident-review-template",
        ],
        "forbidden_document_ids": ["analog-rca-connector-guide"],
        "minimum_recall_at_k": 0.5,
    },
    {
        "case_id": "login-sso-expanded",
        "concept_id": "single-sign-on",
        "query": "troubleshoot single sign-on redirect loop",
        "variant": "expanded_label",
        "locale": "en",
        "corpus_version": "support-corpus-fixture-v3",
        "filters": {"collection": "identity"},
        "top_k": 5,
        "expected_document_ids": ["sso-redirect-troubleshooting"],
        "forbidden_document_ids": ["oauth-device-code-setup"],
        "minimum_recall_at_k": 1.0,
    },
]

Those rows avoid a common trap: generating ten paraphrases from one sentence and calling the result coverage. Variants should cross a meaningful boundary. An acronym may fail lexical matching. An expansion may be missing from source metadata. A product alias may work only in one tenant. A typo may exercise spelling tolerance. If two rows would enter the system as the same normalized query and carry the same intent, one may be enough unless you are testing the normalization itself.

Negative evidence deserves its own fields. A forbidden document is not every irrelevant result. It is a reviewed result whose presence in the application window would expose a specific failure, such as the wrong expansion of an ambiguous acronym. Avoid a blanket assertion that no irrelevant document may appear anywhere in the top k. Relevance is graded, and candidate lists often contain marginal items. Name only the wrong results that matter to the user or the downstream answer.

Dataset integrity checks should run before any retrieval call. Reject duplicate case IDs, empty expected sets, document IDs present in both expected and forbidden sets, nonpositive k values, and thresholds outside zero through one. An invalid fixture is an evaluation error. Calling it a product failure teaches engineers to ignore the gate.

The next script evaluates captured rankings without depending on a particular vector database or search API. A suite-owned adapter writes a JSON object keyed by case ID, and the gate compares those ordered IDs with the reviewed contract. Because the evaluator operates on IDs, teams can replace the adapter without rewriting the oracle.

# retrieval_gate.py
from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable

from retrieval_cases import CASES


@dataclass(frozen=True)
class CaseResult:
    case_id: str
    recall_at_k: float
    missing_ids: tuple[str, ...]
    forbidden_ids: tuple[str, ...]
    passed: bool


def validate_case(case: dict) -> None:
    expected = set(case["expected_document_ids"])
    forbidden = set(case["forbidden_document_ids"])
    if not case["case_id"] or not expected:
        raise ValueError("case_id and expected_document_ids are required")
    if expected & forbidden:
        raise ValueError(f"{case['case_id']}: expected and forbidden IDs overlap")
    if case["top_k"] <= 0:
        raise ValueError(f"{case['case_id']}: top_k must be positive")
    if not 0.0 <= case["minimum_recall_at_k"] <= 1.0:
        raise ValueError(f"{case['case_id']}: minimum recall must be between 0 and 1")


def evaluate(case: dict, ranked_ids: Iterable[str]) -> CaseResult:
    validate_case(case)
    observed = tuple(ranked_ids)[: case["top_k"]]
    expected = tuple(case["expected_document_ids"])
    forbidden = tuple(
        document_id
        for document_id in case["forbidden_document_ids"]
        if document_id in observed
    )
    missing = tuple(document_id for document_id in expected if document_id not in observed)
    recall = (len(expected) - len(missing)) / len(expected)
    passed = recall >= case["minimum_recall_at_k"] and not forbidden
    return CaseResult(case["case_id"], recall, missing, forbidden, passed)


def main(results_path: str) -> int:
    rankings = json.loads(Path(results_path).read_text(encoding="utf-8"))
    failed = False
    for case in CASES:
        if case["case_id"] not in rankings:
            print(f"ERROR {case['case_id']} missing ranking artifact")
            failed = True
            continue
        result = evaluate(case, rankings[case["case_id"]])
        status = "PASS" if result.passed else "FAIL"
        print(
            f"{status} {result.case_id} "
            f"recall@{case['top_k']}={result.recall_at_k:.2f} "
            f"missing={list(result.missing_ids)} "
            f"forbidden={list(result.forbidden_ids)}"
        )
        failed = failed or not result.passed
    return int(failed)


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("usage: python retrieval_gate.py <rankings.json>")
    raise SystemExit(main(sys.argv[1]))

Do not average all rows before deciding whether to fail. A hundred easy expansions can hide one severe wrong-sense result. Keep a case-level outcome, then apply a release policy based on risk tags. A high-risk incident command may block on one miss. A newly collected product alias may remain report-only until its label has been reviewed.

Diagnose the failure at the right layer

The most useful artifact is a per-query record assembled at the retrieval boundary. It should include the raw query, the normalized query if the application creates one, corpus or index version, effective filters, requested k, ordered returned IDs, and any stable stage output the retriever actually exposes. Do not invent a candidate-stage field for a service that only returns final results. Capture less, but keep every captured field truthful.

Read the evidence in this order:

1. Validate the fixture. Confirm the concept, expected IDs, forbidden IDs, filters, and corpus version still exist together.
2. Compare the raw UI input with the retriever request. A gateway may remove punctuation, expand a template, translate text, or attach tenant filters.
3. Check corpus membership. A document absent from the tested snapshot cannot be retrieved, even when the live environment contains it.
4. Inspect the earliest ranked list your implementation exposes. If the relevant ID never appears, investigate indexing, filtering, and candidate generation. If it appears early and later disappears, investigate reranking or post-retrieval policy.
5. Verify the application window. A retriever may return more results than the answer pipeline keeps. Test the IDs that can actually reach the next component.
6. Look at answer generation only after the retrieval contract is understood. Correct evidence with a bad answer belongs to another suite.

Scores can help within one recorded pipeline, but resist treating them as portable truth. Different retrieval methods produce values on different scales. Even the same engine may change score distributions after reindexing or configuration changes. IDs, ranks, filters, and versions are safer primary evidence. Preserve scores as diagnostics when the tool returns them, not as a cross-system unit of relevance.

Pytest parameterization gives each case its own test identity, which is exactly what this suite needs. The fixture below reads an artifact generated by the application's adapter. It fails separately when the artifact is absent, when relevant evidence is missing, and when a known wrong-sense document appears.

# test_retrieval_contract.py
import json
import os
from pathlib import Path

import pytest

from retrieval_cases import CASES
from retrieval_gate import evaluate, validate_case


@pytest.fixture(scope="session")
def captured_rankings() -> dict[str, list[str]]:
    path = Path(os.environ.get("RETRIEVAL_RESULTS", "artifacts/retrieval-rankings.json"))
    if not path.is_file():
        raise FileNotFoundError(f"retrieval artifact does not exist: {path}")
    return json.loads(path.read_text(encoding="utf-8"))


def test_case_ids_are_unique() -> None:
    case_ids = [case["case_id"] for case in CASES]
    assert len(case_ids) == len(set(case_ids)), "duplicate retrieval case_id"


@pytest.mark.parametrize("case", CASES, ids=lambda case: case["case_id"])
def test_dataset_rows_are_valid(case: dict) -> None:
    validate_case(case)


@pytest.mark.parametrize("case", CASES, ids=lambda case: case["case_id"])
def test_retrieval_contract(case: dict, captured_rankings: dict[str, list[str]]) -> None:
    case_id = case["case_id"]
    assert case_id in captured_rankings, f"no ranking captured for {case_id}"
    result = evaluate(case, captured_rankings[case_id])
    assert not result.forbidden_ids, (
        f"{case_id} returned wrong-sense documents: {list(result.forbidden_ids)}"
    )
    assert result.recall_at_k >= case["minimum_recall_at_k"], (
        f"{case_id} recall@{case['top_k']}={result.recall_at_k:.2f}; "
        f"required={case['minimum_recall_at_k']:.2f}; "
        f"missing={list(result.missing_ids)}"
    )

The ids function matters during triage. A failure named incident-rca-acronym points to a reviewed intent. A failure named case[17] sends someone back to the dataset to discover what changed. Keep IDs semantic and stable even when query wording is edited.

Suppose the captured acronym ranking contains only the connector guide and unrelated pages, while the expanded query still finds both incident documents. The transcript below is illustrative output for that deliberately broken artifact. It is not a production benchmark result.

$ RETRIEVAL_RESULTS=artifacts/rankings-with-rca-collision.json \
    python -m pytest -q test_retrieval_contract.py::test_retrieval_contract \
    -k "incident and rca"
F                                                                        [100%]
=================================== FAILURES ===================================
________________ test_retrieval_contract[incident-rca-acronym] _________________
E   AssertionError: incident-rca-acronym returned wrong-sense documents:
E   ['analog-rca-connector-guide']
=========================== short test summary info ============================
FAILED test_retrieval_contract.py::test_retrieval_contract[incident-rca-acronym]
1 failed, 2 deselected

That message establishes a collision, but it does not yet identify the component that caused it. Compare the normalized request and filters for the acronym and expansion rows. If both use the engineering collection and only the acronym brings back the connector guide, the label or ranking path is suspect. If the acronym request lost its collection filter, the retrieval model may be innocent. If the connector guide was accidentally tagged as engineering operations, the metadata is wrong.

Several near-misses produce the same high-level symptom, “expected document not found”:

• A stale gold ID fails every variant after a document migration. The current corpus manifest proves the expected ID no longer exists.
• A top-k boundary change moves a relevant item from rank five to rank six. A larger diagnostic list proves it was generated but did not fit the application window.
• An access-control filter excludes the document for the test principal. The same query with a different authorized fixture may pass, but that is evidence of a permission difference, not permission to bypass it.
• A locale filter selects English while the gold document is tagged only for another locale. The concept mapping can be correct while eligibility is wrong.
• An adapter fails before writing results. A missing artifact must not become an empty ranking, because empty output falsely accuses retrieval.

Keep these outcomes separate in reports. FAIL means a valid contract was violated. ERROR means the contract could not be evaluated. REVIEW means the relevance judgment or intended sense is disputed. Converting all three to zero recall makes dashboards tidy and investigations slow.

Work through three failures that look alike

Real incidents rarely arrive with a label saying “synonym bug.” They arrive as a user complaint, a rank drop, or an answer with the wrong source. Worked cases make the diagnostic boundary concrete.

The acronym disappears before candidate generation

An incident assistant retrieves the post-incident template for “root cause analysis after checkout outage” but not for “RCA after checkout outage.” The corpus snapshot, tenant, locale, and collection filter match. The request trace shows that RCA reaches retrieval unchanged. A diagnostic run with a larger result window still contains no incident-analysis document.

That evidence rules out the answer generator and makes a simple top-k displacement unlikely. It does not prove whether the defect sits in tokenization, index labels, sparse matching, embedding behavior, or a query-rewrite stage. The next useful comparison is the earliest stage your system exposes. If an alias-aware query representation includes the expanded phrase but candidates remain wrong, inspect indexing and document labels. If the representation never connects the acronym to the concept, inspect the curated alias or rewrite path.

One possible fix is adding a reviewed alternate label to the concept metadata used during indexing. Another is expanding the query within the engineering collection. A hybrid retriever may use both lexical and semantic signals. Choose based on the actual architecture; the dataset should state the promise without dictating the implementation.

The cost is precision and maintenance. Adding “RCA” globally can promote connector documentation or another department's material. Scoping the alias by collection reduces that collision but creates configuration that must follow content moves. Query expansion also adds terms, which can change ranking for requests that were already good. Keep the wrong-sense fixture beside the positive case so the fix cannot quietly swap one failure for another.

The acronym is genuinely ambiguous

A second team adds a row with the bare query “RCA.” They expect the incident-analysis runbook because that is what their group usually means. CI returns the connector guide first and fails the new case. At first glance this looks identical to the missing-alias defect.

The evidence says otherwise. The query has no domain phrase. Both documents are present and eligible in a shared technical collection. Search logs show that users issue the letters while working in multiple product areas. Reviewers from each area assign a different intended concept. There is no single defensible gold document for the unqualified input.

Do not repair the test by selecting the most popular expansion and declaring every other sense irrelevant. Options include adding conversational context, applying an existing workspace or collection filter, asking the user to disambiguate, or presenting results from several senses. The right choice belongs to product behavior. The QA fix is to mark the bare acronym as ambiguous and create contextual rows such as “RCA template after an outage” and “RCA cable input troubleshooting.”

Disambiguation costs interaction time or interface space. Contextual filtering can also hide a useful cross-domain result. Asking a clarifying question improves precision but delays the answer. A test suite should expose those costs, not encode one department's vocabulary as a universal language rule.

The gold document left the corpus

A synonym row for “incident postmortem checklist” starts failing after a content migration. The acronym and preferred-label rows also miss the same expected ID. Engineers are tempted to tune retrieval because the release gate is red.

The corpus manifest provides the decisive clue: post-incident-review-template was retired and its content was merged into incident-analysis-runbook. Captured results contain the surviving runbook near the top. The old ID cannot appear under the tested corpus version. This is gold-set drift, not evidence that synonym handling regressed.

Review the replacement document rather than replacing the ID mechanically. The merged page may omit steps that made the retired template relevant. If it still satisfies the query, version the dataset, update the reviewed relevant set, and retain the old fixture with its old corpus snapshot when reproducibility matters. If the replacement is incomplete, the failure belongs to content coverage even though retrieval found the only available page.

Updating gold data costs reviewer time and breaks direct comparisons across dataset versions. Keeping the obsolete ID preserves history but makes the current gate impossible to pass. Versioned corpus and dataset identifiers let the report say which contract ran, so a gold update does not masquerade as a model improvement.

These cases can all print missing=['post-incident-review-template']. The separating evidence is different: no candidate under a valid alias, multiple valid senses for an underspecified query, or an expected ID absent from the corpus. A score alone cannot make that distinction.

Roll the suite into CI without freezing bad labels

Dropping a large, unreviewed synonym list into a blocking job creates noise on day one. Teams then add skips until the gate carries no authority. Rollout should make the dataset more trustworthy before it makes the pipeline stricter.

Begin with concepts tied to user harm or repeated support failures. Production queries can suggest candidate language, but scrub sensitive text and have a subject-matter reviewer confirm the intended sense. Search-frequency data tells you what people typed. It does not prove which document should satisfy them.

Next, freeze a small corpus snapshot or an immutable index reference for deterministic regression runs. Record access rules and filters with it. A frozen corpus makes code and configuration changes comparable, but it will not detect fresh-content failures. Keep a separate scheduled evaluation against a current staging index, and label its results as drift monitoring rather than mixing them into the deterministic pull-request gate.

Capture a baseline before setting blockers. Investigate surprising passes as carefully as failures. A model may retrieve the expected ID because every document repeats the acronym, while a realistic corpus would not. A passing test with a leaked query phrase in metadata validates the fixture construction, not the product behavior users receive.

Put dataset integrity and adapter health in CI first. Add report-only retrieval cases next. Promote a row to blocking only after its concept, scope, gold documents, and failure owner have been reviewed. High-risk rows may use an absolute contract. Broader language-coverage rows may use a reviewed regression budget. Derive any budget from product tolerance and observed baseline variation; do not copy an attractive percentage from an example article.

Keep the result artifact when a job fails. It should contain the case ID, dataset version, corpus version, request text or a safe reference to it, filters, ordered IDs, and the evaluator decision. Redact user content before long-term storage. Hashing a sensitive query is useful for joining records, but it does not let an investigator understand meaning, so preserve a sanitized fixture text for approved test cases.

This workflow is an illustrative repository layout. The action names and keys are standard GitHub Actions syntax; the Python paths are suite-owned files that must exist in your repository. Dependency installation uses a committed, hash-locked requirements file so CI does not choose arbitrary package versions.

name: retrieval-contract

on:
  pull_request:
    paths:
      - "retrieval/**"
      - "tests/retrieval/**"
      - "requirements-ci.txt"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    env:
      RETRIEVAL_RESULTS: artifacts/retrieval-rankings.json
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - name: Install locked test dependencies
        run: python -m pip install --require-hashes -r requirements-ci.txt
      - name: Validate the dataset
        run: python -m pytest -q tests/retrieval/test_dataset_integrity.py
      - name: Capture rankings from the frozen corpus
        run: python retrieval/capture_rankings.py --output "$RETRIEVAL_RESULTS"
      - name: Enforce reviewed retrieval contracts
        run: python -m pytest -q tests/retrieval/test_retrieval_contract.py

The adapter behind capture_rankings.py is the one piece that must understand your application. Keep it thin. It should submit the exact query and filters in the fixture, record the IDs in returned order, and expose failures as errors. It should not add synonyms, reorder results, or replace missing IDs, because those transformations would test the adapter's preferred answer instead of the product.

When retrieval is nondeterministic, repeatability needs an explicit policy. First remove avoidable variation by pinning the corpus, index build, configuration, and service version where your platform permits it. If rankings can still vary, retain each run rather than averaging away a dangerous result. A must-not-return document appearing once may matter more than a mean recall value. Any retry rule should be written before the run and should report first-attempt failures, or retries will turn instability into a silent pass.

Migrate existing suites concept by concept. Map old keyword rows to stable concepts. Label each variant's role. Remove duplicates after normalization. Re-review expected IDs against the frozen corpus. Add wrong-sense cases for risky aliases. Run old and new gates side by side for a limited rollout, then retire the old gate only when differences have owners. Do not rewrite historical result files to fit the new schema; preserve the dataset version that produced them.

Know the cost and when to stop

Concept-based fixtures are not free. Each label needs provenance and scope. Each expected document set ages as content changes. Every corpus version consumes storage or index capacity. Someone must adjudicate disputes between domain experts. If no owner accepts that work, thousands of generated synonyms will decay faster than a small set of reviewed cases.

Broader variant coverage also consumes retrieval calls and CI time. Run the smallest blocking set on every relevant change. Put long-tail locales, rare aliases, and live-index drift checks on a schedule or on retrieval-specific changes. Sharding can shorten elapsed time, but it makes artifact collection and failure triage more complex. Measure your own suite before choosing a split.

Strict gold sets can discourage legitimate improvements. A new, better document may outrank an old expected page and fail an exact-order assertion. That is why relevant sets, forbidden IDs, and corpus versions work better than one sacred result. Human review still has to decide whether the new document is actually better.

Alias fixes can raise recall while lowering precision. They can also reinforce organizational jargon that new users do not know. Include natural task phrases alongside internal abbreviations, and track locale rather than assuming an English label maps cleanly into every language. Transliteration, inflection, and regional terminology deserve reviewed cases, not automated string swaps labeled as ground truth.

There are also clear situations where this suite is the wrong tool:

• Do not add acronym variants when the product contract requires an exact identifier. Expanding a database key, ticket number, or command token may corrupt the request.
• Do not force a synonym expectation for a polysemous word without context. “Postmortem,” “retro,” and “incident review” overlap in some teams but can name different processes in others.
• Do not blame retrieval when the required document is already high in the application window. Test prompt use, citation selection, and answer grounding separately.
• Do not gate a mutable production index with IDs from an unversioned snapshot. Content drift will dominate the signal.
• Do not use a frozen corpus to claim live relevance. It proves regression behavior against that snapshot only.
• Do not retain raw customer queries merely because they improve realism. Build approved, sanitized fixtures or apply the privacy controls your organization requires.
• Do not expand every acronym at ingestion and query time without testing collisions. Duplicate terms can change ranking and storage while hiding which label carried the match.

Stop adding variants when a new row has no distinct risk, scope, or expected behavior. “RCA guide,” “guide for RCA,” and “RCA documentation” may be three strings but one retrieval obligation. Spend the next review on a wrong-sense collision, a locale boundary, a permission filter, or a corpus migration. Those are the rows that explain a failure an engineer can fix.

One final boundary is worth preserving: a retrieval dataset is a contract for evidence selection, not a thesaurus and not a benchmark of general intelligence. If the application has no promise to understand a bare abbreviation, record that product decision. If it does make the promise, give the case enough context, stable evidence, and a named owner to make a red build actionable.