Hard-Negative Mining for Embedding Retrieval Evaluation

An embedding retriever can look excellent against random negatives and still confuse the documents users care about. Random negatives are often from unrelated topics and can be separated with broad semantic signals. Hard negatives expose the narrower decision boundary: two passages may discuss the same product and policy, but only one answers the query under the correct region, date, or account state.

The useful unit is not a vaguely difficult document. It is an adjudicated query-positive-negative relationship with provenance. Mining proposes candidates; a relevance process decides whether they are truly negative; a sealed benchmark measures ranking behavior without donating its cases back to model development.

Define relevance before difficulty

A hard negative must fail a written relevance rule. Similarity alone is insufficient. For a query about canceling an annual plan in India, a passage about monthly cancellation in the same product may be topically close but non-relevant. A global cancellation overview that explicitly covers India could be a valid alternative positive even if it was absent from the original label set.

Choose the evidence identity used for judgments. Exact chunks, canonical passages, documents, and facts support different conclusions. If two chunks overlap heavily and contain the same answer, one should not become a negative merely because its identifier differs. Store canonical source identity, span offsets, version, and duplicate group beside every mined relationship.

Use graded labels when partial utility matters. A passage can provide background without satisfying the information need. Preserve relevant, partially_relevant, non_relevant, and uncertain rather than forcing all boundary cases into a binary benchmark. The final metric can map grades according to a predeclared policy.

Split source families before mining

Partition queries and documents before nearest-neighbor search. Group by source document, entity, intent template, originating conversation, and generated seed. Random row splits allow nearly identical chunks or paraphrases to land in training and test, turning memorized structure into apparent generalization.

Mining for training should search only the training corpus partition or a clearly documented production-like pool. Development mining supports threshold and configuration choices. Test hard negatives remain sealed until the retriever, embedding preparation, and ranking protocol are frozen. Once a test negative is inspected to change the model or prompt, it belongs in development for the next benchmark version.

The LangChain retrieval documentation describes indexing as loading, splitting, embedding, and storing data before runtime retrieval. Record each of those configurations. A hard case may be caused by chunk construction or metadata filtering rather than embedding geometry, and an evaluation that stores only vector model identity cannot attribute the failure.

Mine candidates from diverse failure surfaces

Use several candidate generators because each reveals different confusion. Current embedding neighbors expose the model's own boundary. A lexical ranker finds shared names and wording. An older production retriever finds regressions. Metadata-adjacent sampling can pair documents that differ only by tenant, locale, product tier, or effective date. A cross-encoder or model judge can prioritize review, but it should not assign the final negative label without calibration.

Exclude only exact positive identities at the proposal stage. Do not exclude every item sharing a source document, because nearby sections can be useful hard negatives. Conversely, do not assume a different document is negative. Alternate policies and duplicated knowledge bases often support the same answer.

Keep the mining score and generator in provenance. Difficulty is relative to the system that proposed the candidate. A future retriever may find an old hard negative easy, which is useful evidence of progress rather than a reason to erase the case.

from dataclasses import dataclass

@dataclass(frozen=True)
class NegativeCandidate:
    query_id: str
    positive_id: str
    candidate_id: str
    miner: str
    miner_rank: int
    split: str

def propose_neighbors(query_id, positive_id, ranked_ids, split, limit=20):
    return [
        NegativeCandidate(query_id, positive_id, doc_id, "dense-dev", rank, split)
        for rank, doc_id in enumerate(ranked_ids, start=1)
        if doc_id != positive_id
    ][:limit]

This illustrative function produces a review queue, not labels. Production code should also remove canonical duplicates, enforce split membership, and retain corpus and miner versions.

Adjudicate false-negative risk

Reviewers need the query, proposed negative, known positive, relevant surrounding context, and applicability metadata. Hide the miner score and candidate system name. Ask first whether the candidate independently satisfies the query. If it does, add it to the positive set. If evidence is missing or the query permits multiple interpretations, mark it uncertain and route it to adjudication.

False negatives are especially likely when relevance labels are pooled from one retriever. Measure how many mined candidates become new positives, partial positives, true negatives, or abstentions. A high discovery count is a label-coverage finding, not proof that mining failed. Expand the qrels before scoring systems.

Write a short negative rationale using a controlled taxonomy: wrong entity, wrong time, wrong locale, adjacent procedure, unsupported implication, stale version, duplicate without answer span, or merely topical. That label turns a single rank error into an actionable slice while keeping free-text notes available for nuance.

Construct a benchmark with realistic context

Do not evaluate only one positive against one negative unless the product performs pairwise selection. A search system ranks against a corpus or candidate set, so preserve enough distractors to test that behavior. One practical design has a stable corpus snapshot, natural query distribution, known positives, adjudicated hard negatives, and ordinary background documents.

Balance is an evaluation design choice, not an estimate of production prevalence. If every query receives five hard negatives, an overall error percentage describes that constructed benchmark. Keep a separate traffic-weighted suite or production sample for expected impact. Report both without blending their denominators.

Maintain a stable sentinel panel and a renewable challenge panel. The sentinel set supports comparison across retriever versions. The challenge panel can incorporate newly observed confusions. Version additions, removals, relabeling, and source supersession so trend changes can be distinguished from dataset changes.

Select ranking metrics that expose the boundary

Recall at k measures whether any or all relevant evidence survives to the retrieval depth. Reciprocal rank emphasizes the first relevant result. NDCG can represent graded relevance and position discounting. Choose the primary metric according to how many contexts the downstream generator receives and whether evidence order affects truncation.

Always pair the aggregate with hard-negative slices. Report how often a wrong-date neighbor outranks every valid passage, how many queries lose all positives after filtering, and where near-duplicate boilerplate occupies the context window. A mean score can improve while a severe slice regresses.

Candidate coverage and judgment coverage belong beside rank metrics. If the expected positive is absent from the index, the case is an indexing failure. If returned candidates are unjudged, the metric has a label gap. Neither should be assigned to embedding discrimination without diagnosis.

Use paired uncertainty across queries when comparing retrievers. Resample source or conversation groups when cases are related. Small challenge slices may support only case review, not a stable numeric gate; mark that limitation rather than merging them into a large easy slice.

Separate training negatives from evaluation negatives

Training can use in-batch negatives, mined negatives, and periodically refreshed candidates, but every source needs false-negative controls. Evaluation negatives must not enter training, prompt examples, reranker tuning, or mining-threshold selection. Store benchmark IDs in a denylist checked by data pipelines.

Avoid circular selection. If candidate A mines the entire test set and only its highest-scoring mistakes are labeled, candidate B may be evaluated on A's peculiar boundary. Build test pools from multiple independent miners and assess a common candidate union. Report pool contribution so remaining asymmetry is visible.

Generated queries can amplify coverage, but split the source material before generation and preserve generator provenance. A query that repeats a unique sentence from its positive passage rewards lexical overlap. Include natural queries and rewrite generated prompts that leak answer strings or document headers.

Run ablations for causal diagnosis

Compare random negatives, lexical hard negatives, current-model neighbors, and metadata-confusable negatives as separate panels. This reveals whether a gain comes from broad topic separation or from the intended boundary. Do not average panels until their weights and decision meaning are declared.

Ablate one pipeline stage at a time on development data: chunking, text normalization, embedding input template, metadata filters, vector index settings, and optional reranking. Keep the corpus and qrels fixed where possible. If a stage changes candidate identities, complete missing judgments before crediting its score.

Inspect score margins as diagnostics, not universal pass thresholds. A small gap between positive and hard negative may be acceptable if both remain inside the generator's context budget; it may be critical if only one result is returned. Thresholds should be labeled illustrative until calibrated to the product decision.

Account for latency and operating cost

Hard-negative improvements may require deeper vector search, more filters, or a reranker. Record candidate depth, index state, concurrency, cache condition, payload size, and end-to-end latency distribution. Compare configurations under the same load protocol and timeout policy.

Quality after timeout is the operational metric. A configuration that ranks correctly only when allowed to exceed the service budget can reduce real retrieval quality through fallbacks or empty results. Report successful, timed-out, and degraded paths separately, then apply a release gate that combines ranking quality with the product's own latency objective.

Execute a hard-negative action plan

1. Define relevance grades, canonical evidence identity, and the product rank depth.
2. Split queries, source families, entities, and generated seeds before any mining or tuning.
3. Propose candidates with diverse miners and retain rank, score, corpus, and generator provenance.
4. Blindly adjudicate every candidate; promote relevant alternatives and quarantine uncertain cases.
5. Freeze a corpus-based benchmark with stable sentinels, realistic distractors, and explicit challenge slices.
6. Compare retrievers with paired ranking metrics, judgment coverage, latency, and design-aware uncertainty.
7. Mine training data only from permitted partitions and version the sealed test set when exposure occurs.

Hard negatives are valuable because they make the relevance boundary concrete. Their value disappears when unknown neighbors are mislabeled, split relationships leak, or one miner defines the whole test. Treat mining as candidate discovery and the benchmark as governed evidence, and the resulting failures will point to specific retrieval work rather than a generic embedding score.