How to Evaluate Embeddings: Search Quality Tests for QA Teams

A semantic search upgrade raised the offline similarity score, yet technicians could no longer find the “motor will not start when hot” bulletin with the query “fails after warmup.” The new embedding model clustered documents more neatly by product family, but weakened the symptom language that mattered to the support workflow. The team had optimized an abstract representation instead of the retrieval decision.

Evaluating embeddings means testing whether distances preserve the relationships your product needs. That requires labeled tasks, a fixed candidate-generation pipeline, slice analysis, and migration checks. A few appealing nearest-neighbor examples are not enough.

Define the relationship the vector must preserve

Start with the downstream decision. Embeddings may support document retrieval, duplicate detection, recommendations, clustering, classification, or anomaly detection. Each task needs different labels and metrics. An embedding that separates product categories well can still be poor at matching symptoms to fixes.

For technical support search, write the contract as: given a user query, rank at least one document that resolves the same fault within the first five results, while avoiding documents for incompatible product generations. This identifies both relevant and harmful neighbors.

Freeze the surrounding pipeline when comparing models. Use the same corpus snapshot, normalization, chunk boundaries, distance metric, candidate count, metadata filters, and reranker state. If the embedding model requires a different similarity function or input prefix, document that as part of the candidate configuration. Do not compare raw cosine values across unrelated embedding spaces as though their scales were universal.

Build labels from search judgments, not topic tags

Topic labels are a weak proxy for relevance. Two documents can share a category while solving different problems. Construct query-to-item judgments that reflect the user task.

{
  "queryId": "warm-failure-014",
  "query": "pump stops after twenty minutes",
  "relevance": {
    "bulletin-88": 3,
    "manual-thermal-shutdown": 2,
    "catalog-pump-2026": 0
  },
  "mustExclude": ["bulletin-legacy-voltage"],
  "metadata": {
    "productGeneration": "g4",
    "language": "en",
    "querySource": "resolved-ticket",
    "difficulty": "paraphrase"
  }
}

Use graded relevance when one result is the direct fix and another is only useful background. Add hard negatives that share vocabulary but violate an important condition, such as the wrong product generation or region. Include exact identifiers, paraphrases, abbreviations, misspellings, short queries, multilingual queries, and questions with insufficient information.

Have domain experts judge pooled results from more than one retrieval configuration. If labels come only from the current engine, they inherit its blind spots. Keep an “unjudged” state distinct from irrelevant. Treat disagreements as a signal that the query or acceptance rule needs clarification.

Measure ranking quality at the operating cutoff

Choose metrics that match the UI or downstream consumer. Recall@k asks whether the relevant set appears within the first k candidates. Precision@k measures how much of that shortlist is relevant. Mean reciprocal rank rewards placing the first relevant result early. Normalized discounted cumulative gain works well with graded relevance because it values both order and degree of usefulness.

For a support panel showing five items, report metrics at five, not only at 100. For a RAG pipeline that reranks 30 candidates, embedding recall@30 is important because the reranker cannot recover documents never retrieved. If there is exactly one accepted match, hit rate may be easier to explain than a broad average.

This Python example computes simple hit rate and reciprocal rank from already produced rankings. It is ordinary evaluation code, not an SDK contract:

def query_scores(ranked_ids, relevant_ids, k=5):
    top = ranked_ids[:k]
    ranks = [i + 1 for i, doc_id in enumerate(top) if doc_id in relevant_ids]
    return {
        "hit_at_k": int(bool(ranks)),
        "reciprocal_rank": 0.0 if not ranks else 1.0 / min(ranks),
    }

Report distributions and paired query outcomes, not just means. Count queries fixed by the candidate, queries regressed, and queries unchanged. A one-point average gain can conceal the loss of every exact part-number lookup.

Test geometry without mistaking it for product quality

Intrinsic tests help diagnose the embedding space. They should supplement, not replace, downstream retrieval evaluation. Test duplicate pairs against nonduplicate pairs, nearest-neighbor purity for stable labels, and whether known analogies or semantic pairs have sensible relative distances.

Inspect similarity distributions for positives, hard negatives, and random pairs. Large overlap means a single threshold will be unstable. Select duplicate-detection thresholds on a labeled validation set, then report false accepts and false rejects on a separate test set. Avoid tuning the cutoff on the same cases used for release approval.

Check vector properties deterministically: expected dimension, finite numeric values, nonzero norm, stable serialization, and repeatability for identical text when the provider and configuration claim deterministic output. Detect accidental truncation, empty-input vectors, or mixing vectors from different model versions in one index.

For clustering, use cluster metrics only when the labels represent the desired grouping. Human reviewers should inspect boundary examples and mixed clusters. A high silhouette score can simply mean the model separated language or formatting instead of intent.

Slice the evaluation by language and query behavior

Embedding regressions rarely distribute evenly. Slice results by language, query length, lexical overlap, entity density, product family, document age, and source. Create functional slices such as acronym expansion, symptom-to-cause, natural-language-to-code, and cross-lingual matching.

Track corpus properties too. Long chunks may dilute the key phrase; tables and code may embed differently from prose; boilerplate headers can make unrelated documents appear close. Compare performance by content type and chunk length bucket.

Use minimum slice sizes and show counts so stakeholders do not overreact to a single example. Still allow a critical slice to block release even when small. If exact safety procedure retrieval has ten labeled queries, losing two can matter more than a minor gain across hundreds of low-risk FAQ searches.

Model-graded relevance can accelerate labeling for low-risk cases, but it should receive the query and document text, use a task-specific rubric, and be calibrated against expert judgments. It should not evaluate embedding similarity directly. The question is whether the retrieved item helps the user, not whether two vectors “look close.”

Diagnose misses with neighbor evidence

For every regressed query, retain the baseline and candidate rankings, distances, metadata-filter decisions, raw text or stable document references, and vector model versions. Inspect where the relevant item moved and what replaced it.

Classify failures into useful buckets:

• Semantic miss: paraphrases that should match move apart.
• False attraction: shared boilerplate or vocabulary dominates meaning.
• Entity loss: identifiers, versions, or numeric distinctions disappear.
• Language imbalance: one language or script loses recall.
• Chunk dilution: the relevant passage is embedded with too much unrelated text.
• Pipeline mismatch: query and documents used inconsistent prefixes, normalization, dimensions, or models.
• Label defect: the expected item is outdated or not actually sufficient.

Run counterfactual checks before blaming the model. Remove boilerplate, shorten the chunk, add the missing metadata filter, or compare exact lexical retrieval. If keyword search finds a part number that semantic search misses, hybrid retrieval may be the right design rather than endless embedding tuning.

Evaluate drift and migration safety

Changing embedding models usually requires re-embedding the corpus. Treat it as a data migration. Verify document counts, unique IDs, dimensions, namespaces, metadata parity, and failed batch counts. Sample vectors from every shard and confirm the index contains only the intended model version.

Shadow the candidate index with real, privacy-reviewed queries. Compare overlap, paired relevance, latency, and empty-result rate without changing user results. Dual-read a bounded sample when production cost permits. Plan rollback around versioned indexes or aliases rather than overwriting the only copy.

Monitor after release for changes in query distribution and corpus composition. Track retrieval success proxies such as result clicks, reformulations, downstream answer abstentions, and support escalations, but do not treat them as perfect relevance labels. Sudden vector norm changes, index growth gaps, or slice-specific recall loss can reveal ingestion defects before users report them.

Make the release decision on quality, speed, and cost

Define the gate before running the candidate. An example policy could require no loss on critical exact-identifier and safety-procedure slices, a positive paired improvement on symptom paraphrases, unchanged empty-result rate, and no material regression in P95 embedding plus search latency.

Include migration and operating cost. A larger vector may increase storage and memory; a slower embedding service may delay ingestion; re-embedding millions of documents has a one-time price and rollback burden. Query quality gains must justify those consequences.

Require human review for new high-risk misses and a sample of large ranking changes. Document accepted tradeoffs, such as weaker clustering in exchange for better fault retrieval. Ship only when the candidate improves the relationship the product actually needs, the index migration is reversible, and observed regressions remain within explicit slice thresholds.