Paired Regression Tests for Embedding Model Migrations

An embedding migration changes the geometry used to find evidence. It can improve broad semantic matching while making exact identifiers, rare languages, short codes, or tenant-specific vocabulary harder to retrieve. A single average score cannot tell you whether the migration is acceptable.

Use a paired regression design. Build separate baseline and candidate indexes from the same immutable chunks, run each labeled query against both under identical retrieval policy, and decide from per-query and per-slice deltas. Keep the old read path available until production evidence supports the cutover.

Freeze the Comparison Contract

Create a manifest that identifies the corpus snapshot, chunker, normalization, metadata schema, query set, relevance labels, authorization policy, retriever settings, reranker, and metric implementation. The LangChain retrieval documentation describes embeddings, vector stores, and retrievers as separate building blocks. Preserve those boundaries so an embedding test does not quietly become a chunking or reranking experiment.

Every chunk needs a stable logical ID independent of vector-store row IDs. Hash normalized chunk text and record source document version. If either text or metadata differs between index builds, mark the item unpaired rather than attributing its retrieval effect to the embedding model.

Build Truly Separate Vector Spaces

Store baseline and candidate vectors in separate indexes or namespaces. Record model identity, vector dimension, distance function, normalization, index algorithm, quantization, build parameters, and search parameters. Never assume that two embeddings can be mixed because their dimensions match; equal shape does not establish compatible geometry.

Use the same logical chunk IDs and metadata in both spaces. Verify index cardinality, duplicate counts, missing vectors, metadata hashes, and tombstones before retrieval tests begin. A candidate index with fewer documents may look faster while failing recall for reasons unrelated to semantic quality.

Approximate nearest-neighbor settings can dominate results. Start with equivalent search-effort settings, then run a separate tuning study if the new space needs different parameters. Keep that second change explicit rather than retroactively calling it part of the embedding upgrade.

Construct a Paired Query Set

Cover natural-language questions, exact names, ticket IDs, error codes, abbreviations, spelling variation, multilingual requests, long descriptions, and very short queries. Include hard negatives from documents that use similar vocabulary but answer a different question. Preserve query families so paraphrases of one intent do not count as independent evidence.

For each query, label all known relevant chunks or apply an incomplete-judgment protocol that acknowledges unlabeled candidates. Include production retrieval failures and routine traffic. High-severity slices need protected representation even when they are rare.

Run both indexes for every query within the same experiment window. Randomize or interleave order to reduce systematic load effects, and record timeouts separately from empty results. If a query cannot be paired because one index lacks its authorized corpus, exclude it from the primary metric and report the coverage defect.

Compare Per-Query Deltas, Not Two Headlines

Calculate recall at the context consumption depth, reciprocal rank or rank position, precision, and no-result status for each side. Then derive the candidate-minus-baseline delta per query. Aggregate those paired deltas by slice, and retain the distribution of losses rather than only its mean.

from dataclasses import dataclass

@dataclass(frozen=True)
class Pair:
    query_id: str
    slice_name: str
    baseline_relevant: int
    candidate_relevant: int
    forbidden_candidate_hits: int

def paired_delta(pair: Pair) -> int:
    if pair.forbidden_candidate_hits:
        raise ValueError("authorization regression")
    return pair.candidate_relevant - pair.baseline_relevant

Inspect changed top results manually. A positive labeled-recall delta may come from near-duplicate chunks and add no new evidence. Conversely, a lower overlap with the baseline can be healthy when the candidate retrieves a more direct source. Rank metrics describe behavior; adjudication determines whether that behavior serves the task.

Preserve Authorization Semantics

Embedding migrations are also permission tests. Apply the same trusted tenant, principal, group, region, and document-state filters to both indexes. Validate the filter against source-of-truth metadata before comparing relevance, because a missing ACL field in one index can create apparent recall gains through forbidden documents.

Seed cross-tenant canaries and near-duplicate restricted documents. Query as a principal with no access and assert zero unauthorized chunk IDs at every retrieval stage, including pre-filter candidate pools if those pools are logged or passed to rerankers. Test revoked access, stale metadata, deleted documents, and policy version changes during dual-write operation.

Do not average security with retrieval quality. One forbidden hit is a separate release-blocking class according to the local risk policy, even if aggregate recall improves.

Separate Embedding Effects From Pipeline Effects

Run an ablation ladder. First change only document and query embeddings. Next tune candidate index search parameters. Then, if needed, retune hybrid weights or reranking. This order shows which component caused each movement and provides a fallback when a combined rollout regresses.

The LangSmith evaluation guide describes offline datasets, evaluators, experiments, and experiment comparison. Whatever evaluation runner you use, attach the complete migration manifest to every run and make baseline-candidate pairs query-addressable.

Include deterministic lexical controls for identifiers and exact phrases. A hybrid system may compensate for an embedding weakness, but that should be an intentional product design with its own cost and failure profile, not a hidden rescue that makes the embedding comparison impossible to interpret.

Exercise Ingestion and Cutover Failure Modes

Test dimension mismatches, provider errors, rate limits, partial batches, duplicate events, out-of-order updates, delete failures, and a crash after one index write. During backfill, verify resumability from a durable cursor and reconcile source documents against both index inventories.

New documents arriving during migration need dual writes. Record independent success states and retry safely. A read routed to the candidate index must not assume its vector exists merely because the baseline write succeeded. Define what happens when only one representation is available: route to baseline, enqueue repair, or return a controlled degraded result.

Keep the baseline index immutable except for the same production updates and preserve a replay log. Destructive replacement removes the ability to explain regressions and turns rollback into another migration.

Set Slice-Aware Migration Gates

Choose gates before examining candidate results. An illustrative policy might require zero cross-tenant hits, no confirmed loss on protected identifier queries, and a positive paired result on the target semantic slice with an uncertainty interval that stays above a locally tolerated boundary. These are example rules, not universal thresholds.

Report counts and intervals for each slice, plus build cost, index size, query latency distribution, and provider failures. Cost or latency gains should not erase a severe relevance loss. If tradeoffs are acceptable, record who accepted them and which monitor will detect them after cutover.

Use shadow reads before user-visible routing. Compare candidate and baseline results on sampled authorized traffic without exposing candidate context to generation. Then canary a small cohort with an immediate version flag rollback. Avoid a one-way alias swap until recovery has been rehearsed.

Migration Checklist

• Version the corpus, chunks, labels, queries, filters, retriever, and metrics.
• Give chunks stable logical IDs and verify text plus metadata hashes.
• Build baseline and candidate vectors in isolated spaces.
• Reconcile counts, tombstones, duplicate rows, and missing vectors.
• Pair every query under identical retrieval and authorization settings.
• Inspect per-query losses and protected slices, not only averages.
• Test exact identifiers, languages, hard negatives, and no-result cases.
• Fault-inject backfill, dual-write, delete, and routing paths.
• Shadow production reads before a bounded canary.
• Preserve the old index and rehearse rollback.

Action Plan: Move From Benchmark to Reversible Cutover

Freeze one representative corpus and query snapshot, build the candidate index beside the baseline, and verify structural parity before scoring anything. Run the paired experiment, adjudicate the largest rank changes, and resolve authorization or ingestion defects before tuning search parameters.

After the offline gate passes, backfill through a resumable pipeline, enable monitored dual writes, and shadow real queries. Canary reads behind an explicit index-version flag while both paths remain healthy. Promote only after the rollback drill succeeds and slice monitors confirm that production traffic matches the risks represented in the paired benchmark.