Text similarity is not the same thing as a correct answer

The reference says the customer can return an unopened item within thirty days. The candidate says the same thing in plain language, but an overlap-based gate rejects it. Another candidate copies most of the reference, changes "can" to "cannot," and looks deceptively similar.

Both failures come from asking one metric to make a decision it was never designed to make. Text similarity compares a candidate with a reference. It does not know which token carries the policy, whether a cited document supports the answer, or whether an added sentence invents a fact. A high score can be wrong, and a low score can be acceptable.

OpenAI's legacy text_similarity grader takes an input, a reference, a pass threshold, and an evaluation metric. The documented choices are fuzzy_match, BLEU, GLEU, METEOR, cosine similarity, and ROUGE variants. OpenAI states that fuzzy matching uses RapidFuzz and cosine uses text-embedding-3-large; cosine is documented as available only for evals. Grader templates can read {{ sample.output_text }} and fields such as {{ item.reference_answer }}.

Those APIs now sit on a published migration path. OpenAI deprecated the Evals platform in June 2026. Existing evals are scheduled to become read-only on October 31, 2026, and the Evals dashboard and API are scheduled to shut down on November 30, 2026. Maintain existing tests while you need them, but keep references, labels, metric intent, thresholds, and row-level results portable. The enduring engineering problem is choosing the right oracle, not preserving one hosted configuration.

Pick the failure you want similarity to catch

Begin with the release decision, then ask whether distance from a reference represents that decision. Similarity fits when the product accepts variation but expects content to remain close to one or more trusted examples. It is much weaker when a tiny difference can reverse meaning or when a good answer may introduce entirely different but valid evidence.

For extractive summaries, lexical overlap may be a useful signal because important phrases should remain tied to the source. For open-ended support replies, accepted wording may vary so widely that a semantic comparison is more practical. For a required legal notice, neither is the first choice: an exact string or structured requirement can test the obligation directly. The same candidate text can therefore be suitable for one metric and dangerous for another.

Do not choose a metric because its name sounds semantic or familiar. Create a small contrast set first. Include a valid paraphrase, a copied answer with one reversed fact, a shorter answer that preserves the decision, a longer answer with an unsupported addition, and an unrelated answer that shares domain vocabulary. Run candidate metrics against those examples and inspect the ordering. The metric should rank cases in the same direction as the product decision before anyone tunes a threshold.

The following fixture generator creates three contrast families. The expected values are human decisions, not claimed grader scores. Each family isolates a distinct weakness: negation, irrelevant keyword overlap, and added unsupported content.

import json
from pathlib import Path

cases = [
    {
        "case_id": "returns-paraphrase",
        "family": "policy_negation",
        "reference": "An unopened item can be returned within 30 days of delivery.",
        "candidate": "You may send the item back during the first 30 days if it is unopened.",
        "human_pass": True,
        "reason": "Preserves eligibility, condition, and time window."
    },
    {
        "case_id": "returns-negated-copy",
        "family": "policy_negation",
        "reference": "An unopened item can be returned within 30 days of delivery.",
        "candidate": "An unopened item cannot be returned within 30 days of delivery.",
        "human_pass": False,
        "reason": "One word reverses the policy."
    },
    {
        "case_id": "battery-keyword-decoy",
        "family": "domain_overlap",
        "reference": "Stop using a swollen battery and contact the safety team.",
        "candidate": "Battery life depends on screen brightness and background apps.",
        "human_pass": False,
        "reason": "Shares the topic but omits the safety action."
    },
    {
        "case_id": "incident-supported-summary",
        "family": "unsupported_addition",
        "reference": "The checkout service timed out after the inventory call.",
        "candidate": "Inventory completed, then checkout timed out.",
        "human_pass": True,
        "reason": "Restates the observed sequence without adding a cause."
    },
    {
        "case_id": "incident-invented-cause",
        "family": "unsupported_addition",
        "reference": "The checkout service timed out after the inventory call.",
        "candidate": "A database lock caused checkout to time out after inventory completed.",
        "human_pass": False,
        "reason": "Adds a cause that is absent from the evidence."
    }
]

output = Path("similarity-contrasts.jsonl")
output.write_text("".join(json.dumps(case) + "\n" for case in cases), encoding="utf-8")
print(f"wrote {len(cases)} contrast cases to {output}")

One reference is rarely enough for open-ended language. If three distinct responses are all acceptable, storing only the most formal one trains the evaluation toward style. Add independently authored accepted variants or describe required semantic units separately. Do not generate dozens of paraphrases from the reference and call that diversity. They share one origin and may repeat its blind spots.

Reference quality is part of the oracle. A reference that omits a required caveat can punish a candidate for being more complete. A reference copied from an old policy can make the correct current answer look dissimilar. Give references owners and versions, and link policy-derived cases to a review date. Otherwise threshold tuning slowly compensates for stale truth.

Calibrate the threshold on labeled boundaries

A threshold is a classification policy applied to a continuous score. It should separate accepted from rejected outputs at a risk level the team understands. It is not a property handed down by the metric, and a value that works for summaries may be useless for short labels or multilingual answers.

Collect candidate outputs that resemble the traffic you will evaluate. Have qualified reviewers label them without seeing similarity scores. Include normal cases, hard boundaries, and adversarial cases. Preserve disagreements and adjudication notes. If reviewers cannot agree on what passes, a numeric cutoff cannot make the underlying requirement coherent.

Split calibration from final evaluation. Use the calibration set to choose the metric, normalization, and threshold. Hold back a set to verify that those choices generalize. Keep related prompt families together when splitting; near-duplicate siblings on both sides will make performance look better than it is. If you later tune the candidate system against the held-out set, it is no longer held out.

Inspect a threshold sweep rather than reporting only the winning cutoff. For every candidate threshold, count false passes and false failures, then review their severity by slice. A false pass on an unsafe battery answer is not interchangeable with a false fail caused by a harmless style difference. Aggregate accuracy discards that distinction.

This script reads human labels and grader scores, prints the error counts at explicit thresholds, and lists boundary cases. The threshold list is illustrative input to the script, not an assertion about where your gate should sit.

import json
import sys
from pathlib import Path

rows = [
    json.loads(line)
    for line in Path(sys.argv[1]).read_text(encoding="utf-8").splitlines()
    if line.strip()
]
thresholds = [float(value) for value in sys.argv[2:]]

print("threshold,false_pass,false_fail,total")
for threshold in thresholds:
    false_pass = 0
    false_fail = 0
    for row in rows:
        predicted = float(row["similarity_score"]) >= threshold
        expected = bool(row["human_pass"])
        false_pass += int(predicted and not expected)
        false_fail += int(not predicted and expected)
    print(f"{threshold:.3f},{false_pass},{false_fail},{len(rows)}")

selected = float(sys.argv[2])
boundary = sorted(
    rows,
    key=lambda row: abs(float(row["similarity_score"]) - selected)
)[:10]

print("\nclosest cases to first threshold:")
for row in boundary:
    print(
        row["case_id"],
        row["family"],
        row["human_pass"],
        row["similarity_score"],
        sep="\t",
    )

The ten closest rows are often more informative than the summary. Read the candidate and reference side by side. A cluster of valid paraphrases just below the line suggests the metric is too lexical or the reference set is too narrow. Wrong answers just above the line may share domain vocabulary, copy structure, or differ through a small decisive token. Those are metric-shape problems, not necessarily threshold problems.

Recalibrate whenever the candidate task, reference style, supported languages, or metric implementation changes. Do not compare scores across a metric change as though the scale stayed constant. Run old and new configurations on the same frozen calibration set, publish row-level differences, and make a fresh release decision.

Work through three failures that look similar on a chart

The first worked example is an incident summarizer. The reference says, "Payment authorization succeeded, inventory reservation failed, and no order was created." A good candidate might say, "The card was authorized, but the inventory step failed before order creation." Lexical overlap can be modest even though the operational sequence is preserved. A semantic metric may rank that paraphrase more appropriately, but it still needs separate checks for the three required events.

Represent the sequence as structured facts when possible: payment=authorized, inventory=failed, order_created=false. Test those exactly. Use similarity only for the free-text explanation left after the facts pass. This hybrid design gives the engineer a useful failure: either the state is wrong or the wording drifted, rather than one unexplained score.

The second example is a policy notice. The product requires a specific sentence because counsel approved it. A semantically equivalent rewrite is still a defect. Cosine similarity could consider the rewrite close, while an overlap metric might happen to pass it depending on wording and threshold. Neither expresses the requirement as clearly as an exact or normalized string assertion. If capitalization and whitespace are irrelevant, normalize only those properties and document it.

The third example is retrieval-grounded Q&A. The reference contains a correct warranty period. A candidate repeats that period and then adds an unsupported exception. Similarity may remain high because most content aligns. The failure is entailment or grounding, not distance. Check quoted numbers and named entities deterministically where practical, verify citations against retrieved text, or use a calibrated judge focused on unsupported claims. Do not keep raising a similarity threshold in hope that it will learn factual support.

These three examples can produce the same dashboard symptom: a score near the cutoff. Their fixes differ completely. The summarizer needs paraphrase tolerance plus structured event checks. The legal notice needs an exact assertion. The grounded answer needs claim-level support evaluation. A global metric setting cannot repair all three.

You can use the documented legacy endpoint to test a grader on controlled cases before attaching it to an eval. The shell script below validates a cosine configuration and runs one item. It uses the official field names and makes the template boundaries visible. Because the Evals shutdown is scheduled, treat this as an extraction or maintenance diagnostic and confirm current availability in the official docs.

#!/usr/bin/env bash
set -euo pipefail

: "${OPENAI_API_KEY:?OPENAI_API_KEY must be set}"

grader='{
  "type": "text_similarity",
  "name": "support_answer_similarity",
  "input": "{{ sample.output_text }}",
  "reference": "{{ item.reference_answer }}",
  "pass_threshold": 0.78,
  "evaluation_metric": "cosine"
}'

curl --fail-with-body --silent --show-error \
  https://api.openai.com/v1/fine_tuning/alpha/graders/validate \
  -H "Authorization: Bearer ${OPENAI_API_KEY}" \
  -H "Content-Type: application/json" \
  --data "$(python -c 'import json,sys; print(json.dumps({"grader": json.loads(sys.argv[1])}))' "$grader")"

curl --fail-with-body --silent --show-error \
  https://api.openai.com/v1/fine_tuning/alpha/graders/run \
  -H "Authorization: Bearer ${OPENAI_API_KEY}" \
  -H "Content-Type: application/json" \
  --data "$(python -c 'import json,sys; print(json.dumps({"grader": json.loads(sys.argv[1]), "item": {"reference_answer": sys.argv[2]}, "model_sample": sys.argv[3]}))' \
    "$grader" \
    "An unopened item can be returned within 30 days of delivery." \
    "You may send it back within 30 days if it remains unopened.")"

The 0.78 in that diagnostic is explicitly illustrative. It is useful only to make the request complete. Replace it with a value derived from your labeled task, and save the raw response rather than copying a rounded score into a spreadsheet.

Diagnose the metric before blaming the model

When a similarity gate moves, freeze the candidate output first. If the exact same candidate-reference pair now scores differently, inspect metric configuration, provider version, normalization, reference version, and data rendering. A candidate regression cannot change text that is byte-for-byte identical.

Next, check whether the reference selected for the row is the intended one. Dataset joins fail. Locale fallbacks select English text for another language. Policy IDs point to the current document while labels were authored against the previous revision. A similarity tool will happily produce a number for the wrong pair. Save case_id, reference ID, reference hash, and candidate hash with every result.

Rendering defects often mimic semantic drift. Markdown wrappers, JSON escaping, duplicated system text, or a missing field can alter the compared strings. Log the exact input and reference after templating, subject to data-handling rules. If sensitive text cannot enter CI artifacts, log cryptographic hashes and approved redacted excerpts, then provide a protected debugging path.

Normalization can create a quieter near-miss. One pipeline may compare Unicode in its composed form while another stores visually identical decomposed characters. Smart quotes, nonbreaking spaces, line-ending changes, HTML entity decoding, and case folding can also move a lexical score without changing what a reviewer sees. Save both the raw value hash and the normalized value hash. When only the normalized hash matches, inspect the transformation version before blaming the candidate. Apply normalization narrowly: removing punctuation may help one task but erase a decimal point, a minus sign, or code syntax in another.

Another common look-alike is reference fan-out. A row may allow several references, while an adapter accidentally compares only the first or averages them when the release rule intended the best accepted match. Record every reference ID considered and the combination rule in the result. The per-reference scores show whether the candidate moved or the selection logic did.

Inspect length and language slices. Overlap-based metrics can respond differently when the candidate is much shorter than the reference, and tokenization or morphology can make behavior vary across languages. Do not "fix" a multilingual slice by silently translating it for the metric unless translation is part of the tested production path. That creates a second model or service whose errors are folded into the score.

An offline diagnostic can expose surface-shape differences without pretending to reproduce the OpenAI grader. The code below calculates Python's standard-library sequence ratio and token Jaccard overlap. These values are investigation aids only. They are not RapidFuzz, BLEU, ROUGE, METEOR, embedding cosine, or a substitute for the configured grader result.

import json
import re
import sys
from difflib import SequenceMatcher
from pathlib import Path

def tokens(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def jaccard(left: str, right: str) -> float:
    a, b = tokens(left), tokens(right)
    return len(a & b) / len(a | b) if a | b else 1.0

for line in Path(sys.argv[1]).read_text(encoding="utf-8").splitlines():
    if not line.strip():
        continue
    row = json.loads(line)
    sequence = SequenceMatcher(None, row["reference"], row["candidate"]).ratio()
    token_overlap = jaccard(row["reference"], row["candidate"])
    print(
        row["case_id"],
        f"configured={row.get('similarity_score', 'not-run')}",
        f"sequence_diag={sequence:.3f}",
        f"token_diag={token_overlap:.3f}",
        sep="\t",
    )

If the configured semantic score is stable while token diagnostics fall on valid paraphrases, that may be expected. If every score is identical, look for empty strings or a reused result. If candidates with reversed facts remain close on all similarity measures, the evidence tells you to add a correctness oracle, not hunt for a magical distance function.

Separate errors from failed grades. A missing score, API error, malformed row, or unavailable metric is not a semantic failure and must not be averaged as zero unless the release policy explicitly says infrastructure failure blocks the run. Report errored, failed, and passed independently. Retrying an infrastructure error can be reasonable; retrying a genuine low similarity until it passes is score laundering.

Prove that each score belongs to the compared pair

Parallel evaluation introduces a second failure that can look exactly like a bad similarity metric: a valid score is attached to the wrong row. One worker finishes out of order, a retry emits a second result, or duplicate case IDs collide during aggregation. The dashboard then shows a correct paraphrase below threshold and a contradiction above it. Tuning the metric cannot repair an association defect.

Turn the candidate and reference hashes already saved for diagnosis into a request manifest before sending any comparison. For each run-local item, pair them with the stable case ID, a unique correlation value, metric identity, and configuration revision. Reconcile every returned result against that manifest before semantic gating. The correlation value belongs to the harness and should not depend on array position or completion order. If an external grader does not echo project identifiers, the adapter still has to preserve the relationship around the call rather than reconstruct it from timing.

The diagnostic record should display manifest identity and result identity together. A healthy row has one request, one terminal result, matching candidate and reference hashes, and the expected metric configuration. A broken row may have two terminal results after a retry, a result whose hashes belong to another case, or a missing manifest entry. A score that repeats perfectly across reruns is misleading if every rerun keeps attaching the same other row's result. Reproducibility of the wrong mapping is not metric stability.

Isolation provides the decisive check. Run the disputed candidate-reference pair alone through the same adapter and compare its raw score with the value attached during the parallel run. Then run the neighboring pair that may have been swapped. If the isolated values follow the text pairs while the batch values follow positions or duplicate IDs, the association layer is at fault. If the disputed pair retains the same surprising score in isolation and all identities match, return to metric shape, normalization, and oracle choice.

Retries require an explicit terminal rule. A timed-out attempt followed by a successful attempt must produce one adjudicated result, while preserving both attempt records for operations. Do not average attempts or accept whichever arrives last without declaring that policy. A late result from the first attempt can otherwise overwrite the successful retry after the release report was already assembled. The semantic gate should consume only the reconciled terminal result and should hold when more than one result claims that role.

Roll this out before increasing concurrency or migrating adapters. First generate manifests and reconciliation reports in shadow mode for the existing serial path. Repair duplicate IDs and any row that cannot be tied to exact compared bytes. Next run the same frozen bundle serially and in parallel, comparing row identities as well as scores. Make association errors invalidate the run before allowing new thresholds or metric implementations. Finally enable the faster path. If concurrency and a metric change land together, a swapped result can be misdiagnosed as an implementation difference.

Legacy dashboards are usually the first component to break. Many treat case ID as the unique database key, even though retries and repeated configurations require a run-local identity. Others preserve rounded scores but discard hashes and attempt history. Keep old reports readable, but mark them non-auditable when the pair cannot be reconstructed. Do not manufacture hashes from a later dataset revision and present them as historical evidence.

The cost is concrete. Hashes, manifests, and attempt records add artifact volume and reconciliation code. Running a frozen subset in both serial and parallel modes adds wall-clock time during adapter changes. Rejecting ambiguous results can hold a release that previously received a number. Those costs are smaller than spending reviewer time recalibrating a metric against scores that belong to different text.

The evaluation-platform owner owns correlation, retry reconciliation, and result attachment. The dataset owner guarantees stable case IDs and reference identity. The metric owner validates isolated scores once association is proven. A handoff should contain the run manifest entry, all attempt references, candidate and reference hashes, metric configuration identity, completion order, selected terminal result, neighboring rows that shared an ID or position, and the isolated replay. Sensitive text can stay in the protected artifact because hashes are enough to prove the mapping in the general ticket.

Association checks do not catch an unsuitable metric. A correctly attached cosine or overlap score can still reward copied contradictions, miss unsupported additions, or penalize valid paraphrases. Once pair identity is proven, the product decision still needs the hybrid, deterministic, or human oracle appropriate to that behavior.

Roll out the signal without hiding its cost

Run the metric in shadow mode against current candidate outputs and human review. Publish row-level disagreements by slice. Do not start with a hard global gate, especially when a single reference represents diverse acceptable answers. Shadow results let you improve references and identify unsuitable behaviors without blocking unrelated releases.

Promote only the slices where similarity has demonstrated useful separation. An extractive incident summary might become blocking after required facts also pass. A creative response generator may keep similarity as a drift alert. A policy notice should probably graduate to an exact assertion instead. One evaluation suite can use different oracles for each behavior.

Keep metric configuration in version control and require a reason for threshold changes. A pull request that raises a cutoff should include the frozen calibration report, changed false-pass and false-fail cases, and owner approval. Otherwise threshold changes become a quiet way to make a failing candidate green or to satisfy an arbitrary coverage target.

This CI job validates fixtures, runs the configured adapter, and applies separate release policies. The artifact step uploads redacted results only. Its commands assume scripts owned by the repository; each script has one job so failures remain diagnosable.

name: text-similarity-eval

on:
  pull_request:
    paths:
      - "prompts/**"
      - "evals/similarity/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Validate labeled fixtures
        run: python scripts/evals/validate_similarity_rows.py evals/similarity/cases.jsonl
      - name: Generate candidate outputs
        run: python scripts/evals/run_candidate.py --output artifacts/candidates.jsonl
      - name: Run configured similarity adapter
        run: python scripts/evals/run_similarity.py --output artifacts/results.jsonl
      - name: Apply slice release policy
        run: python scripts/evals/check_similarity_gate.py artifacts/results.jsonl
      - uses: actions/upload-artifact@v4
        with:
          name: redacted-similarity-results
          path: artifacts/redacted-results.jsonl

The costs are concrete. More references require expert authoring and maintenance. Semantic grading adds latency and service cost. Lexical metrics are cheaper but can penalize valid variation. Human calibration slows rollout. Hybrid checks add code and ownership. Record which cost you accepted and which failure it prevents, or later optimization will remove the most important control first.

During migration from OpenAI Evals, export the row inputs, references, labels, metric name, threshold rationale, and raw results. Run the frozen calibration bundle in the replacement harness. Expect numerical differences across implementations, especially if the embedding model or metric library changes. Approve the new gate from human decisions and error cases, not by forcing every old floating-point score to match.

Know when similarity is the wrong oracle

Do not use it for exact contracts. JSON schema, tool arguments, enum labels, arithmetic, identifiers, and mandatory text are better tested with deterministic assertions. Similarity makes these requirements less clear and produces false arguments about thresholds when a direct check would identify the field that broke.

Avoid it as the sole measure of factuality or grounding. A candidate can copy a reference and append a false claim. It can preserve vocabulary while reversing a relationship. It can be semantically close to a plausible answer that the supplied evidence does not support. Check claims against authoritative data or use a separately calibrated grounding evaluation.

Skip a single-reference similarity gate for creative tasks with many valid outputs. Marketing copy, brainstorming, conversational empathy, and code explanations may succeed without resembling one canonical response. Evaluate task-specific constraints, pairwise preferences, human outcomes, or deterministic properties instead.

Do not use a legacy hosted grader as the foundation of a new long-lived pipeline when its shutdown is already scheduled. The concepts and calibration assets remain valuable, but the transport should be replaceable. Separate the metric adapter from the release policy now, while you can compare both systems on the same cases.

Most importantly, stop tuning when the metric disagrees with a clear product decision. If a safe, correct paraphrase repeatedly fails while a copied contradiction passes, the metric is telling you about textual distance, exactly as asked. The defect is in the oracle choice. Replace it with checks that can see the behavior you actually intend to ship.