Metamorphic Testing for LLM Outputs Without Exact Expected Answers

Exact expected answers are often the wrong oracle for an LLM. Two responses can differ in wording and both satisfy the task, while a fluent response can violate a crucial constraint. Metamorphic testing replaces the single golden string with a stronger question: when the input changes in a controlled way, how should the relevant behavior stay the same or change?

The method is useful only when the transformation and expected relation are specified independently of the output. A good test family records its seed, applies one purposeful change, validates that change, scores both outputs, and makes a paired judgment with uncertainty and slice context.

Test Relations Instead of Exact Sentences

A metamorphic relation has four parts: a seed input, a transformation, an expected relation, and an oracle. Reordering irrelevant fields might require semantic invariance. Tightening a maximum budget should require a non-increasing recommendation. Removing a source document should require citations to that document to disappear. The relation is about behavior, not lexical similarity.

The DeepEval introduction presents evaluation in terms of test cases, metrics, and evaluation workflows. Metamorphic families fit that general structure, but the team must define and validate its own domain relations. A framework cannot decide whether a transformed request should preserve or alter the answer.

Select Seeds With Provenance and Purpose

Choose seeds from production traces, incidents, domain specifications, or expert-designed cases. Record source, collection time, product surface, privacy treatment, original labels, and any transformations already applied. A seed should have a reason to exist: it represents a common workflow, a known severe failure, a boundary, or a slice where the system is uncertain.

Do not use only memorable failures. Include routine behavior so transformations can detect broad fragility, and reserve protected allocations for severe or historically weak slices. If redaction or normalization changes the premise, treat the result as a different seed rather than claiming direct lineage.

Group all descendants under a family identifier. This prevents one productive seed with dozens of variants from receiving dozens of times the statistical influence of an independent case.

Build a Transformation Catalog

Classify transformations by expected effect. Invariance transformations change presentation while preserving task semantics: whitespace, field order, harmless paraphrase, or equivalent units after exact conversion. Equivariance transformations require a predictable output change: translating an input may require the response language to change while preserving facts. Monotonic transformations strengthen a constraint and require a directional response. Removal transformations delete evidence and require dependent claims to disappear or become uncertain.

Every catalog entry should state preconditions, transformation code version, changed field, preserved facts, expected relation, allowed tolerance, invalidation checks, and risk owner. A paraphrase that changes urgency or modality is not valid for semantic invariance. Sample transformed inputs for human validation before scaling them.

Avoid transformations whose expected effect is debatable. When experts disagree about whether the answer should change, the case belongs in rubric research, not an automated release gate.

Validate the Transformation Before the Model

False violations often originate in the test generator. Apply schema checks, unit conversions, entity tracking, and diff constraints before calling the model. For text transformations, use blind human review on a calibration sample to verify semantic preservation or the intended semantic delta.

Store a machine-readable contract:

{
  "relationId": "budget-monotonicity/v2",
  "seedId": "trip-plan-018",
  "transformation": "reduce-budget",
  "changedFields": ["budgetUsd"],
  "preservedFields": ["city", "dates", "travelers"],
  "expectation": "estimatedTotal must not increase",
  "familyId": "trip-plan-018-family"
}

The contract supports review and replay. It does not prove the relation is valid; that remains a domain decision.

Layer Deterministic and Semantic Relation Checks

Use deterministic oracles whenever the relation is machine-checkable. Parse structured outputs, compare selected fields, verify set membership, recalculate totals, and inspect tool arguments. These checks are reproducible and explain failures directly.

type Plan = { estimatedTotal: number; currency: string; items: string[] };

export function budgetMonotonicity(base: Plan, reduced: Plan) {
  if (base.currency !== reduced.currency) {
    return { pass: false, reason: "currency changed" };
  }
  return {
    pass: reduced.estimatedTotal <= base.estimatedTotal,
    reason: "reduced-budget estimate must not exceed base estimate",
  };
}

Use model graders for semantic equivalence, entailment, tone preservation, or appropriate uncertainty only after writing a narrow rubric. Give the grader both inputs, both outputs, the exact transformation, and the relation. Require evidence and allow uncertain; a forced binary answer hides grader limits.

Calibrate Human and Model Oracles

Assemble paired examples containing clear satisfaction, clear violation, subtle boundary cases, and invalid transformations. Human annotators label them independently using the same relation rubric. Measure agreement before adjudication, revise ambiguous criteria, and preserve both raw and final labels.

Compare the model grader with adjudicated outcomes by relation and slice. Inspect false passes and false failures separately because their release costs differ. Freeze the grader prompt, model configuration, parsing logic, and rubric version for a candidate comparison.

Do not calibrate and report final accuracy on the same pairs. Keep a grader-development split and an untouched audit split, grouped by seed family to prevent transformed siblings from crossing the boundary.

Control Randomness With a Paired Design

Run base and transformed cases under matched settings and as close in time as operationally practical. If the system is stochastic, repeat complete pairs rather than running all bases first and all transformations later. Record random seeds where the stack honors them, plus prompt, tool, retrieval, and dependency versions.

Define the unit of analysis before execution. You might score whether each pair ever violates the relation, the proportion of repetitions that violate it, or a severity-weighted outcome. These answer different questions. Do not select the most favorable aggregation after seeing results.

Pairing reduces some background variation but does not eliminate it. Retrieval changes, external tools, and nondeterministic graders can still affect the relation. Capture their traces and classify infrastructure failures separately from model-behavior violations.

Analyze Violations by Slice

Report results per transformation type, source, locale, input length, domain, tool path, and consequence severity. A stable aggregate can conceal that invariance fails for one language or that citation removal works only on short contexts. Use family-aware counts so related variants do not masquerade as independent evidence.

Show confidence or bootstrap intervals for relation-violation rates when sample design supports them. Rare critical slices may be too small for a precise interval; report the count and escalate confirmed severe failures rather than relying on an unstable percentage.

Track invalid transformations and grader abstentions as separate rates. A transformation catalog with a high invalidation rate is not producing efficient evidence, even if the valid pairs look healthy.

Prevent Leakage Across Test Families

Derived pairs can leak through prompt tuning, model training, few-shot examples, or grader calibration. Record parent-child lineage and compare family identifiers across development and holdout sets. Exact hash checks are not enough because transformed siblings are intentionally different.

When a failed sealed pair is disclosed for debugging, move that family into the development regression suite. Replenish the sealed relation set from clean seeds and document the version change. Repeatedly choosing candidates against the same relations can overfit the transformation catalog even when exact examples remain hidden.

Turn Relations Into Release Gates

Predefine which relations are advisory, aggregate-gated, or critical vetoes. An illustrative rule might block release on any adjudicated violation that fabricates removed evidence, while requiring a confidence-bound comparison for low-severity formatting invariance. These are policy examples, not universal standards.

Compare candidate and baseline on the same seed families and relation versions. A release can ship, hold, or escalate. Escalation is appropriate when transformation validity, grader judgment, or sample size leaves a critical result unresolved. Keep the reason visible instead of forcing uncertainty into a pass.

Failure Modes and Tradeoffs

Weak relations test trivial consistency and miss task quality. Overly strict invariance rejects legitimate adaptation. Generated transformations can alter hidden semantics. Lexical similarity can reward copied wording rather than correctness, while a broad model grader can accept persuasive contradictions. Too many variants from one seed inflate confidence.

Metamorphic testing adds coverage where exact references are unavailable, but it does not replace direct correctness checks, production monitoring, or expert review. Use it for relationships you can defend and combine it with other evidence.

Operational Checklist

• State the expected relation before generating transformed inputs.
• Preserve seed provenance and family identifiers.
• Define transformation preconditions, changed facts, and preserved facts.
• Validate generated transformations independently of model outputs.
• Prefer deterministic oracles for structured and directional properties.
• Calibrate semantic graders on independent, adjudicated pairs.
• Run matched pairs and record the complete execution configuration.
• Analyze relation violations, invalid transformations, and abstentions by slice.
• Keep sibling transformations in the same data split.
• Attach uncertainty and severity to every release rule.
• Retire relations that do not reveal actionable failures.

Action Plan: Launch One High-Value Relation Family

Choose one consequential behavior whose exact wording may vary, such as removing unsupported citations when evidence is deleted. Collect clean seeds across relevant slices, define a single transformation, and have domain reviewers validate that its expected effect is unambiguous. Implement the deterministic parts of the oracle, then calibrate any semantic judgment on a separate pair set.

Run baseline and candidate as matched pairs, review every severe violation, and publish family-aware slice results with uncertainty. Gate the release only after transformation validity, grader error, and escalation rules are documented. One defensible relation family is more valuable than a large catalog of transformations no one can explain.