The answer is wrong because the model saw a different conversation

A customer uploads a damage photo, explains the problem by voice, and asks, “Can you replace the left one?” The assistant answers about the right item. The model may be weak, but an attachment-order bug, a clipped recording, or stale conversation state can produce the same wrong answer.

Multimodal testing starts by proving what each component received and when it received it, because only then can you make a fair claim about reasoning across text, images, and audio.

Model the conversation as an ordered evidence trail

A chat transcript is not enough. Text appears inline, while images and audio often travel through separate upload services, object stores, preprocessing workers, and model requests. The message “compare this with the first one” becomes meaningless if the trace does not show which artifact “this” referenced or whether the first artifact was still in context.

Give each user task a conversation ID, each turn a monotonic sequence number, and each binary artifact an immutable identifier. Store the artifact identifier on the turn that introduced it and on every later request that references it. Do not rely on the second thumbnail, the latest upload, or an array position as identity. Positions change when uploads finish out of order, retries create replacements, or the UI removes a failed item.

Preserve four clocks without pretending they are interchangeable: client capture time, server receipt time, preprocessing completion time, and model-request time. Wall clocks across devices can disagree. Sequence numbers and explicit causal links establish order within the conversation; timestamps help diagnose latency. If an audio upload completes after a later text message, the system needs a defined rule for whether that audio belongs to the earlier turn, the later turn, or neither.

Record the actual artifact properties used downstream. For an image, that can include media type, byte length, dimensions after decoding, orientation handling, content hash, and preprocessing revision. For audio, useful facts include media type, byte length, duration derived from the decoded media, channel count, and the transcription revision. Browser-requested constraints are not proof of captured settings. A device session can inspect the resulting track with MediaStreamTrack.getSettings() and record the settings the browser reports.

Artifact identity and content identity solve different problems. A new upload attempt should get a new artifact ID even if it contains identical bytes, because its lifecycle and permissions differ. A content hash can reveal accidental duplication or prove that a preprocessing step produced the expected fixture. Never place raw personal data in the identifier or trace header.

Define state transitions for every artifact. A simple lifecycle might be declared, uploaded, decoded, accepted, attached, and expired, with rejected available from any processing state. The model request should reference only artifacts in an accepted state. A UI preview does not prove acceptance; it may render a local object URL before the server has decoded anything.

The following TypeScript contract enforces five rules: sequence numbers are unique across events and turns, turns arrive in increasing sequence order, no turn is empty of both text and artifacts, every referenced artifact is known, and no turn references an artifact before its accepted event.

Each rule gets its own negative fixture, because a contract is only worth as much as the branches its tests actually reach. It is easy to write one happy-path call plus one throwing case and describe the whole function as covered, and that is how a validator ends up shipping with four guards nobody has ever executed. Mutate any single guard in the function below to if (false) and exactly one of the five assertions stops throwing. That is the property being claimed here, and it is the property you should verify on your own version before trusting it.

import assert from "node:assert/strict";

type ArtifactEvent = {
  artifactId: string;
  state: "declared" | "uploaded" | "accepted" | "rejected";
  sequence: number;
};

type Turn = {
  turnId: string;
  sequence: number;
  role: "user" | "assistant";
  text?: string;
  artifactIds: string[];
};

export function validateConversation(
  events: ArtifactEvent[],
  turns: Turn[],
): void {
  const sequences = [
    ...events.map((event) => event.sequence),
    ...turns.map((turn) => turn.sequence),
  ];
  if (new Set(sequences).size !== sequences.length) {
    throw new Error("conversation sequence numbers must be unique");
  }
  if (turns.some((turn, index) => index > 0 && turns[index - 1]!.sequence >= turn.sequence)) {
    throw new Error("turns must be supplied in sequence order");
  }

  const acceptedAt = new Map<string, number>();
  for (const event of events) {
    if (event.state === "accepted") {
      acceptedAt.set(event.artifactId, event.sequence);
    }
  }

  for (const turn of turns) {
    if (!turn.text && turn.artifactIds.length === 0) {
      throw new Error(`${turn.turnId} has no text or artifacts`);
    }
    for (const artifactId of turn.artifactIds) {
      const acceptedSequence = acceptedAt.get(artifactId);
      if (acceptedSequence === undefined) {
        throw new Error(`${turn.turnId} references unknown ${artifactId}`);
      }
      if (acceptedSequence >= turn.sequence) {
        throw new Error(`${turn.turnId} uses ${artifactId} before acceptance`);
      }
    }
  }
}

const events: ArtifactEvent[] = [
  { artifactId: "image-left", state: "accepted", sequence: 2 },
];
const validTurns: Turn[] = [
  {
    turnId: "turn-1",
    sequence: 1,
    role: "user",
    text: "I will upload the damaged item.",
    artifactIds: [],
  },
  {
    turnId: "turn-2",
    sequence: 3,
    role: "user",
    text: "Can you replace this one?",
    artifactIds: ["image-left"],
  },
];

validateConversation(events, validTurns);

// Guard 1: a duplicate sequence number across events and turns.
assert.throws(
  () =>
    validateConversation(
      [...events, { artifactId: "image-right", state: "accepted", sequence: 1 }],
      validTurns,
    ),
  /conversation sequence numbers must be unique/,
);

// Guard 2: turns supplied out of order.
assert.throws(
  () => validateConversation(events, [validTurns[1]!, validTurns[0]!]),
  /turns must be supplied in sequence order/,
);

// Guard 3: a turn carrying neither text nor artifacts.
assert.throws(
  () =>
    validateConversation(events, [
      ...validTurns,
      {
        turnId: "turn-silent",
        sequence: 4,
        role: "user",
        artifactIds: [],
      },
    ]),
  /turn-silent has no text or artifacts/,
);

// Guard 4: a reference to an artifact the events never accepted.
assert.throws(
  () =>
    validateConversation(events, [
      ...validTurns,
      {
        turnId: "turn-3",
        sequence: 4,
        role: "user",
        text: "Use the other photo.",
        artifactIds: ["image-missing"],
      },
    ]),
  /references unknown image-missing/,
);

// Guard 5: a turn that uses an artifact before its accepted event.
assert.throws(
  () =>
    validateConversation(events, [
      {
        turnId: "turn-early",
        sequence: 1,
        role: "user",
        text: "Can you replace this one?",
        artifactIds: ["image-left"],
      },
    ]),
  /turn-early uses image-left before acceptance/,
);

Guard four and guard five are worth reading together, because a single fixture cannot exercise both. An unknown artifact never reaches the acceptance comparison, since the lookup returns undefined and the function throws first. An artifact used too early has to be genuinely known, which is why the last fixture reuses image-left and places the turn at sequence one, before the accepted event at sequence two. Reusing the unknown-artifact fixture for both would leave the acceptance branch untouched while the suite still looked green.

This contract does not score the answer. It proves that the test harness and application agree on the evidence boundary. Once it passes, a semantic assertion can ask whether the answer identifies the damage, compares the intended objects, or requests more information. Without it, a low model score may simply punish the model for an artifact it never received.

Conversation state needs its own version. Systems commonly compress old turns, drop large artifacts, or replace images with descriptions when a context limit is approached. Record the state-builder revision and the exact artifact IDs included in each model request. “The conversation contained the image” is too vague. The relevant fact is whether that request contained the image bytes, a derived description, a reference understood by the provider, or nothing.

Prove each modality path before testing them together

Start with controlled text because it isolates conversation state and response rendering. Then test image upload and transformation without asking for a sophisticated answer. Finally test audio capture, upload, decode, and transcription with a known signal. Cross-modal scenarios become useful only after each path can show that the expected evidence reached the request boundary.

For images, keep a compact fixture set with purposeful variation. Include a normal RGB image, a rotated photo whose orientation metadata matters, a high-resolution image that triggers resizing, a transparent image if the product accepts one, and a file with a declared media type that does not match its bytes. The expected result is not always acceptance. A clean rejection with an actionable message is correct for unsupported content.

Compare transformations at the stage you own. If the application rotates and resizes an image before model submission, test the decoded dimensions and a checked-in expected output produced by that version of the transformation. A browser screenshot can verify the preview layout, but it cannot prove the model request contained the same bytes. Link the preview artifact, transformed artifact, and request artifact explicitly.

Audio needs signal fixtures rather than arbitrary voice recordings. A short WAV file can contain a known spoken phrase, a leading silence interval, or two separated utterances. Keep the expected duration and hash. If the application accepts several containers or codecs, test acceptance and decoding per supported path. Do not claim microphone coverage from a test that uploads a WAV file. Upload coverage and device-capture coverage are different rows in the matrix.

The browser media APIs have conditions worth representing in tests. MDN's getUserMedia() reference explains its secure-context, permission, and device requirements. A request can reject when permission is denied or no matching device exists, and a user may leave the permission prompt unanswered. Your UI therefore needs explicit states for requesting access, recording, denied access, unavailable input, upload in progress, and retry. A single happy-path recording test misses most user-visible failures.

MediaRecorder delivers captured data through dataavailable events. A timeslice passed to start() is not a precise clock, so chunk counts should not be used to infer exact elapsed time. Track elapsed time separately and derive media duration from the recorded artifact when the product needs it. This distinction matters when tests report “recording was five seconds” only because five chunks arrived.

Transcription is a derived artifact. Preserve the audio artifact ID, transcription version, text, language decision, and word or segment timings when the service returns them. Avoid fabricating timing fields when the service does not provide them. Tests can compare the transcript with expected facts, but the original audio remains necessary for diagnosing clipping, channel errors, or a wrong recording.

For captions and time-aligned text, validate timing syntax and ordering independently from word accuracy. WebVTT defines timed cues for audio and video. An application can produce the right words with overlapping, inverted, or badly shifted cue intervals. Those are rendering and timing defects, not necessarily transcription defects.

Build paired controls. One image should contain the fact needed to answer, while a matched image should omit it. One audio clip should contain the account suffix, while another should contain a different suffix. If the answer is identical for both, the system may be ignoring the modality or relying on text leakage. Avoid embedding the expected answer in the filename, alt text, test title sent to the model, or surrounding prompt.

Catch cross-modal failures with evidence-based scenarios

The first worked scenario is attachment reordering. A user selects two product photos quickly. The larger first file uploads more slowly, so the server receives the second file first. The UI still labels the previews “first” and “second” by selection order, while the backend builds the request from completion order. The user asks about the first product, and the assistant analyzes the wrong one.

The diagnostic signature is specific. Client events show selection order A then B. Upload completion shows B then A. The turn manifest points “first” to position zero in the completion array rather than artifact A. Both images decode correctly and the model answers consistently with B. That is an application state defect. Repeating the model call with artifact A may produce the desired answer, but the decisive proof is the mismatched reference before inference.

Fix it by assigning IDs at selection time and carrying those IDs through upload completion, preview state, turn submission, and request assembly. The trade-off is more state management. Retries cannot casually reuse an ID, and deletion must remove the intended lifecycle without breaking references in historical turns. The additional complexity buys a stable meaning for follow-up language.

The second scenario is a voice note with its first word clipped. The user says, “Do not cancel the booking,” but capture begins late and the stored audio starts with “cancel the booking.” The transcript is accurate for the bytes it received, and the model follows that transcript. A semantic evaluator labels the response dangerously wrong.

Inspect the capture-start event, recorder-start event, first non-silent audio region, stored duration, and transcript. If the stored artifact lacks the word, this is capture or voice-activity logic. If the waveform contains it but the transcript does not, the transcription path is the stronger suspect. If both are correct and the response still cancels, investigate reasoning or tool policy. The same user complaint reaches three different owners.

Do not “fix” clipping by adding an arbitrary delay before every recording. That raises interaction latency and may still fail on slow devices. Make the UI wait for an observed recording-ready state before indicating that speech is being captured, preserve a short approved pre-roll if the architecture supports it, and test the transition. Device behavior varies, so include real-device sessions in addition to synthetic upload tests.

The third scenario is stale visual context. A shopper uploads a red chair, asks for matching rugs, then uploads a blue sofa and asks, “Which of those would work with this?” The answer discusses the chair because the state builder retained the first image summary but dropped the new image after preprocessing timed out.

Evidence should show the second image declared, its preprocessing failure, the follow-up turn referencing it, and the actual model request omitting it. The UI must not present the turn as fully sent. A good product response may ask the user to retry the image rather than answer from stale context. Treating omission as an empty but valid attachment invites confident guesses.

The following Python test verifies response evidence against the artifacts each turn actually required. It catches a stale-image answer even if the prose happens to mention a plausible color.

The important detail is where the expectation comes from. Both sides are read from the run's trace directory: the manifest the application recorded when it assembled the model request, and the answer record the application produced. Neither is a literal in the test body. Writing the required IDs inline next to an answer built from the same two strings would produce a test that passes when verify_answer_evidence is replaced by return None, because both sides of the comparison would be constants the test itself authored. Reading the manifest keeps the check pointed at the system under test.

import json
from dataclasses import dataclass
from pathlib import Path

TRACE_DIR = Path("artifacts/multimodal/turn-trace")


@dataclass(frozen=True)
class TurnManifest:
    turn_id: str
    attached_artifact_ids: frozenset[str]
    carried_artifact_ids: frozenset[str]

    @property
    def required_artifact_ids(self) -> frozenset[str]:
        return self.attached_artifact_ids | self.carried_artifact_ids


@dataclass(frozen=True)
class AnswerRecord:
    turn_id: str
    status: str
    evidence_artifact_ids: frozenset[str]


def load_turn_manifest(turn_id: str) -> TurnManifest:
    record = json.loads(
        (TRACE_DIR / f"{turn_id}.manifest.json").read_text(encoding="utf-8")
    )
    return TurnManifest(
        turn_id=record["turnId"],
        attached_artifact_ids=frozenset(record["attachedArtifactIds"]),
        carried_artifact_ids=frozenset(record["carriedArtifactIds"]),
    )


def load_answer(turn_id: str) -> AnswerRecord:
    record = json.loads(
        (TRACE_DIR / f"{turn_id}.answer.json").read_text(encoding="utf-8")
    )
    return AnswerRecord(
        turn_id=record["turnId"],
        status=record["status"],
        evidence_artifact_ids=frozenset(record["evidenceArtifactIds"]),
    )


def verify_answer_evidence(
    answer: AnswerRecord,
    manifest: TurnManifest,
) -> None:
    if answer.turn_id != manifest.turn_id:
        raise AssertionError(
            f"answer {answer.turn_id} does not belong to {manifest.turn_id}"
        )
    if answer.status != "completed":
        raise AssertionError(f"{answer.turn_id} did not complete")
    missing = manifest.required_artifact_ids - answer.evidence_artifact_ids
    if missing:
        raise AssertionError(
            f"{answer.turn_id} omitted required evidence: {sorted(missing)}"
        )


def test_follow_up_uses_new_sofa_and_prior_rug_options() -> None:
    manifest = load_turn_manifest("turn-follow-up")
    answer = load_answer("turn-follow-up")

    assert manifest.required_artifact_ids, (
        "the recorded manifest required no artifacts, "
        "so this turn cannot judge cross-modal evidence"
    )
    verify_answer_evidence(answer, manifest)


def test_stale_chair_context_cannot_pass_for_new_sofa() -> None:
    manifest = load_turn_manifest("turn-follow-up")
    stale_answer = AnswerRecord(
        turn_id=manifest.turn_id,
        status="completed",
        evidence_artifact_ids=frozenset(
            {"image-red-chair", "image-rug-options"}
        ),
    )

    try:
        verify_answer_evidence(stale_answer, manifest)
    except AssertionError as error:
        assert "image-blue-sofa" in str(error)
    else:
        raise AssertionError("stale image evidence was accepted")

Three things make this pair sensitive rather than decorative. The positive test fails if the application omits an artifact the manifest says the turn needed, which is the stale-context defect. The empty-manifest guard stops the positive test from passing vacuously when instrumentation regresses and records no required artifacts at all, because a subtraction against an empty set can never find anything missing. The negative test fails if verify_answer_evidence stops checking, since a verifier that returns without raising falls through to the explicit else branch. It also fails if the recorded manifest no longer requires the blue sofa, which ties the negative fixture back to the same run data instead of to a constant.

The trace files themselves are the application's own output, written when it assembled the request and recorded the answer. Producing them from the test would recreate the circularity this design removes. If your system does not yet emit a per-turn manifest, that instrumentation is the first thing to build, because without it no downstream evaluation can distinguish a model that reasoned badly from a request that never carried the image.

Evidence IDs are not a full correctness oracle. A response can reference the right image and misread it. Add task facts next: the damaged corner is on the left, the visible total is 42.50, or the audio explicitly contains a refusal. Use deterministic extraction only where the fact can be checked reliably. Use a reviewed rubric for ambiguous visual interpretation, and preserve the reviewer’s rationale.

Negative scenarios matter as much as successful fusion. Ask a question whose answer is not visible, submit silent audio, upload a corrupt image, refer to an expired artifact, and interrupt an upload with a later text turn. The desired behavior is often uncertainty, retry, or clarification. A system that always produces a fluent answer will look productive while violating the evidence contract.

Tell a model defect from a browser or pipeline defect

Begin at the earliest boundary with independent evidence. Did the browser obtain permission? Did it create a track of the intended kind? What settings did the track report? Did a nonempty artifact leave the client? Did the server verify its bytes and media type? Did preprocessing create an accepted output? Did request assembly include that output? Only after those checks should the model response enter the investigation.

Browser console messages and DOM state can identify permission and UI failures. Network records show upload status and request sequencing. Server logs show decode and transformation outcomes. The model-request manifest shows what crossed the provider boundary. The response record shows which artifact references came back, if the integration provides such evidence. Avoid filling gaps with assumptions.

A permission denial and a missing device can look identical if the UI maps every exception to “microphone unavailable.” Preserve the browser exception name in restricted diagnostic telemetry and present a user-safe message. NotAllowedError and NotFoundError lead to different fixes. Do not log a device label or stable device identifier unless the product has a justified, reviewed need.

Codec mismatch is another near-miss. Recording succeeds locally, but a CI browser emits or uploads a media type the backend decoder does not accept. The model never sees audio. Check MediaRecorder.isTypeSupported() before choosing a recording type, record the actual blob type, and have the server validate what it receives. Support should be based on the end-to-end path, not browser capability alone.

Image orientation failures often masquerade as weak vision. The preview uses browser decoding that honors orientation, while a server library reads raw dimensions and sends a sideways image downstream. Compare a canonical transformed fixture or inspect the preprocessing output. If the model received the rotated bytes, do not spend time adjusting the prompt.

Playback can create the opposite illusion: the request and model response are correct, but the customer hears a truncated or duplicated answer. Keep the response text, generated audio artifact, playback-start event, interruption event, and playback-end state distinct. If the stored audio contains the complete answer but playback stops when an unrelated UI sound gains focus, the model is not the failing component. If the interruption was caused by real user speech, the same stop may be correct behavior. Your oracle needs the source of the interruption, not merely the fact that playback ended early.

Test this boundary with one synthetic response that is allowed to finish and another that is intentionally interrupted. Confirm that the first reaches its completed playback state and that the second retains an explicit interruption reason. Listening to a recording is useful during exploration, but CI should assert the state transitions and artifact identity. The added instrumentation costs event volume and implementation work, yet it prevents every audio-output complaint from becoming an expensive model investigation.

The browser test below verifies a deterministic attachment UI without requiring a physical camera. It uses Playwright’s setInputFiles() buffer support to upload known bytes, then checks that the selected artifact remains attached to the intended turn after another text message. This covers file input and UI state, not camera capture or model perception.

import { test, expect } from "@playwright/test";

test("an uploaded image remains linked to its originating turn", async ({
  page,
}) => {
  await page.goto("/multimodal-fixture");

  await page.getByLabel("Attach image").setInputFiles({
    name: "left-item.png",
    mimeType: "image/png",
    buffer: Buffer.from(
      "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=",
      "base64",
    ),
  });
  await page.getByLabel("Message").fill("Inspect the left item");
  await page.getByRole("button", { name: "Send message" }).click();

  const firstTurn = page.getByTestId("turn-user-1");
  await expect(firstTurn.getByRole("img", { name: "left-item.png" }))
    .toBeVisible();

  await page.getByLabel("Message").fill("Keep using that image");
  await page.getByRole("button", { name: "Send message" }).click();

  await expect(firstTurn.getByRole("img", { name: "left-item.png" }))
    .toHaveCount(1);
  await expect(page.getByTestId("turn-user-2"))
    .toHaveAttribute("data-references", "image-left-item");
});

When this fails, inspect the action log, DOM snapshots, and network requests in the Playwright trace. A missing preview after the first click is different from a second-turn reference pointing at the wrong ID. Trace Viewer shows the browser-side journey; it does not reveal private server artifacts unless the application deliberately exposes safe identifiers.

Semantic non-determinism needs repeated execution, but repetition should answer a defined question. Run the same frozen request several times to see whether a required fact is consistently preserved, not to hunt for a lucky pass. Report the set of outcomes and configuration used. Do not average a severe unsupported action into an acceptable score.

Put honest coverage into CI and device runs

Split the suite by what it can prove. A fast contract job validates manifests, artifact lifecycles, reference integrity, preprocessing fixtures, and refusal paths. A browser job validates file upload, transcript ordering, previews, retries, and accessible status. A service-integration job exercises supported media types against a test deployment. Real-device sessions cover permissions, physical capture, playback, echo, and interruption.

Name jobs after their evidence. “Multimodal E2E” is misleading if the runner uploads a one-pixel PNG and mocks every service. “Attachment state contract” is precise. The distinction helps release owners understand what remains untested when a device lab is unavailable.

Keep binary fixtures small, reviewed, and licensed for test use. Synthetic media avoids accidental personal data and makes expected transformations stable. Store fixture provenance and creation notes. Do not repeatedly transcode the golden input during test setup, because toolchain updates can change the bytes and turn a product test into an untracked fixture migration.

Pin preprocessing and evaluation versions in the result. A library upgrade may improve format support while changing resize output. Review expected artifacts through an explicit migration instead of updating snapshots until CI turns green. If a semantic rubric changes, re-label the calibration cases and preserve the old version for historical results.

The following workflow wires separate contract and browser jobs. It sets up pnpm before asking the Node action to use the pnpm cache, installs the Chromium browser required by the selected Playwright project, and uploads traces only when the browser job fails. The paths are illustrative repository paths and should match the suite that owns these tests.

name: multimodal-contracts

on:
  pull_request:
    paths:
      - "multimodal/**"
      - "tests/multimodal/**"

jobs:
  contracts:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install pytest
      - run: python -m pytest tests/multimodal/test_evidence.py -q

  attachment-ui:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
        with:
          version: 10
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - run: pnpm exec playwright install --with-deps chromium
      - run: pnpm exec playwright test tests/multimodal/attachment.spec.ts --project=chromium
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: multimodal-playwright-traces
          path: test-results/
          retention-days: 7

CI has costs beyond runtime. Browser binaries and media fixtures increase cache and artifact size. Repeated model calls cost money and can produce unstable gates. Device labs add scheduling delay. Human review slows release feedback. Put deterministic contracts on every change, targeted integration tests on relevant changes, and expensive semantic or device suites on a cadence matched to risk.

Roll out from evidence contracts outward. First make artifact IDs and lifecycle events observable. Next add one modality at a time with matched positive and negative fixtures. Then add two-turn references. Only after those stabilize should the team add long conversations, modality switching, interruption, and semantic grading. This sequence prevents a complicated failure from landing in an uninstrumented system.

Quarantine should be rare and specific. A physical-device test can be unavailable because the lab is down, but its absence must remain visible in the release report. Do not mark it passing or replace it silently with an upload test. An intermittently wrong product outcome deserves investigation, not indefinite retrying until one answer satisfies the rubric.

Know when automation is the wrong judge

Do not use a screenshot assertion to decide whether an image answer is semantically correct. Screenshots are excellent for the application’s preview, crop controls, focus state, caption layout, and error presentation. The same pixels do not prove that a medical image interpretation, damage assessment, or creative description is acceptable.

Avoid synthetic audio as the only evidence for a voice product used in noisy, accented, or assistive contexts. Synthetic fixtures provide control and privacy, but they do not represent microphones, rooms, speech patterns, or network conditions. Recruit reviewed, consented coverage that matches the product population, and keep the claims narrower than the sample.

Do not send sensitive production media to a third-party grader merely because automated review is convenient. Confirm data handling, retention, regional, and access requirements first. A local deterministic check or trained human reviewer in the approved environment may be the right trade.

Skip exact transcript matching when several transcriptions preserve the same required meaning. Exact matching creates noise around punctuation, casing, or harmless wording. Assert required entities and negation carefully, and retain word-level review for cases where wording itself controls an action. Conversely, do not reduce a safety-critical phrase to loose similarity that can miss “not.”

Never claim physical camera, microphone, or speaker coverage from file injection. That technique is valuable because it is deterministic, but it bypasses permission prompts, device selection, capture readiness, acoustic echo, and playback. Put those properties in a device matrix with named environments and recorded settings.

Some cross-modal judgments remain product decisions. Whether an answer gives enough visual detail, handles an ambiguous gesture well, or asks the right clarification may need expert review. Automation can assemble the evidence, enforce artifact integrity, and route uncertain cases. It should not manufacture certainty where the acceptance boundary is still being negotiated.