When multimodal JSON is valid but still wrong

Your vision endpoint returns HTTP 200 and a JSON object, yet the UI highlights a button that is not in the uploaded screenshot. In another run, an audio observation points at an image ID, and the ordinary schema check still passes because both fields are strings. A green JSON validator proves syntax and shape, not that the answer is grounded in the media you sent. For teams doing multimodal structured JSON output testing, the useful question is which boundary broke, not whether a brace was missing.

Why valid JSON can still be the wrong answer

A multimodal response crosses several boundaries before an application can trust it. The client has to transmit the intended bytes. The adapter has to preserve the association between those bytes and their artifact IDs. The model-facing layer has to request the expected output form. The returned text has to parse. Its fields have to match the current contract. Finally, every observation has to refer to the correct artifact and use evidence that makes sense for that modality.

Those boundaries fail independently. Treating them as one “JSON failed” result wastes the first hour of an incident. It also encourages retries, which are particularly dangerous here. A retry may return parseable output and turn a repeatable adapter bug into an intermittent test. The suite goes green while the original response, the only useful evidence, disappears.

Start with an application-owned contract. The examples below use a fictional but concrete endpoint, POST /v1/analyze. It accepts a request ID and a list of image or audio artifacts. The response contract is version 1 and contains exactly one observation for every supplied artifact. An image observation carries a normalized rectangle. An audio observation carries a millisecond range. These are not claims about a model vendor. They are rules chosen for this example service, and the tests enforce them at the service boundary. Every ID, confidence, coordinate, duration, endpoint, and schedule value in the snippets is illustrative. None is presented as a result measured in an experiment.

That distinction matters. “The model supports JSON” is not a testable product contract. “Our endpoint returns one bare JSON document with Content-Type application/json, schemaVersion 1, the original requestId, and one grounded observation per artifact” is testable. A team may choose different fields, but it needs the same precision. If the product allows partial coverage, replace the one-observation-per-artifact rule with an explicit completeness field or a documented omission reason. Do not let absence mean whatever the current response happens to imply.

Think of the oracle as five layers.

The transport layer checks the status and response media type before touching the body. A 502 HTML page from a gateway is not malformed model output. A 401 JSON error is not a schema regression. The Content-Type response header tells a client what media type was returned, so recording it gives you direct evidence about the HTTP boundary. It does not prove the body is valid, which is why the parser still runs next.

The syntax layer feeds the unmodified body to JSON.parse. That function throws a SyntaxError when the text does not follow JSON grammar. A markdown fence, an introductory sentence, a trailing comma, or a truncated object therefore fails if your consumer expects a bare JSON document. Resist the convenient cleanup function that searches for the first opening brace. It changes the tested contract and can accept prose containing a JSON-looking fragment that production rejects.

The shape layer checks names, types, required properties, allowed values, and the schema version. This catches confidence returned as the string "0.92", a missing observations array, or a time range represented in seconds when the contract requires milliseconds. Schema versioning belongs here because an old consumer should not silently interpret a new representation.

The grounding layer joins output back to input. Every artifactId must exist in the request manifest. Its declared modality must match the registered modality. Image coordinates must stay within the normalized image plane. Audio ranges must stay within the reviewed duration. This is where a structurally perfect hallucinated ID finally fails.

The content layer answers a harder question: is the claim itself supported by the pixels or sound? Deterministic checks can cover reviewed fixtures with known facts, such as a screenshot that contains one disabled “Pay now” button or a recording with silence in a fixed interval. Open-ended descriptions may need human review or a separately calibrated evaluator. Do not let a probabilistic content score override a deterministic unknown-artifact error. They measure different failures.

Consider a checkout screenshot registered as img-checkout and a call recording registered as aud-confirmation. A response cites img-checkout correctly but uses artifactId img-cart in its first observation. The JSON parses, every property has the expected type, and confidence sits between zero and one. The artifact registry has no img-cart entry, so the grounding layer reports an unknown reference. The code under test can make this assertion fail by returning the missing or correct ID. That is a real oracle, not a check against a fixture that already contains its own answer.

A second response cites aud-confirmation but declares modality image and supplies a rectangle. A generic shape check can distinguish image and audio variants, yet it cannot know the modality recorded in this specific request unless the test joins the output to that request. The useful message is not “object matched image variant.” It is “observations[1].modality is image, but aud-confirmation was registered as audio.” That wording points the investigation at ID association or response assembly.

A third response ends halfway through the observations array. The syntax layer should own it. If the HTTP status is 200 and Content-Type is application/json, inspect generation termination and adapter serialization. If the status is 504 with text/html, inspect the gateway or upstream timeout. The visible symptom may be the same unexpected-end parser error after a careless client ignores status, but the evidence separates two different owners.

Build an oracle that checks each failure layer

Keep parsing and contract inspection in one small helper, but return the layer with the issues. Save this example as tests/helpers/multimodal-contract.ts. It is intentionally strict. It rejects unknown keys because this example deploys consumers that deserialize a fixed version. A service designed for additive fields could permit unknown keys while still requiring schemaVersion. Choose that policy deliberately; strictness is not universally better.

The artifact manifest is trusted test setup, so duplicate input IDs and missing or non-positive audio durations throw immediately. Product output cannot be diagnosed against an ambiguous fixture. The response then passes through syntax and shape before grounding. This order prevents an invalid cast from producing a misleading “unknown artifact” message when observations was not an array at all.

export type Artifact = {
  id: string;
  modality: "image" | "audio";
  durationMs?: number;
};

type ImageObservation = {
  artifactId: string;
  modality: "image";
  claim: string;
  confidence: number;
  evidence: {
    region: { x: number; y: number; width: number; height: number };
  };
};

type AudioObservation = {
  artifactId: string;
  modality: "audio";
  claim: string;
  confidence: number;
  evidence: {
    timeRangeMs: { start: number; end: number };
  };
};

export type AnalysisOutput = {
  schemaVersion: "1";
  requestId: string;
  observations: Array<ImageObservation | AudioObservation>;
};

export type ContractResult =
  | { ok: true; value: AnalysisOutput }
  | {
      ok: false;
      layer: "syntax" | "shape" | "grounding";
      issues: string[];
    };

const isRecord = (value: unknown): value is Record<string, unknown> =>
  typeof value === "object" && value !== null && !Array.isArray(value);

const isFiniteNumber = (value: unknown): value is number =>
  typeof value === "number" && Number.isFinite(value);

function unexpectedKeys(
  value: Record<string, unknown>,
  allowed: readonly string[],
  path: string,
): string[] {
  const allowedSet = new Set(allowed);
  return Object.keys(value)
    .filter((key) => !allowedSet.has(key))
    .map((key) => path + "." + key + ": unexpected property");
}

export function inspectMultimodalOutput(
  rawBody: string,
  expectedRequestId: string,
  artifacts: readonly Artifact[],
): ContractResult {
  if (new Set(artifacts.map((artifact) => artifact.id)).size !== artifacts.length) {
    throw new Error("Test fixture contains duplicate artifact IDs");
  }
  for (const artifact of artifacts) {
    if (
      artifact.modality === "audio" &&
      (!isFiniteNumber(artifact.durationMs) || artifact.durationMs <= 0)
    ) {
      throw new Error("Audio fixture must have a positive duration: " + artifact.id);
    }
  }

  let candidate: unknown;
  try {
    candidate = JSON.parse(rawBody);
  } catch (error) {
    return {
      ok: false,
      layer: "syntax",
      issues: [error instanceof Error ? error.message : "JSON parsing failed"],
    };
  }

  if (!isRecord(candidate)) {
    return { ok: false, layer: "shape", issues: ["$: expected an object"] };
  }

  const shapeIssues = unexpectedKeys(
    candidate,
    ["schemaVersion", "requestId", "observations"],
    "$",
  );

  if (candidate.schemaVersion !== "1") {
    shapeIssues.push("$.schemaVersion: expected literal 1");
  }
  if (typeof candidate.requestId !== "string" || candidate.requestId.length === 0) {
    shapeIssues.push("$.requestId: expected a non-empty string");
  }
  if (!Array.isArray(candidate.observations)) {
    shapeIssues.push("$.observations: expected an array");
  } else {
    candidate.observations.forEach((observation, index) => {
      const path = "$.observations[" + index + "]";
      if (!isRecord(observation)) {
        shapeIssues.push(path + ": expected an object");
        return;
      }

      shapeIssues.push(
        ...unexpectedKeys(
          observation,
          ["artifactId", "modality", "claim", "confidence", "evidence"],
          path,
        ),
      );

      if (typeof observation.artifactId !== "string" || observation.artifactId.length === 0) {
        shapeIssues.push(path + ".artifactId: expected a non-empty string");
      }
      if (observation.modality !== "image" && observation.modality !== "audio") {
        shapeIssues.push(path + ".modality: expected image or audio");
      }
      if (typeof observation.claim !== "string" || observation.claim.trim().length === 0) {
        shapeIssues.push(path + ".claim: expected a non-empty string");
      }
      if (
        !isFiniteNumber(observation.confidence) ||
        observation.confidence < 0 ||
        observation.confidence > 1
      ) {
        shapeIssues.push(path + ".confidence: expected a number from 0 to 1");
      }
      if (!isRecord(observation.evidence)) {
        shapeIssues.push(path + ".evidence: expected an object");
        return;
      }

      if (observation.modality === "image") {
        shapeIssues.push(...unexpectedKeys(observation.evidence, ["region"], path + ".evidence"));
        const region = observation.evidence.region;
        if (
          !isRecord(region) ||
          !["x", "y", "width", "height"].every((key) => isFiniteNumber(region[key]))
        ) {
          shapeIssues.push(path + ".evidence.region: expected four numeric coordinates");
        } else {
          shapeIssues.push(
            ...unexpectedKeys(
              region,
              ["x", "y", "width", "height"],
              path + ".evidence.region",
            ),
          );
        }
      }

      if (observation.modality === "audio") {
        shapeIssues.push(
          ...unexpectedKeys(observation.evidence, ["timeRangeMs"], path + ".evidence"),
        );
        const range = observation.evidence.timeRangeMs;
        if (
          !isRecord(range) ||
          !isFiniteNumber(range.start) ||
          !isFiniteNumber(range.end)
        ) {
          shapeIssues.push(path + ".evidence.timeRangeMs: expected numeric start and end");
        } else {
          shapeIssues.push(
            ...unexpectedKeys(
              range,
              ["start", "end"],
              path + ".evidence.timeRangeMs",
            ),
          );
        }
      }
    });
  }

  if (shapeIssues.length > 0) {
    return { ok: false, layer: "shape", issues: shapeIssues };
  }

  const output = candidate as AnalysisOutput;
  const registry = new Map(artifacts.map((artifact) => [artifact.id, artifact]));
  const groundingIssues: string[] = [];

  if (output.requestId !== expectedRequestId) {
    groundingIssues.push(
      "$.requestId: expected " + expectedRequestId + ", received " + output.requestId,
    );
  }

  for (const [index, observation] of output.observations.entries()) {
    const path = "$.observations[" + index + "]";
    const artifact = registry.get(observation.artifactId);
    if (!artifact) {
      groundingIssues.push(path + ".artifactId: unknown artifact " + observation.artifactId);
      continue;
    }
    if (artifact.modality !== observation.modality) {
      groundingIssues.push(
        path +
          ".modality: returned " +
          observation.modality +
          ", but artifact " +
          artifact.id +
          " was registered as " +
          artifact.modality,
      );
      continue;
    }

    if (observation.modality === "image") {
      const { x, y, width, height } = observation.evidence.region;
      if (x < 0 || y < 0 || width <= 0 || height <= 0 || x + width > 1 || y + height > 1) {
        groundingIssues.push(path + ".evidence.region: rectangle is outside normalized bounds");
      }
    } else {
      const { start, end } = observation.evidence.timeRangeMs;
      if (
        artifact.durationMs === undefined ||
        start < 0 ||
        end <= start ||
        end > artifact.durationMs
      ) {
        groundingIssues.push(
          path + ".evidence.timeRangeMs: range is outside artifact duration",
        );
      }
    }
  }

  const referenceCounts = new Map<string, number>();
  for (const observation of output.observations) {
    referenceCounts.set(
      observation.artifactId,
      (referenceCounts.get(observation.artifactId) ?? 0) + 1,
    );
  }
  for (const artifact of artifacts) {
    const count = referenceCounts.get(artifact.id) ?? 0;
    if (count === 0) {
      groundingIssues.push("$.observations: missing artifact " + artifact.id);
    } else if (count > 1) {
      groundingIssues.push(
        "$.observations: expected one reference to " +
          artifact.id +
          ", received " +
          count,
      );
    }
  }

  if (groundingIssues.length > 0) {
    return { ok: false, layer: "grounding", issues: groundingIssues };
  }
  return { ok: true, value: output };
}

This helper has several ways to fail when product behavior changes. Rename schemaVersion, emit confidence as text, cite an absent ID, swap modalities, return an oversized rectangle, move an audio end point beyond its duration, or omit one requested artifact. None of the expected values are derived from the output being judged. The registry comes from the request fixture, and the boundaries come from the application contract.

Test the oracle itself before trusting it against a live service. A happy-path-only helper can contain the same blind spot as the product. Save the following file beside the helper as tests/helpers/multimodal-contract.spec.ts. Its valid baseline and numeric values are illustrative. The first controlled mutation stays structurally valid but creates an unknown reference. The second keeps the artifact ID valid but crosses the audio duration. Each assertion checks both ownership and diagnostic detail.

import { expect, test } from "@playwright/test";
import {
  type AnalysisOutput,
  type Artifact,
  inspectMultimodalOutput,
} from "./multimodal-contract";

const artifacts: Artifact[] = [
  { id: "img-checkout", modality: "image" },
  { id: "aud-confirmation", modality: "audio", durationMs: 4_800 },
];

const baseline: AnalysisOutput = {
  schemaVersion: "1",
  requestId: "case-1042",
  observations: [
    {
      artifactId: "img-checkout",
      modality: "image",
      claim: "The Pay now button is disabled",
      confidence: 0.91,
      evidence: { region: { x: 0.61, y: 0.72, width: 0.22, height: 0.09 } },
    },
    {
      artifactId: "aud-confirmation",
      modality: "audio",
      claim: "The speaker confirms the order number",
      confidence: 0.86,
      evidence: { timeRangeMs: { start: 1_250, end: 2_900 } },
    },
  ],
};

test("rejects a structurally valid observation that cites an unknown image", () => {
  const changed: AnalysisOutput = {
    ...baseline,
    observations: [
      { ...baseline.observations[0], artifactId: "img-missing" },
      baseline.observations[1],
    ],
  };

  const result = inspectMultimodalOutput(
    JSON.stringify(changed),
    baseline.requestId,
    artifacts,
  );

  expect(result.ok).toBe(false);
  if (result.ok) throw new Error("Expected the contract to fail");
  expect(result.layer).toBe("grounding");
  expect(result.issues).toContain(
    "$.observations[0].artifactId: unknown artifact img-missing",
  );
});

test("rejects an audio range that exceeds the reviewed duration", () => {
  const audio = baseline.observations[1];
  if (audio.modality !== "audio") throw new Error("Fixture is not audio");
  const changed: AnalysisOutput = {
    ...baseline,
    observations: [
      baseline.observations[0],
      { ...audio, evidence: { timeRangeMs: { start: 4_200, end: 5_100 } } },
    ],
  };

  const result = inspectMultimodalOutput(
    JSON.stringify(changed),
    baseline.requestId,
    artifacts,
  );

  expect(result.ok).toBe(false);
  if (result.ok) throw new Error("Expected the contract to fail");
  expect(result.layer).toBe("grounding");
  expect(result.issues).toContain(
    "$.observations[1].evidence.timeRangeMs: range is outside artifact duration",
  );
});

Add mutation tests for every rule that can block a release. You do not need one test for every typo in a JSON property, but you do need proof that each branch is reachable. Flip valid to invalid and invalid back to valid. If a test still passes both ways, its oracle is not observing the mutation.

The content layer needs a different fixture style. For the checkout image, reviewers can record that the expected UI state is disabled and that the relevant control lies inside an approved coarse region. The test can assert the canonical label chosen by your application and overlap with that region. It should not assert a generated sentence character for character. For the recording, the fixture can identify an interval containing the confirmation. A claimed interval that overlaps no approved interval is a content failure even if it stays within the file duration.

Keep those reviewed annotations outside the response fixture. Copying the model’s previous output into expected.json and then comparing a new output to it only freezes one sample. A proper oracle derives expected facts from a separately reviewed annotation file with its own owner and revision. This independence is what lets a product regression change the actual value without silently changing the expected value at the same time.

Work through failures that look alike

Malformed output and wrong response handling often collapse into the same exception. Suppose the test prints “Unexpected token '<'” while parsing. That can mean the model adapter returned an HTML gateway page. It can also mean an authentication redirect produced a login document, or an application error handler rendered HTML. The first bytes of the body identify the symptom, while status, final URL, and Content-Type identify the boundary. Do not label it a model formatting defect until those transport fields agree with the expected endpoint contract.

Playwright’s APIResponse exposes status, headers, URL, text, and the response body as a buffer. Read the buffer once, hash those exact bytes, decode it for the contract parser, and attach only a policy-safe diagnostic record. APIResponse also has a json method, and the official API reference states that it throws when the body cannot be parsed with JSON.parse. Reading the buffer directly lets this test retain an exact fingerprint and its own failure-layer classification without putting response contents in the report.

The test below drives the application endpoint through Playwright’s built-in request fixture. When data is an object, APIRequestContext serializes it as JSON and sets application/json unless the header is explicitly set. Save it as tests/multimodal-output.spec.ts after adding two independently reviewed fixtures at the named paths. The sample reads local files, not a public URL that could change. The API token, media bytes, and response contents are omitted from the attachment. The diagnostic retains IDs, hashes, byte length, and a coarse prefix class instead.

import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { expect, test } from "@playwright/test";
import {
  type Artifact,
  inspectMultimodalOutput,
} from "./helpers/multimodal-contract";

const serviceUrl = process.env.MULTIMODAL_BASE_URL;
const apiToken = process.env.MULTIMODAL_API_TOKEN;

function loadFixture(path: string): { contentBase64: string; sha256: string } {
  const bytes = readFileSync(join(process.cwd(), path));
  return {
    contentBase64: bytes.toString("base64"),
    sha256: createHash("sha256").update(bytes).digest("hex"),
  };
}

function classifyResponsePrefix(rawBody: string): string {
  const prefix = rawBody.trimStart().slice(0, 16).toLowerCase();
  if (prefix.length === 0) return "empty";
  if (prefix.startsWith("<!doctype") || prefix.startsWith("<html")) {
    return "html-like";
  }
  if (prefix.startsWith("```")) return "markdown-fence";
  if (prefix.startsWith("{") || prefix.startsWith("[")) return "json-like";
  return "other";
}

function classifyResponseLocation(
  actualUrl: string,
  expectedUrl: string,
): { sameOrigin: boolean; samePath: boolean; sameQuery: boolean } {
  const actual = new URL(actualUrl);
  const expected = new URL(expectedUrl);
  return {
    sameOrigin: actual.origin === expected.origin,
    samePath: actual.pathname === expected.pathname,
    sameQuery: actual.search === expected.search,
  };
}

test("returns one grounded observation for each submitted artifact", async (
  { request },
  testInfo,
) => {
  expect(serviceUrl, "Set MULTIMODAL_BASE_URL to the adapter origin").toBeTruthy();
  expect(apiToken, "Set MULTIMODAL_API_TOKEN for the test account").toBeTruthy();

  const image = loadFixture("tests/fixtures/checkout-disabled.png");
  const audio = loadFixture("tests/fixtures/order-confirmation.wav");
  const requestId = "checkout-disabled-with-confirmation";
  const artifacts: Artifact[] = [
    { id: "img-checkout", modality: "image" },
    { id: "aud-confirmation", modality: "audio", durationMs: 4_800 },
  ];

  const endpointUrl = new URL("/v1/analyze", serviceUrl as string).toString();
  const response = await request.post(endpointUrl, {
    headers: {
      accept: "application/json",
      authorization: "Bearer " + apiToken,
    },
    data: {
      requestId,
      contractVersion: "1",
      task: "Return one observation for each supplied artifact.",
      artifacts: [
        {
          id: artifacts[0].id,
          modality: artifacts[0].modality,
          mediaType: "image/png",
          contentBase64: image.contentBase64,
        },
        {
          id: artifacts[1].id,
          modality: artifacts[1].modality,
          mediaType: "audio/wav",
          durationMs: artifacts[1].durationMs,
          contentBase64: audio.contentBase64,
        },
      ],
    },
  });

  const rawBytes = await response.body();
  const rawBody = rawBytes.toString("utf8");
  const contentType = response.headers()["content-type"] ?? "";
  const responseLocation = classifyResponseLocation(response.url(), endpointUrl);
  await testInfo.attach("multimodal-response.json", {
    body: JSON.stringify(
      {
        requestId,
        requestArtifacts: [
          { id: artifacts[0].id, sha256: image.sha256 },
          { id: artifacts[1].id, sha256: audio.sha256 },
        ],
        response: {
          status: response.status(),
          location: responseLocation,
          contentType,
          bodySha256: createHash("sha256").update(rawBytes).digest("hex"),
          bodyBytes: rawBytes.length,
          prefixClass: classifyResponsePrefix(rawBody),
        },
      },
      null,
      2,
    ),
    contentType: "application/json",
  });

  expect(response.status()).toBe(200);
  expect(responseLocation).toEqual({
    sameOrigin: true,
    samePath: true,
    sameQuery: true,
  });
  expect(contentType.split(";", 1)[0].trim().toLowerCase()).toBe(
    "application/json",
  );

  const result = inspectMultimodalOutput(rawBody, requestId, artifacts);
  if (!result.ok) {
    throw new Error(result.layer + " contract failed:\n" + result.issues.join("\n"));
  }
});

This is a live adapter test, so it requires the two named fixture files and a test account. That is still runnable code, not pseudocode. The endpoint names belong to the explicit example contract and should be replaced as a unit when adapting it. Do not copy only the assertion while leaving your production request shape implicit.

The attachment solves a common diagnostic failure. When the contract helper throws, the report still shows which response status, media type, location comparison, request IDs, body hash, body size, and prefix class produced the issue. Playwright’s testInfo.attach accepts either a body or a file path and makes the attachment available to reporters that display attachments. No response text or raw URL enters this report, so an unexpected HTML page, redirect query, or verbose upstream error cannot leak through those fields.

Redaction must happen before attachment. This example takes the safer route and omits response content entirely. Hashing fixture bytes identifies the reviewed input without embedding a customer screenshot or call recording in CI. A hash is not a universal anonymization technique; low-entropy or known files may still be recognizable by comparison. Use synthetic or specifically approved fixtures for routine CI, and apply your organization’s data retention rules to failures.

Another near-miss appears when artifact grounding passes but the prompt was paired with the wrong bytes. If img-checkout refers to a different screenshot than the manifest claims, every returned reference can be internally consistent. Compare the recorded SHA-256 digest with the reviewed fixture digest. A mismatch before the request leaves the test runner is setup failure. A matching local digest plus a different digest in adapter telemetry points at upload or storage. A matching digest end to end, followed by a false claim, is finally evidence about analysis quality.

Coordinate conventions create their own look-alike. A box with x 120 and width 300 may be valid pixel coordinates but invalid under a normalized zero-to-one contract. That is not necessarily a perception failure. Check the schema version and coordinate unit first. If a migration changed from pixels to normalized values without changing the version, the producer and consumer contract is broken. If both sides agree on normalized coordinates and the box lies outside the image, the grounding check is correct to fail.

Audio has the same trap with time units. A range ending at 5,100 could mean milliseconds or seconds. The type number cannot distinguish them. Put the unit in the property name, retain the reviewed media duration, and reject ranges that exceed it. If the source file was transcoded and its duration changed, the manifest is stale. If the manifest matches the uploaded media and only output crosses the end, the response is wrong.

Collect evidence before adding retries

Run the failing case alone with the same fixture bytes, adapter version, contract version, and authentication path. For a Playwright test named as above, a useful local command is:

MULTIMODAL_BASE_URL="https://qa-ai.example.test" \
MULTIMODAL_API_TOKEN="$QA_AI_TOKEN" \
pnpm exec playwright test tests/multimodal-output.spec.ts \
  --grep "returns one grounded observation" \
  --reporter=line,html \
  --trace=on

The hostname is plainly an example, and QA_AI_TOKEN is a shell variable supplied by the operator. The command does not invent a framework flag. Playwright documents the test command, grep filtering, reporters, and trace option. Use your real QA origin and credential source.

Open the HTML report first because the test adds a purpose-built multimodal-response.json attachment. Check status and Content-Type before requesting access to any retained response body. Confirm that all three location booleans are true; a false value establishes a redirect boundary without exposing its destination. Match the requestId and artifact hashes to the approved fixture record. Then read the prefix class, contract layer, and issue paths. That sequence tells you whether to contact platform, adapter, contract, or model-quality owners.

The trace is supporting evidence, not the sole record. In the API action, confirm that the failing call ran and correlate it with the test step and timing. Use the explicit attachment for the response fingerprint and artifact manifest because your test controls exactly what it contains. Treat a trace as sensitive because request metadata and payloads may contain credentials or media. Disable tracing where those values must not be retained, or use a restricted test account and approved synthetic fixtures under the team’s retention policy.

Parser errors need the raw prefix and body length. A body beginning with three backticks and json contains fenced output. A body ending inside a quoted claim is incomplete, with truncation and interrupted serialization among the possible causes. A body beginning with an HTML doctype points away from the structured-output contract. Do not assert the exact JavaScript engine wording of the SyntaxError; engines and versions can phrase parse failures differently. Assert the syntax layer and retain the body evidence.

Shape failures need paths and actual types. “Schema invalid” forces a developer to reproduce the entire call. “$.observations[0].confidence expected a number from 0 to 1” identifies the relevant field without copying its value. When deeper inspection is authorized and a restricted service log retains bodies, retrieve the matching one by its hash. A useful diagnostic reveals the broken contract without creating a second data leak.

Grounding failures need the request manifest beside the response. For an unknown ID, list the allowed IDs. For a modality mismatch, print both sides, since only the pair identifies the defect: the grounding check above emits $.observations[1].modality: returned image, but artifact aud-confirmation was registered as audio. Printing the registered side alone would tell a reader what the manifest says and leave them to reopen the response to learn what the model claimed, which is the lookup the diagnostic exists to save. For a range error, include the approved duration and returned start and end. For a missing artifact, state which request artifact received no observation. Those facts are deterministic and safe to compare even when claim wording varies.

Content failures require reviewed expectations. If a screenshot annotation says the payment button is disabled, save the annotation revision and reviewer. If an audio fixture identifies a spoken order number, store the expected normalized value separately from the media transcript generated by the same system under test. The test must not ask one model output to validate another output from the identical path and call that independence.

Retries answer a narrow operational question: does the same case pass on another attempt? They do not repair the first failure. Keep the first attempt’s evidence and report retry status separately. A parse failure followed by a valid response suggests nondeterminism or an intermittent boundary, while two identical unknown IDs suggest a repeatable association defect. Both deserve different triage even if the test runner’s final display can mark a retried test as passed.

Avoid averages for binary contract rules. Ninety-nine valid artifact references do not cancel one reference to data that was never supplied. Aggregate rates help monitor a live canary, but each syntax, shape, version, and unknown-reference failure remains a failed row. Content scores can be summarized only after the deterministic contract has admitted the sample.

Also separate latency from truncation. A client timeout is observed at the test runner. A complete HTTP response containing truncated JSON is observed at the response boundary. A server may generate a partial body for many reasons, so do not claim a specific cause from the parser error. Correlate with server-side termination metadata that your adapter actually records. If no such field exists, say “partial response received” and stop there.

Roll the contract into an existing suite

Do not switch hundreds of cases to a strict new validator in one pull request. Start by inventorying the consumers. A UI may tolerate unknown additive fields while a typed data pipeline rejects them. A batch export may require stable ordering even though the interactive application does not. One contract cannot honestly represent all three if their compatibility rules differ.

Freeze a version 1 example set with a small number of reviewed artifacts. Include one normal image, one image where the expected object is absent, one short audio clip with a known event, and one audio clip without that event. Add a mixed image-and-audio request because single-modality tests cannot expose cross-modality ID swaps. Every fixture needs provenance, permitted use, expected duration or dimensions, and an owner who can review changes.

Introduce the validator in observation mode first. Run it against existing responses and store issue categories without blocking. Observation mode should have an expiry date and a named review. Otherwise it becomes a permanent dashboard that everyone assumes someone else reads. Sample the failures manually to separate legacy-but-accepted behavior from genuine defects and fixture mistakes.

Next, block only transport, syntax, and required-version failures on deterministic adapter tests. These rules should be stable because the local adapter or stub controls the response. Add unknown artifact references and modality mismatches once request manifests are reliable. Add coordinate and timestamp bounds after every producer agrees on units. Content assertions come last, one reviewed fixture at a time.

Keep live-model cases out of the fastest pull-request lane unless your team accepts their cost, latency, credential needs, and availability risk. Pull requests can exercise the parser, validator, response assembler, and error mapping against controlled fixtures. A scheduled or manually triggered canary can call the deployed model adapter with approved media. The two lanes answer different questions and should have different release authority.

Here is a GitHub Actions job for that live canary. It installs pnpm before asking setup-node to enable pnpm caching, which avoids invoking a package manager that is not yet on PATH. It runs only on a schedule or manual dispatch, requires the two secrets at job runtime, and uploads the Playwright report even when the test fails.

name: Multimodal contract canary

on:
  workflow_dispatch:
  schedule:
    - cron: "17 3 * * 1-5"

jobs:
  live-contract:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7

      - uses: pnpm/action-setup@v6
        with:
          version: 10

      - uses: actions/setup-node@v7
        with:
          node-version: 22
          cache: pnpm

      - run: pnpm install --frozen-lockfile

      - name: Run the approved live fixture
        run: >
          pnpm exec playwright test
          tests/multimodal-output.spec.ts
          --reporter line,html
          --trace retain-on-failure
        env:
          MULTIMODAL_BASE_URL: ${{ secrets.MULTIMODAL_BASE_URL }}
          MULTIMODAL_API_TOKEN: ${{ secrets.MULTIMODAL_API_TOKEN }}

      - name: Upload diagnostics
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: multimodal-contract-report
          path: playwright-report/
          if-no-files-found: ignore
          retention-days: 7

The schedule is an example policy, not a claimed best frequency. Pick a cadence from release risk and vendor spend. The timeout and retention values are configuration choices, not measurements. If media or response bodies cannot be retained for seven days, shorten retention or change the attachment content before enabling uploads.

During rollout, publish counts by layer rather than one success percentage. Syntax failures usually go to response generation or adapter serialization. Shape and version failures go to contract owners. Unknown IDs and modality mismatches go to request-response association owners. Content disagreements go to fixture reviewers or model-quality owners only after deterministic checks pass. This routing cuts the temptation to assign every red test to “AI flakiness.”

Version changes need dual-read tests. When introducing schema version 2, keep version 1 fixtures and consumer tests while the migration is active. Have the producer select a version through a documented request field or endpoint behavior that your service actually implements. Do not silently change expected fields based on deployment date. Remove version 1 only after telemetry or repository evidence shows that no supported consumer requests it.

The cost of strict contracts is maintenance. Adding a required observation field breaks old producers and fixtures. Rejecting unknown fields slows additive rollout. Hashing and attaching evidence adds code and storage. Live canaries consume paid model calls and can fail because the external service is unavailable. Reviewed multimodal fixtures need privacy review and human upkeep. These are acceptable costs when the structured output drives automation, but they must appear in the test plan and ownership model.

There is also a coverage cost. Two clean fixtures do not prove performance across camera conditions, accents, languages, file encodings, or long recordings. Contract tests protect integration invariants. They do not replace a representative evaluation dataset. Keep the release claim narrow: “These supported payloads preserve schema and grounding invariants,” not “the multimodal model is correct.”

Choose assertions that survive harmless variation

Generated wording, observation order, and confidence values may vary even when the user-visible decision is acceptable. Snapshotting the entire response turns those harmless differences into noise. Once a team stops reading snapshot diffs, a meaningful artifact change can hide inside a wall of rewritten claims.

Assert exact values where the contract promises exactness. schemaVersion, requestId, artifactId membership, modality association, required coverage, units, and bounds are good candidates. Assert a canonical content label only when the application defines that label and the fixture has an independent expected value. Treat free-form claim prose as diagnostic unless its exact text is itself a product requirement.

Order deserves an explicit rule. If consumers treat observations as an unordered set, sort stable identifiers in the test before comparison. If rank communicates importance, preserve order and assert the ranking rule against reviewed cases. Do not sort merely to make a failure disappear. That would change product semantics inside the test.

Confidence needs similar discipline. A range check from zero to one validates representation, not calibration. A threshold such as 0.8 is a product decision only if the application uses it. Testing that every fixture exceeds an arbitrary threshold teaches the suite to reward self-reported certainty. Calibration requires labeled examples and analysis across them; one response cannot prove it.

For content, prefer relations over prose equality. An image box should overlap the approved target region by a documented rule. An audio time range should intersect the reviewed event window. A normalized identifier should equal the independently annotated value. Each relation can fail because product output changed, and each expected value comes from outside that output.

Metamorphic cases can extend coverage without inventing a second oracle. Pair an image with a copy that changes only irrelevant metadata and verify that artifact association remains correct. Add silence before an audio event and check that the returned timestamp shifts by the inserted duration if your product contract promises timestamp alignment. Swap the order of two artifacts and assert that IDs, not array positions, preserve association. These tests need transformations you control exactly; do not assume a model must preserve subjective wording.

One valuable negative case removes an artifact while leaving the prompt text unchanged. If the response still cites the removed ID, the grounding oracle catches it. Another replaces the audio clip but preserves its filename. If the returned claim remains identical despite independently different reviewed content, investigate caching or request assembly, but do not declare the cause from that observation alone. Check hashes and adapter logs before assigning ownership.

Property-based generation can hammer the pure validator with malformed shapes, strange numeric values, duplicate observations, and unfamiliar IDs. Keep media understanding out of that generator unless you can derive the expected content. Random bytes are useful for input validation, not for claiming perceptual quality. Property-Based Evals for Structured LLM JSON Outputs covers the neighboring contract technique, while Structured LLM Output Evaluation Against JSON Contracts goes deeper on schema-only output.

When a failure occurs, report the narrowest broken invariant. “Image box exceeds normalized bounds” is stronger than “vision response bad.” “Response references aud-confirmation as image” is stronger than “multimodal confusion.” Precise language protects the investigation from confident guesses and gives developers a reproducible target.

Know when strict structured-output checks are the wrong tool

Do not impose this contract on a feature whose product interface is deliberately free-form text. Wrapping every answer in JSON creates complexity without improving a consumer that only displays prose. Test the user-visible behavior, safety rules, and relevant content qualities instead.

Avoid exact field or full-response snapshots for exploratory assistants. If the product permits several valid descriptions and no downstream automation depends on their wording, a rigid golden response will mostly measure sampling variation. Keep transport and safety checks, then use reviewed content evaluation suited to the decision.

Do not require one observation per artifact when omission is valid. A request may contain context images that need no individual description, or a silent audio track that should produce no event. In that design, add an explicit processedArtifacts list, per-artifact status, or omission reason. The important property is that absence is represented intentionally, not that every input generates a claim.

Skip live external calls in pull-request CI when credentials, cost, rate limits, or provider availability would make unrelated code changes wait on them. Exercise the adapter contract with deterministic responses in the pull-request lane. Run approved live cases on a cadence and with ownership that fits the risk.

Never upload production screenshots or customer calls merely to make a regression realistic. Synthetic media, licensed fixtures, and specifically approved incident samples are safer. If a defect can only be reproduced with sensitive media, use the controlled investigation environment and retain the minimum diagnostic record permitted by policy.

Finally, do not mistake grounding checks for proof that a claim is true. A response can cite the correct image ID and place a box inside the image while labeling the wrong object. The deterministic contract has done its job: it proved the response is parseable, structurally compatible, and associated with supplied evidence. A separate reviewed content oracle must decide whether the claim matches that evidence.