Your API errors are part of the contract

A client sends an expired token and receives a 200 response containing {"error":"Unauthorized"}. Another endpoint returns a 401 with an HTML page from the gateway. Both calls failed, but only one even resembles the JSON contract the SDK expects.

Find the boundary where the error changes shape

An error response has at least three observable layers: the HTTP status, representation metadata such as Content-Type, and the body. A reliable test checks all three on the wire. If it starts by parsing JSON, it can miss the protocol mistake that caused the client to take the wrong branch.

Most error-contract bugs do not begin in the final serializer. They appear when several layers each believe another layer owns the response.

A handler may throw a domain exception. Middleware maps it to a status and JSON object. An authentication filter can reject the request before the handler runs. A gateway can replace a timeout with its own response. A proxy can add or remove headers. Finally, a client library may parse JSON, expose text, or discard the body based on its configuration.

Write down the path for each error family. "Validation errors come from application middleware" is different from "missing credentials are rejected by the edge." If the edge owns authentication failures, a component test that calls the handler directly cannot prove the deployed 401 contract.

The wire status matters because generic HTTP software does not need to understand your JSON. A success status with an error object can be cached, retried, logged, or counted as success by infrastructure. The body cannot repair that false protocol signal. Conversely, a correct 404 with an undocumented HTML body may satisfy an HTTP-level check while breaking a generated client that expects JSON.

Media type is the next discriminator. application/problem+json tells the client that the body uses the problem-details JSON format. A body that happens to contain type and title but is labeled text/html is not the same representation. A plain application/json error can be a valid product contract, but the API description must say so. Tests should enforce the contract you published, not silently reinterpret a near match.

RFC 9457 defines a standard problem-details object for HTTP APIs. Its core members have distinct jobs:

• type identifies the problem type. When omitted, the specification treats it as about:blank.
• title is a short human-readable summary of the type and should remain stable across occurrences except for localization.
• status repeats the HTTP status as an advisory value. When a generator includes it, the specification requires it to match the actual response status.
• detail explains this occurrence to a human. Clients should not parse it to discover structured facts.
• instance identifies this occurrence, when the API chooses to provide one.
• Extension members carry problem-specific data, such as a stable application code or field errors.

Not every member is mandatory in every problem object. Your API can define a narrower house contract, but label it as your contract. For example, requiring an absolute HTTPS type URI, a code extension, and a correlation-oriented instance is a product decision layered on the RFC.

The status in the body must not become the client's only status. Intermediaries can alter a response, and persisted problem objects may be viewed without the original HTTP message. Preserve both values in test evidence. If they disagree, show both in the failure instead of normalizing one away.

Human text is a poor automation key. Copy edits, localization, and more useful detail can change it. A mobile app that branches on detail == "Card already used" has coupled behavior to prose. Give it a stable problem type or machine code instead, then test the text for safety and usefulness at a less brittle level.

Define one contract before multiplying test cases

Start with a small registry of problem types rather than a generic "error" schema that accepts any object. Each entry should answer four questions: which status applies, which stable identifier represents the condition, which extensions clients may rely on, and which data must never appear.

Here is an OpenAPI component for one validation problem. It deliberately allows extra properties because RFC 9457 supports extension members and the API may add non-breaking diagnostic fields. The required list is this example API's decision, not a universal rule from the RFC.

openapi: 3.1.0
info:
  title: Orders API
  version: 1.4.0
paths:
  /orders:
    post:
      responses:
        "201":
          description: Order created
        "422":
          description: Request validation failed
          content:
            application/problem+json:
              schema:
                $ref: "#/components/schemas/ValidationProblem"
components:
  schemas:
    ValidationProblem:
      type: object
      required:
        - type
        - title
        - status
        - code
        - errors
      properties:
        type:
          type: string
          format: uri
          const: "https://api.example.test/problems/validation"
        title:
          type: string
        status:
          type: integer
          const: 422
        detail:
          type: string
        instance:
          type: string
          format: uri-reference
        code:
          type: string
          const: "ORDER_VALIDATION_FAILED"
        errors:
          type: array
          minItems: 1
          items:
            type: object
            required: [pointer, reason]
            properties:
              pointer:
                type: string
              reason:
                type: string
            additionalProperties: false
      additionalProperties: true

A schema can look convincing in review while never being applied. Resolve every $ref during CI, then run negative examples that must fail validation. If a payload without code passes, either the reference is broken or the validator is not exercising the expected schema.

The first runnable test should use the deployed HTTP boundary. Node's built-in test runner and fetch API are enough for a focused wire check. This helper intentionally validates the house rules shown above.

import assert from "node:assert/strict";
import test from "node:test";

type Problem = {
  type: string;
  title: string;
  status: number;
  detail?: string;
  instance?: string;
  code: string;
  errors?: Array<{ pointer: string; reason: string }>;
};

async function readProblem(response: Response): Promise<Problem> {
  const contentType = response.headers.get("content-type") ?? "";
  assert.match(
    contentType,
    /^application\/problem\+json(?:\s*;|$)/i,
    "wrong error media type",
  );

  const raw = await response.text();
  let parsed: unknown;

  try {
    parsed = JSON.parse(raw);
  } catch {
    throw new Error("error body is not valid JSON");
  }

  assert.equal(typeof parsed, "object");
  assert.notEqual(parsed, null);

  const problem = parsed as Problem;
  assert.equal(problem.status, response.status);
  assert.match(problem.type, /^https:\/\//);
  assert.equal(typeof problem.title, "string");
  assert.notEqual(problem.title.trim(), "");
  assert.equal(typeof problem.code, "string");

  return problem;
}

test("missing quantity returns the documented validation problem", async () => {
  const baseUrl = process.env.API_BASE_URL;
  assert.ok(baseUrl, "API_BASE_URL is required");

  const response = await fetch(baseUrl + "/orders", {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ sku: "CHAIR-1" }),
  });

  assert.equal(response.status, 422);
  const problem = await readProblem(response);
  assert.equal(
    problem.type,
    "https://api.example.test/problems/validation",
  );
  assert.equal(problem.code, "ORDER_VALIDATION_FAILED");
  assert.deepEqual(problem.errors, [
    { pointer: "/quantity", reason: "required" },
  ]);
});

The exact field pointer and reason are stable only if the API promises them. If the product allows several validation rules to fire together, compare the entries by pointer rather than assuming array order. If the reason is localized prose, assert the pointer and a machine rule code instead.

Test data should select one cause at a time. A request missing both authentication and a required field may correctly fail authentication before validation. That case cannot prove the validation contract. Use a valid credential with an invalid body for 422, an invalid credential with an otherwise valid body for 401, and a valid request against a missing resource for 404.

For every registry entry, include one positive contract case and one mutation that must be rejected. Useful mutations include a success status, missing media type, absent code, wrong field type, mismatched body status, and a prohibited stack trace. This negative half proves the assertion can detect the defect it claims to cover.

Work through three failures from request to evidence

Worked example one: validation middleware returns 200.

The request body omits quantity. The response body contains a sensible error object, but the status is 200. A JSON-only assertion sees code == "ORDER_VALIDATION_FAILED" and passes. The frontend's fetch wrapper sees a successful response and tries to render an order.

The diagnostic is the first HTTP line, not the prettified body. Capture it with the headers and raw payload:

curl --silent --show-error \
  --dump-header /tmp/order-error.headers \
  --output /tmp/order-error.body \
  --request POST \
  --header 'content-type: application/json' \
  --data '{"sku":"CHAIR-1"}' \
  "$API_BASE_URL/orders"

sed -n '1,20p' /tmp/order-error.headers
sed -n '1,80p' /tmp/order-error.body

In a real CI script, fail if API_BASE_URL is empty and create temporary paths safely. The command is for diagnosis, not a claim about a particular service's output.

If the status line is 200 before the response reaches the gateway, inspect the application's exception mapper or handler return path. If the service logs 422 but the wire shows 200, compare gateway access logs and route configuration. Do not change the test to accept either status. A union of 200 and 422 erases the behavior clients need.

The fix may be a shared mapper that returns the documented status and body. Its trade-off is coupling: every service using the mapper must coordinate breaking changes. Version the problem type or schema when clients need a transition, and test a representative endpoint in each deployment path.

Worked example two: the gateway returns HTML for an expired credential.

The application contract describes a JSON 401, but the gateway performs token validation first. When the token expires, the request never reaches the service. The gateway emits an HTML login page or a vendor-specific JSON shape.

The signature is clear. Application logs contain no matching request, the Content-Type is text/html, and the first raw bytes begin with markup. A JSON parser error is only the symptom. Save the raw bytes before the test framework replaces them with "Unexpected token <".

Run the request through two paths if your architecture permits it: the deployed gateway URL and an internal service URL in a controlled environment. A valid service response plus an invalid edge response assigns the defect to the edge contract. An invalid response on both paths points to shared middleware or duplicated configuration.

The preferred fix is not always "make the gateway emit the application schema." Some gateways cannot safely or consistently do that. Another valid design is to document an edge-owned authentication problem type and teach clients to handle both documented families. The cost is a broader client contract and more integration tests.

Worked example three: a dependency failure leaks an internal exception.

An inventory dependency times out. The order endpoint responds with 503, which looks reasonable, but detail contains a database host, package path, or stack frame. The schema passes because detail is a string. The release still should not.

Add a focused leak check to representative 5xx tests. Pattern matching cannot prove absence of every secret, so combine it with an allowlist for public fields and review the raw artifact under restricted retention.

import assert from "node:assert/strict";

const prohibited = [
  /\b(?:password|secret|api[_-]?key)\s*[=:]/i,
  /\bat\s+\S+\s+\([^)]*:\d+:\d+\)/,
  /\b(?:postgres|mysql):\/\/\S+/i,
  /\/(?:home|Users|srv|app)\/\S+\.(?:js|ts|java|py):\d+/,
];

export function assertNoInternalDetails(rawBody: string): void {
  for (const pattern of prohibited) {
    assert.doesNotMatch(
      rawBody,
      pattern,
      "error response exposes an internal detail",
    );
  }
}

const safeBody = JSON.stringify({
  type: "https://api.example.test/problems/dependency-unavailable",
  title: "A required service is unavailable",
  status: 503,
  code: "INVENTORY_UNAVAILABLE",
  detail: "Try the request again later.",
});

assertNoInternalDetails(safeBody);

These patterns are a guardrail, not a universal secret detector. They can produce false positives and miss formats you have not listed. Seed known leak strings in negative tests to prove each rule fires. Add patterns from confirmed incidents, and avoid printing the prohibited value again in the assertion report.

A near-miss looks similar: the 503 body is safe, but it carries Retry-After: 0 or no retry guidance even though clients rely on it. That is a resilience-contract issue, not a payload-schema issue. Test retry headers only where the problem definition promises them. RFC 9457 allows a problem type definition to specify Retry-After; it does not require the header on every problem.

Separate application defects from edge and parser defects

When the failure report says "invalid error JSON," retrieve the raw response. Test frameworks often show only the parsing exception. You need the status line, all response headers, and the unparsed body to identify ownership.

An empty body can be legitimate for some statuses or methods, but it is a contract failure if this endpoint documents a problem representation. Check content-length and transfer behavior, then inspect whether an intermediary removed the body. A service log that includes a complete object does not prove those bytes reached the client.

Compression adds another near-miss. If manual curl output looks like binary noise, check Content-Encoding and whether the diagnostic command requested decompression. Do not report malformed JSON until the representation has been decoded according to its headers. Your normal HTTP client may decode automatically while a raw packet capture does not.

Parser configuration can also create false evidence. A client may coerce dates, drop unknown fields, or deserialize a number into a narrower type. Preserve the response text next to the parsed object. When they disagree, the network contract may be correct and the client model wrong.

Use a request identifier only if the deployed system exposes one. Do not invent a required traceId because it seems useful. If the contract offers instance or a request-ID header, capture it and correlate it with sanitized logs. If it does not, rely on test-owned case IDs, timestamps, and isolated data.

Authentication and authorization deserve separate cases. A missing credential may produce 401 and an authentication challenge. An authenticated user lacking permission may produce 403. The exact choice depends on the resource and contract, so assert the published behavior. Avoid a blanket rule that every security failure must look identical, since uniformity can erase meaningful HTTP semantics.

Not-found responses can also have two owners. A route miss at the gateway and a missing domain entity inside the service may both return 404. Give them distinct problem types if clients act differently. If clients do not need the distinction, keep one external type and preserve internal ownership only in logs.

The strongest diagnostic matrix crosses failure family with entry path:

The entries are illustrative outcomes, not measurements. Replace them with evidence from your architecture. The value of the matrix is that it stops the team from debugging a handler for a response the handler never created.

Migrate the contract without freezing every word

A new error format is a client migration, even when every endpoint already returns JSON. Inventory the current shapes before introducing one shared schema. Group them by actual client behavior, not by visual similarity.

Start with one error family that has a clear owner, usually request validation. Publish its schema and stable identifiers. Add producer tests and one consumer test. Run the checks in report-only mode across other endpoints so undocumented variants become visible without blocking every deployment.

Keep message assertions loose during rollout. Require a non-empty title and safe detail, but branch client behavior on type or a documented extension. If localized messages are supported, run at least one alternate locale and verify the identifier remains stable while text changes.

For existing clients, a versioned field or negotiated media type may be safer than changing an object in place. Dual formats cost maintenance and can become permanent. Set an owner, an observation period, and a removal condition before adding them.

Schema strictness has a trade-off. additionalProperties: false catches accidental fields and typos quickly, but it makes adding a harmless extension a breaking change. Allowing additional properties supports evolution, but it will not catch statsu beside a missing status unless status is required. A practical contract requires core fields, restricts sensitive nested structures, and allows documented extension growth where clients ignore unknown values.

Wire the checks at three levels:

• Mapper unit tests cover every domain-exception mapping cheaply.
• Component tests prove routing, serialization, and headers with controlled dependencies.
• Deployed tests pass through authentication and gateway layers for a small set of representative failures.

A CI job can keep the fast contract suite separate from environment checks:

name: api-error-contract

on:
  pull_request:
    paths:
      - "openapi/**"
      - "src/api/**"
      - "tests/contract/**"
  workflow_dispatch:

jobs:
  schema-and-component:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: npm
      - run: npm ci
      - run: npm run openapi:validate
      - run: npm run test:error-contract

  deployed-smoke:
    if: github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
      - run: npm ci
      - run: npm run test:error-contract:deployed
        env:
          API_BASE_URL: ${{ secrets.API_BASE_URL }}
          API_TEST_TOKEN: ${{ secrets.API_TEST_TOKEN }}

The script names are project placeholders. Do not add them unless your repository defines the corresponding commands. What should carry over is the progression: validate the description, run controlled component cases, then test the deployed edge deliberately.

Retain sanitized first-failure artifacts. Runner retries can hide a malformed first response behind a valid second one. If retries are enabled for environment instability, save every attempt and fail the contract case when any supported response violates the schema. A later success does not make the earlier wire contract valid.

The cost of strong error contracts is governance. Stable identifiers need owners. Problem types need documentation. Shared mappers require coordinated changes. Deployed tests take longer and need safe credentials. Those costs are smaller than letting every client reverse-engineer prose and edge fallbacks independently.

Separate malformed JSON from a response truncated in transit

Two failures can produce the same parser message and the same red contract test. The application can serialize an incomplete object, or it can serialize a complete problem while the client receives only the first part of the bytes. Both may surface as an unexpected end of JSON input. The first belongs to the response mapper or serializer. The second belongs to delivery, connection handling, or an intermediary. Rewriting the mapper cannot repair bytes lost after the mapper finished.

The separating evidence is a pair of raw representations taken at different boundaries. Capture the sanitized serialized body as it leaves the application in a controlled component test, then capture the raw body received through the deployed path for the same deterministic case. If the application-side body is already incomplete, the producer owns the defect. If that body parses and the wire body ends earlier, compare the edge path, connection events, and byte counts. When a Content-Length header is present, a received body shorter than that declared length is direct evidence of incomplete delivery. When the response uses another transfer mechanism, rely on the HTTP client's incomplete-message signal and boundary captures rather than inventing a length.

Read diagnostic output in transport order. The status is the actual wire status, not the application's intended value. The media type should identify the promised representation. Content encoding tells the diagnostic client whether decoding must happen before JSON parsing. The raw byte count tells you whether there is a body to inspect. Only after those fields should the report show the parsed type, status, and stable application code. A healthy validation response has the documented wire status and media type, a complete decodable body, a body status equal to the wire status when that member is present, and the expected identifiers.

A broken producer might show the correct status and media type but a complete, parseable object missing a required application code. A broken delivery path might show those same headers followed by a body that stops halfway through a quoted value. The misleading case is a correct Content-Type. That header describes the intended representation, not proof that the received bytes form one. Another misleading value is a body status copied from the service log when the actual wire status was rewritten. Keep observed and intended values separate.

In an existing suite, the first thing likely to break is the helper that calls a JSON parser immediately and discards the response text. Change that helper to retain a sanitized bounded raw artifact on failure before adding stricter assertions. Next, add one deliberately malformed producer fixture and one controlled incomplete-delivery fixture at the lowest layer where your test environment can create them safely. Prove that the report assigns them to different boundaries. Then run the new diagnostics against deployed representative cases without making them a release gate.

Land the producer schema checks before the deployed truncation gate. Otherwise every old empty body, alternate edge response, and client parsing assumption arrives as one migration-sized failure. Once each documented family passes at component level, turn on the deployed check for one route through one edge path. Expand only after the raw artifact, headers, and service-side evidence agree. The change is working when a parser exception no longer forces a rerun to discover which team owns it, and when a complete but invalid object is reported differently from an incomplete response.

Raw capture is not free. Buffering a response for diagnostics can hold the body in memory a second time, and retaining error bodies increases the chance of storing personal or confidential text. Bound the captured size, redact under a reviewed policy, restrict artifact access, and record when truncation was performed by the test itself. That last marker matters because an intentionally shortened artifact must not be mistaken for a shortened wire response. The cost is less convenient forensics in exchange for safer retention and bounded test memory.

The service API owner owns the serialized problem before it leaves the application. The platform or gateway owner owns changes between that boundary and the public response. The client or SDK owner owns decoding and parsing after complete bytes arrive. Security owns the retention rules for diagnostic bodies. A useful handoff contains the request case, observed wire status and headers, encoding state, received byte count, whether the client reported an incomplete message, a sanitized prefix and suffix, and the corresponding application-side parse result. Include a request identifier only when the system already provides one.

This contract technique does not catch information disclosed through timing or through differences across many otherwise valid responses. A schema-perfect authorization problem can still participate in a resource-enumeration weakness if protected and nonexistent resources are distinguishable in a way the security design forbids. That requires dedicated authorization and side-channel testing, not a stronger JSON schema.

Know when one problem shape would be dishonest

Do not force RFC 9457 onto a protocol that already defines its own error envelope unless the protocol permits it and clients expect it. GraphQL responses, for example, have protocol-specific handling that can include data and errors together. Wrapping that response only to look uniform can break tooling and hide partial success.

A body is not always appropriate. Responses with semantics that exclude a body, connection failures before an HTTP response exists, and some authentication interactions need client fallbacks outside the JSON schema. Test those paths as transport or protocol behavior.

Avoid translating third-party errors field for field. Map the external condition to a stable product-facing type and keep vendor details in protected logs. Passing through every upstream field leaks coupling and can expose data the public API never meant to promise.

Do not use a universal 400 because it simplifies the mapper. Status codes let generic components distinguish authentication, authorization, missing resources, conflicts, rate limits, and server failures. Choose the narrow status your contract supports, then verify it at the wire.

A single schema should not make all extensions mandatory. Validation errors may need field pointers. A conflict may need the current resource version. A rate response may use headers. Forcing every problem to include empty arrays and null placeholders produces a larger contract without helping a client.

Finally, a schema pass cannot certify error quality. A perfectly valid object can use the wrong status, expose confidential text, give unsafe retry advice, or identify the wrong problem type. Keep semantic cases beside structural validation, and make each assertion explain which client decision would be wrong if it failed.