Contract Testing Tool Schema Evolution Across Agent Releases

Tool schemas are executable contracts between a planner, runtime validator, implementation, and downstream consumer. A change that looks harmless in JSON can alter tool selection, argument meaning, authorization, or side effects. Treat schema evolution as a producer-consumer migration, not a documentation edit.

Build a compatibility corpus from real and designed calls, run old agents against the new tool boundary, run new agents against every supported boundary, and inspect observable state. Structural validation is necessary, but behavior decides whether a release is compatible.

Define the Contract Beyond Input Shape

Version the tool name, description, input schema, output schema, defaults, error model, authorization requirements, approval policy, idempotency behavior, and side-effect semantics. Include examples used in prompts and any adapter that rewrites arguments. A consumer contract also covers how callers interpret missing output fields and failures.

The MCP tools specification defines tool metadata including a unique name, description, inputSchema, and optional outputSchema. Use parsed JSON Schema for structural comparisons, while maintaining a separate behavioral contract for concerns JSON Schema cannot express.

Classify Changes by Producer and Consumer Impact

Adding a required input, renaming a property, narrowing a type, removing an enum value, or changing units is structurally breaking for old producers. Adding an optional field may validate old calls, but a new default can change effects. Relaxing additionalProperties can allow misspelled fields to pass silently. Tightening it can reject calls older agents emitted.

Output changes have the reverse direction. Removing a field, changing nullability, narrowing an enum, or altering error structure can break existing consumers. Adding a field is usually easier for consumers that ignore unknown properties, but strict deserializers may still fail.

Descriptions are behavioral inputs to an agent. A rewritten description can change tool choice or argument construction even when the schema is byte-for-byte identical. Classify prompt-facing metadata changes and run end-to-end agent comparisons for them.

Build a Versioned Compatibility Corpus

Collect sanitized historical argument objects with tool contract and agent release IDs. Add designed boundaries: missing required fields, omitted optional fields, explicit null, unknown fields, minimum and maximum values, empty strings, Unicode where supported, enum case differences, duplicate semantic fields, and malformed nested objects.

Include calls that previously failed. A new schema should not accidentally accept a dangerous request simply because it is more permissive. Label expected parse result, canonical operation, authorization decision, approval requirement, and side-effect outcome.

Group examples by task family and source trace. Keep a sealed set for release decisions so adapter development does not overfit every observed call. For rare mutation tools, use expert-designed cases even when historical volume is low.

Compare Schemas Structurally With a Parser

Resolve references and compare semantic JSON Schema constructs rather than diffing formatted text. Flag required-set changes, property addition or removal, type and format changes, enum deltas, bounds, patterns, defaults, additionalProperties, conditional branches, and output schema movement.

Represent the expected compatibility direction explicitly:

{
  "tool": "create_invoice",
  "from": "1.4",
  "to": "2.0",
  "oldProducerToNewConsumer": "adapter_required",
  "newProducerToOldConsumer": "unsupported",
  "changes": [
    {"path": "/properties/customer_id", "kind": "renamed", "replacement": "account_id"},
    {"path": "/properties/currency", "kind": "required_added", "defaultPolicy": "no_implicit_default"}
  ]
}

Schema tools can identify likely compatibility breaks, but they cannot decide whether changing amount from dollars to cents preserves meaning. Require a human-reviewed semantic change record for units, identity, permissions, and side effects.

Test Adapters as First-Class Code

An adapter should accept a declared source version, produce one canonical internal command, and reject ambiguous combinations. If both old and new field names are present, define whether equal values are accepted and conflicting values fail. Log adapter path and deprecation usage without exposing sensitive arguments.

type V1 = { customer_id: string; amount_dollars: number };
type V2 = { account_id: string; amount_cents: number; currency: string };

export function adaptV1(input: V1): V2 {
  if (!Number.isSafeInteger(input.amount_dollars * 100)) {
    throw new Error("amount cannot be represented exactly in cents");
  }
  return {
    account_id: input.customer_id,
    amount_cents: input.amount_dollars * 100,
    currency: "USD"
  };
}

This example's fixed currency is valid only if the old contract was explicitly USD-only. If that historical invariant did not exist, the adapter must request the missing value or reject the call. Migration code cannot invent business facts.

Property-test adapters around boundaries, then replay the compatibility corpus. Verify canonical request hashes, authorization inputs, and effect-bearing values before any live mutation.

Run the Producer-Consumer Matrix

At minimum, test old agent with old boundary, old agent with new boundary, new agent with supported old boundary, and new agent with new boundary. Add every deployed client or server release still inside the support window. Record validation result, selected tool, normalized arguments, user clarification, guardrail decision, and final state.

Use a sandbox or fake provider for mutations. The old agent may satisfy the new schema through an adapter yet choose the tool more often because its description changed. The new agent may send only new fields and fail when routed to an older server during a rolling deployment.

Test mixed fleets under realistic routing. A conversation can begin on one runtime version and resume on another. Persist tool contract identity with pending approvals and checkpoints so resumed work uses the expected decoder.

Protect Authorization and Side-Effect Semantics

Run permission checks after adaptation on the canonical command, and preserve trusted tenant context outside model arguments. A renamed resource field must not bypass an authorization rule keyed to the old path. Test cross-tenant IDs, revoked roles, and an adapter that receives a syntactically valid but unauthorized target.

The OpenAI Agents SDK guardrails guide distinguishes agent input/output checks from tool guardrails that wrap custom function-tool invocations. Regardless of framework, contract tests should exercise validation and authorization at the actual tool boundary for every supported schema route.

Keep idempotency stable across adapters. Equivalent old and new requests should map to the same canonical operation fingerprint when they represent one retry. Conversely, a changed effect must not collide with an old key.

Exercise Errors, Outputs, and Rollback

Schema releases often focus on successful inputs and neglect errors. Test validation error codes, field paths, retryability, permission denial, conflict, partial success, and unknown provider state. Older agents must not interpret a new final error as retryable and repeat a mutation.

Replay output consumers against candidate results. Verify optional fields, null handling, structured content, pagination, and human-readable fallback. If an adapter transforms outputs, test information loss and ensure it cannot turn an error into success.

Prepare rollback before rollout. Retain the prior schema, adapter, and tool implementation until pending calls and checkpoints have drained. If the tool list is cached, invalidate it deliberately so clients do not keep generating calls for a removed contract.

Set Compatibility Gates and Deprecation Evidence

Gate hard failures separately: unauthorized effects, argument misbinding, duplicate mutations, and old accepted calls that now execute differently. Report ordinary validation changes, clarification rates, tool-selection movement, and output parse failures by producer-consumer pair.

Illustrative local policy might require every supported old call to validate or receive a documented adapter result, zero semantic mismatches in effect-bearing fields, and no critical state delta. Mark any numeric tolerance as illustrative until the product owner defines it.

Deprecate from evidence. Track which agent releases still use the old route, whether pending workflows reference it, and whether rollback needs it. Remove compatibility code only after the support contract and observed traffic agree.

Contract Review Checklist

• Version names, descriptions, inputs, outputs, errors, permissions, and effects.
• Compare schemas as parsed structures, including references and conditionals.
• Replay accepted historical calls and rejected boundary cases.
• Label compatibility in both producer-consumer directions.
• Test adapters for conflicts, precision, defaults, and authorization binding.
• Run old and new agents against every supported tool boundary.
• Verify final state and side effects, not only schema acceptance.
• Exercise errors, pending approvals, checkpoints, and rolling deployments.
• Preserve idempotency identity across equivalent versions.
• Keep the previous contract available until rollback and drain are complete.

Action Plan: Evolve One Tool Without Breaking Its Fleet

Freeze the current tool contract and collect a sanitized call plus output corpus. Classify the proposed change structurally and semantically, then decide whether an adapter can preserve meaning. Implement that adapter behind an explicit source version and reject every case where missing information would require guessing.

Run the full producer-consumer matrix in a side-effect-controlled environment, inspect tool-selection and state deltas, and rehearse rollback with pending workflows. Advertise the new contract gradually, measure old-route usage, and remove the legacy path only after supported agents, caches, and resumable operations no longer depend on it.