The age gate passed, but the AI response did not

A 15-year-old account receives the restricted answer, while a 16-year-old account gets the refusal intended for younger users. The policy wording is fine. The boundary comparison in the delivery service is reversed.

That defect will survive a prompt-only evaluation because the model is not the only component making the decision. A reliable suite has to cover age resolution, response classification, policy selection, final delivery, and any side effect the response can trigger.

Turn the age rule into a testable contract

Before writing cases, ask the policy owner for a versioned decision table. Do not derive thresholds from a marketing page, copy a value from another country, or ask a model what the rule should be. Product, safety, privacy, and legal owners decide the policy. QA turns that approved policy into executable examples.

A useful row contains more than a minimum age:

• policy version and effective date
• product or feature to which the rule applies
• source of the age signal
• resolved audience band, including unknown
• response class being considered
• allowed delivery mode
• required user-facing handling
• side effects that must remain blocked

The response class is crucial. "Safe" and "unsafe" are too broad to debug. A product might distinguish general information, sensitive guidance, and an action that changes an external system. The labels are local to that product. Their names do not create a legal standard.

Age evidence needs the same discipline. A declared age, a profile attribute, and an identity-provider assertion are not interchangeable. If the request contains two of them, the service needs a documented precedence rule. Taking the largest value is an obvious bypass. Taking the smallest value may over-restrict legitimate users and still conceal a data-quality problem.

Unknown is also a real state. It is not an error value to coerce into an adult band. A missing profile, an unreadable claim, and a user who has not completed an age step can require different user journeys even when they share the same conservative content decision.

Enforcement belongs on a trusted server path. Client-side controls can prevent accidental navigation and explain why a feature is unavailable, but they do not protect an API. The browser can be skipped. A test suite should prove that direct requests receive the same decision as requests made through the intended page.

The final gate should receive structured inputs:

1. A resolved age state from the approved identity or profile path.
2. A response class produced before delivery.
3. The policy version selected for this request.
4. The proposed external action, if any.

It should return a structured decision before content is streamed or an action executes. If text has already reached the client, replacing it with a refusal later is not enforcement.

The TypeScript example below is intentionally small. Its thresholds and labels are synthetic fixtures for demonstrating a contract. They are not recommendations for any real product. Save it as src/delivery-policy.ts to run the next listing unchanged.

export type AgeEvidence =
  | { kind: "verified"; age: number }
  | { kind: "declared"; age: number }
  | { kind: "unknown" };

export type ResponseClass =
  | "general-information"
  | "sensitive-guidance"
  | "external-action";

export type DeliveryPolicy = {
  version: string;
  minimumAge: Record<ResponseClass, number>;
  allowWhenAgeUnknown: readonly ResponseClass[];
};

export type DeliveryDecision = {
  deliver: boolean;
  reason:
    | "meets-minimum-age"
    | "class-allowed-with-unknown-age"
    | "below-minimum-age"
    | "age-required"
    | "invalid-age-evidence";
  policyVersion: string;
  responseClass: ResponseClass;
  evidenceKind: AgeEvidence["kind"];
};

export function decideDelivery(
  policy: DeliveryPolicy,
  evidence: AgeEvidence,
  responseClass: ResponseClass,
): DeliveryDecision {
  const common = {
    policyVersion: policy.version,
    responseClass,
    evidenceKind: evidence.kind,
  };

  if (evidence.kind === "unknown") {
    const deliver = policy.allowWhenAgeUnknown.includes(responseClass);
    return {
      ...common,
      deliver,
      reason: deliver
        ? "class-allowed-with-unknown-age"
        : "age-required",
    };
  }

  if (!Number.isInteger(evidence.age) || evidence.age < 0) {
    return {
      ...common,
      deliver: false,
      reason: "invalid-age-evidence",
    };
  }

  const deliver = evidence.age >= policy.minimumAge[responseClass];
  return {
    ...common,
    deliver,
    reason: deliver ? "meets-minimum-age" : "below-minimum-age",
  };
}

The function does not distinguish verified from declared evidence in its threshold. That is a visible limitation of this sample, not a hidden assumption. If the approved policy treats them differently, put that distinction in the table and add rows. Do not scatter special cases through prompts and route handlers.

This design also keeps classification separate from enforcement. A classifier can be deterministic, model-assisted, or human-reviewed. Whatever produces the class needs its own evaluation. The delivery gate should be tested with fixed classes so a classifier miss does not masquerade as a boundary-comparison bug.

Exercise boundaries without asking a model to behave

The smallest valuable matrix uses three ages for each threshold: one below, exactly at, and one above. It also includes unknown and malformed evidence. This catches greater-than versus greater-than-or-equal errors, string-to-number mistakes, and default branches.

Run that matrix against a pure function on every pull request. Save this Playwright Test file as tests/delivery-policy.spec.ts. It uses the runner and assertion library without starting a browser.

import { test, expect } from "@playwright/test";
import {
  decideDelivery,
  type DeliveryPolicy,
  type ResponseClass,
} from "../src/delivery-policy";

const policy: DeliveryPolicy = {
  version: "fixture-2026-08-04",
  minimumAge: {
    "general-information": 0,
    "sensitive-guidance": 16,
    "external-action": 18,
  },
  allowWhenAgeUnknown: ["general-information"],
};

const boundaryCases: Array<{
  name: string;
  age: number;
  responseClass: ResponseClass;
  deliver: boolean;
}> = [
  {
    name: "one below sensitive threshold",
    age: 15,
    responseClass: "sensitive-guidance",
    deliver: false,
  },
  {
    name: "exact sensitive threshold",
    age: 16,
    responseClass: "sensitive-guidance",
    deliver: true,
  },
  {
    name: "one above sensitive threshold",
    age: 17,
    responseClass: "sensitive-guidance",
    deliver: true,
  },
  {
    name: "one below external action threshold",
    age: 17,
    responseClass: "external-action",
    deliver: false,
  },
  {
    name: "exact external action threshold",
    age: 18,
    responseClass: "external-action",
    deliver: true,
  },
  {
    name: "one above external action threshold",
    age: 19,
    responseClass: "external-action",
    deliver: true,
  },
];

for (const item of boundaryCases) {
  test(item.name, () => {
    const actual = decideDelivery(
      policy,
      { kind: "verified", age: item.age },
      item.responseClass,
    );
    expect(actual.deliver).toBe(item.deliver);
    expect(actual.policyVersion).toBe("fixture-2026-08-04");
  });
}

test("unknown age allows only the configured class", () => {
  expect(
    decideDelivery(policy, { kind: "unknown" }, "general-information").deliver,
  ).toBe(true);
  expect(
    decideDelivery(policy, { kind: "unknown" }, "external-action"),
  ).toEqual({
      deliver: false,
      reason: "age-required",
      policyVersion: "fixture-2026-08-04",
      responseClass: "external-action",
      evidenceKind: "unknown",
    });
});

test("invalid numeric evidence is blocked and identified", () => {
  const actual = decideDelivery(
    policy,
    { kind: "declared", age: Number.NaN },
    "general-information",
  );
  expect(actual.deliver).toBe(false);
  expect(actual.reason).toBe("invalid-age-evidence");
});

Notice what these tests do not assert. They do not call a model and look for a refusal phrase. They do not count keywords in free-form text. They prove the rule inputs and delivery decision first.

Add a smaller number of integration tests for the real delivery route. Seed a controlled account through the application's supported test-data path and replace generation with a fixed, already classified response. Then send a direct API request with Playwright's request fixture and assert the documented rejection status and body.

Do not put the resolved audience band in a client-controlled field merely to make the test easy. The server should derive it through the same trusted path used in production. If the test environment needs a shortcut, keep that hook out of production builds and cover its isolation separately.

The integration assertion should also inspect the fake side-effect adapter. A blocked response with one recorded external action is a failure even when the response status and user message are correct. Conversely, a blocked request that records no action but uses the wrong reason code points to policy selection or age resolution rather than execution.

For browser coverage, check the visible explanation and confirm that restricted content was never inserted into the DOM. That is useful, but it remains secondary to the server assertion. A green browser test cannot prove that another client is protected.

Work through three failures with different causes

An incorrect response at an age boundary does not identify the broken layer. The same symptom can come from age calculation, stale identity data, classifier error, policy selection, or delivery timing.

The birthday calculation is one day early

A profile stores a date of birth, while the policy service expects an age band. The conversion runs at midnight UTC. The product policy defines the relevant day in the user's account region. Around the birthday, the UTC date and local date differ, so the account enters the next band too early or too late.

Evidence for this defect includes the original date-only value, the policy's reference date, the time-zone rule used by the resolver, and the computed band. Do not log a full birth date broadly just to debug it. A controlled test can use synthetic dates and retain only the fixture ID plus the computed values in ordinary CI output.

The fix is to centralize the conversion and pass an explicit policy date or clock. Avoid calling the current time from several services. The cost is an additional contract and careful migration of existing callers.

Test Feb 28, Feb 29, Mar 1, the last day of a month, the first day of a month, and a run on each side of the chosen midnight. These are calendar cases, not model evaluations. If the system stores only a numeric age entered months ago, the deeper problem is stale data rather than date math.

The session carries an old audience band

An account completes an age-verification step, but the existing session token or cache still contains unknown. The next request receives a conservative refusal. The inverse is more serious: a profile becomes restricted, but a long-lived session keeps the older permissive band.

This looks like a threshold bug in the final answer. The trace tells a different story when the profile revision and session revision disagree. Record both revision identifiers, not the raw identity evidence. Check whether the route resolved the band for every request, at session creation, or from a cache with a documented invalidation path.

The fix may be shorter token lifetime, server-side revocation, revision checking, or forced reauthentication for a sensitive transition. Each option costs something. Short lifetimes add refresh traffic. Revision checks add a read. Forced reauthentication adds user friction.

Do not "fix" this by allowing the more permissive value when two stores disagree. Treat the mismatch as an identity-state conflict and apply the approved fail-safe behavior.

The response was classified into the wrong bucket

The age and policy version are correct, but a response that initiates an external action is labeled general-information. The delivery gate allows it because it received the wrong class.

A boundary suite that reports only deliver true will blame enforcement. Inspect the classifier input, classifier revision, returned class, and delivery decision separately. Replay the exact generated fixture against the classifier evaluation set. Then feed the correct fixed class to the gate. If the gate blocks it, the boundary logic works and the classifier owns the defect.

The lasting fix may be a deterministic rule around tool capability rather than better text classification. If a response is paired with a payment or account-changing tool call, the orchestration layer already knows an external action is proposed. Use that structured fact. Text classification can still cover content that has no corresponding tool.

This split costs implementation work, but it makes failure ownership clear. One giant end-to-end score cannot tell whether age resolution, classification, or enforcement regressed.

The refusal arrives after content started streaming

A system may begin sending tokens while a downstream policy check is still running. The UI eventually replaces the text with a refusal, and a screenshot taken at the end looks correct. The restricted content already crossed the delivery boundary.

Look at server event order and the browser's network response, not only the final DOM. The policy decision must precede the first content chunk for classes that require gating. If classification cannot happen before generation, buffer the response until the decision is available.

Buffering increases time to first visible content and memory use. Early classification can reduce latency but may be wrong if the class depends on the complete response. The architecture has to choose a safe point based on the policy, then the test should assert that point directly.

The current policy belongs to the wrong product scope

A selection defect can produce the same boundary symptom as a bad comparison. The trace contains a valid age, the expected response class, a current policy version, and a refusal at the threshold. The arithmetic may be correct. The request was evaluated against the table for another feature or region.

Do not stop after reading policyVersion. Resolve that version to its approved scope, then compare the scope with the trusted product, feature, and region inputs selected for the request. In an illustrative healthy trace, a request for the controlled chat feature resolves to the policy row approved for that feature, and the recorded threshold agrees with that row. In a broken trace, the age evidence and response class are unchanged, but the selected row belongs to a different surface. A misleading trace shows the newest available policy version and a reason such as below minimum age. Both values can be internally consistent while answering the wrong policy question.

This separates selection from cache staleness. A stale-cache case reports an older revision of the correct scoped table. A selector case reports a current revision of the wrong table. Compare the effective scope before flushing caches or changing the numeric boundary. Clearing a cache may temporarily move traffic and conceal the routing error without fixing it.

For a suite that already asserts delivery decisions, land policy-selection evidence before adding new expected outcomes. First give each controlled account an explicit expected product, feature, and governed region in the fixture catalog. Then record the selector's trusted inputs and the resulting approved table identity without changing delivery. Add a shadow assertion that compares expected and selected scope, and repair ambiguous fixtures. Once every supported route has a named scope, block on selection mismatches. Keep the threshold assertions after that check so a failure points to the earliest wrong decision. Existing tests that relied on a default table will break first, which is useful only if the report identifies the missing fixture dimension rather than calling the user underage.

The cost is fixture growth and operational complexity. Region and feature variants multiply the rows that must be reviewed when policy changes. Splitting caches by scope can lower hit rates and add a policy lookup on routes that previously used one global value. Those costs are more concrete than a generic warning about configuration complexity, and they should be measured separately from model latency.

Product safety or legal policy owners approve the scoped table. Identity teams own trusted region evidence, product teams own the feature classification, and the policy-platform team owns selection and cache behavior. QA should hand over the request ID, synthetic account, expected scope, trusted selector inputs and their revisions, selected table identity, policy version, response class, decision reason, and first-delivery event. That evidence prevents three teams from each proving its own local value while nobody checks the join.

Selection tests do not establish that the approved policy is lawful, fair, or up to date. They prove that software chose and enforced the table it was given. Policy review remains a separate governance control.

Use the trace to locate the broken layer

A useful trace follows one request across the boundary:

• requestId links the API request, model or fixture generation, classification, decision, and side effect.
• policyVersion proves which table was loaded.
• ageEvidenceKind and ageSourceRevision describe the resolved input without copying unnecessary identity data.
• resolvedBand records the output of age resolution.
• responseClass and classifierRevision identify classification.
• decision, reason, and decidedAt describe enforcement.
• firstDeliveryAt shows whether delivery began before approval.
• sideEffectCount or named action events show whether anything protected ran.

The evidence should be structured. A log line saying "minor policy applied" is hard to compare and easy to lose. A decision event with stable fields can be asserted, redacted, and aggregated.

Run IDs must be unique per attempt. If a test retries after an infrastructure failure, keep both attempts. A passing second attempt does not prove the first response was never delivered.

The bash diagnostic below reads synthetic JSON Lines events for one request. It reports the stage order and flags delivery before a blocking decision. It uses source timestamps from one recorder; do not compare unrelated host clocks without a synchronization contract.

#!/usr/bin/env bash
set -euo pipefail

trace_file="$1"
request_id="$2"

if [[ -z "$trace_file" || -z "$request_id" ]]; then
  echo "usage: inspect-age-gate.sh TRACE.jsonl REQUEST_ID" >&2
  exit 2
fi

jq -s --arg request "$request_id" '
  map(select(.requestId == $request))
  | sort_by(.recordedSequence)
  | {
      requestId: $request,
      events: map({
        recordedSequence,
        type,
        policyVersion,
        resolvedBand,
        responseClass,
        decision,
        reason
      }),
      decisionSequence: (
        [.[] | select(.type == "delivery_decision") | .recordedSequence] | first
      ),
      firstDeliverySequence: (
        [.[] | select(.type == "content_delivery") | .recordedSequence] | first
      )
    }
  | . + {
      deliveredBeforeDecision: (
        .firstDeliverySequence != null
        and (
          .decisionSequence == null
          or .firstDeliverySequence < .decisionSequence
        )
      )
    }
' "$trace_file"

Use a recorder-owned sequence for ordering when events pass through one boundary. It avoids pretending that ISO timestamps from several machines have perfect clock alignment. If events are produced by independent services, preserve causal identifiers and service-local sequence numbers. A total order may not exist.

Typical deterministic test failures point to different owners:

• Expected band younger, received older: inspect age calculation and source precedence.
• Expected policy version fixture-2026-08-04, received an earlier version: inspect configuration rollout and cache invalidation.
• Expected response class external-action, received general-information: inspect the classifier or structured action mapping.
• Expected decision before delivery, observed content_delivery first: inspect streaming and buffering.
• Expected zero protected actions, received one: inspect orchestration even if the user-facing refusal is correct.

Playwright traces can help with the browser and API portion. They show test actions, network activity, and attachments created by the test. Attach a redacted server decision event when the test fails. The browser trace will not automatically contain internal service events, so correlation still depends on your request ID and server instrumentation.

Do not attach raw identity tokens, birth dates, or unrestricted model content to every CI artifact. A trace that proves the decision with a synthetic fixture ID, band, revisions, and reason is usually enough.

Roll out the gate without hiding regressions

Begin with the approved policy table and a named owner. Add a schema check so malformed policy data fails deployment before requests use it. A missing response class must not silently inherit the most permissive rule.

Next extract age resolution and delivery enforcement into functions that accept their dependencies. Add a controllable clock for date boundaries. Keep production adapters thin, then test the functions with fixed data.

Build a fixture catalog with several dimensions:

• below, at, and above every threshold
• declared, verified, unknown, missing, malformed, and conflicting evidence
• each response class and protected side effect
• current and previous policy versions during migration
• birthday, leap-day, month-end, and time-zone boundaries when date of birth is used
• cached and refreshed identity states
• streaming allowed and streaming blocked paths

Do not generate the full Cartesian product blindly. Many combinations add runtime without adding a new mechanism. Use pairwise selection for ordinary rows, then keep explicit cases for every high-risk boundary and previously found defect.

Run pure decision tests on each change. Run API integration tests against a controlled service build. Keep a smaller browser suite for the user journey, accessible explanation, and absence of restricted DOM content. Live-model evaluations can run separately to measure response classification and language quality.

A CI layout can make those responsibilities visible:

name: age-policy-contract

on:
  pull_request:
  workflow_dispatch:

jobs:
  decision-matrix:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: >-
          npx playwright test tests/delivery-policy.spec.ts
          --reporter=line

This job runs the deterministic listing from the article. Add the repository's real API and browser test paths only after those tests use supported, isolated fixture data. Keeping placeholder commands out of the job prevents a green step that never collected the intended cases.

During migration, compare the old and new decision functions in shadow mode with synthetic or properly governed traffic. Record disagreements by rule and fixture, not only a percentage. There is no universal acceptable disagreement rate. Each changed decision needs review against the approved policy.

Move gates in a controlled order. Block pull requests on deterministic boundary cases first. Add API integration gating when instrumentation is complete. Treat live-model class changes as a separate evaluation with an owner and review path. This prevents a nondeterministic wording change from disabling a critical enforcement suite.

When the policy version changes, keep old fixtures for historical behavior and add new expected rows. Label which version each case targets. Do not edit yesterday's expected value in place and lose evidence of what changed.

Monitor production decisions without collecting more identity data than necessary. A sudden rise in unknown states may indicate an identity integration failure, but alerting needs a baseline from the actual system. Do not copy illustrative counts from a test article into an operational threshold.

Accept the cost and know when not to gate

Server-side enforcement adds latency, especially when age resolution or classification requires another service. Caching can reduce that cost but creates stale-state risk. Buffering a generated response protects the delivery boundary but increases time to first content. Name those costs in the design review.

Detailed evidence improves diagnosis and increases privacy exposure. Prefer synthetic accounts in CI. In production, log policy inputs at the band and revision level when possible, not the raw date of birth. Limit access and retention.

Conservative unknown handling protects restricted paths but can deny service to users whose profile failed to load. Provide a recovery path. A generic refusal with no way to refresh or complete verification turns a safe failure into a support burden.

Classification adds another error surface. If the product can decide from a structured capability or tool action, use that signal instead of inferring everything from prose. Keep model-based classification for the content that genuinely requires it.

Do not use age as a proxy for a risk that the policy does not tie to age. General abuse prevention, fraud controls, and account authorization need their own rules. Combining them under one age gate makes both the tests and user explanation misleading.

Do not put real identity verification into ordinary browser tests when a controlled fixture can prove the integration. External verification flows may be expensive, rate-limited, or legally sensitive. Cover the provider contract in a small dedicated suite and test internal decisions with seeded evidence.

Do not assert that one set of thresholds applies everywhere. Products, features, and regions can have different approved requirements. The test must select a policy version explicitly and fail if selection is ambiguous.

Do not block harmless, age-neutral information because the classifier lacks a class. The safe default must come from the approved table, and unknown classes should be visible as configuration defects. A permanent blanket refusal encourages teams to bypass the gate.

Most importantly, do not treat a correct refusal sentence as proof of a correct system. The content may have streamed first, a tool may already have acted, or a stale session may have selected the wrong band. The enforceable contract lives in the ordered decision and side effects, not in how convincing the final paragraph sounds.