Stop uncertain safety classifications from becoming silent passes

A moderation check times out during a traffic spike, and the application serves the generated answer anyway. The dashboard records the result as “not flagged,” so the release report treats missing evidence as a clean classification. That is not a model miss. It is a broken decision contract.

Why a binary label hides the risky cases

Most production failures happen after someone compresses several different facts into one Boolean. The classifier call either completed or it did not. A completed call may contain a top-level flag, category labels, category scores, a model identifier, and input-type information. The product then makes a separate choice: release the content, block it, reduce capability, or ask for review. Calling all of that safe or unsafe throws away the exact distinction QA needs.

The OpenAI moderation guide is a useful concrete example. Its documented result includes flagged, per-category categories, category_scores, and category_applied_input_types. The guide describes the scores as model confidence signals and explicitly says an application should treat them as policy inputs, not automatic blocking decisions. It also documents error variants for inline moderation. None of those fields promises that a failed request should become false.

Abstention belongs between observation and action. It means the automated path declines to make its normal allow-or-block decision because its evidence does not satisfy the application contract. There are several honest reasons to do that:

• A network error, timeout, or rate limit left no classifier result.
• The response arrived but omitted a category the policy requires.
• A score was not finite, sat outside the documented range, or could not be parsed.
• The result was valid but fell inside an uncertainty band established from a labeled evaluation set.
• The classifier did not support the supplied modality for a category that matters to the product.
• The returned model or policy version was not one approved by the release configuration.

Those cases do not deserve the same reason code. A reviewer can resolve a borderline input. A reviewer cannot reconstruct category evidence that never arrived unless the system safely retries with the original input. A platform engineer owns repeated timeouts. A model-evaluation owner investigates distribution changes after a model update. One generic review status hides all four owners.

The top-level flag also needs a clear precedence rule. If the provider flags content, a custom score band should not silently downgrade that result to uncertain. A team may impose stricter blocking rules for its domain, but its adapter should preserve the provider flag and the raw evidence used by its policy. That gives the product a conservative floor while leaving room for domain-specific restrictions.

Do not name every score a probability of harm. A score can be useful for ranking and routing without proving a calibrated real-world probability. The moderation documentation warns that custom policies based on category scores may need recalibration as the underlying model changes. That warning is operationally important. A test suite that asserts only that a score is a number will stay green while the score distribution crosses an application threshold.

The mechanism becomes easier to test when the system keeps three layers separate:

The last layer is frequently missing from AI evals. A unit test proves that boundary content maps to abstain, but the review queue rejects the message and the request handler falls through to publish. The classifier and policy both behaved correctly. The safety control still failed. A release test must follow the decision until the fallback accepts responsibility.

Human review is not a decorative label. OpenAI’s safety best-practices guide recommends human review, especially in high-stakes uses, and says reviewers need the information required to verify an output. The W3C ethical principles for Web machine learning also emphasize human oversight for high-impact or difficult-to-reverse decisions. In test terms, this means a review route needs a defined payload, an acknowledgement, an owner, and a response the user can safely receive while review is pending.

How to make abstention an explicit contract

Start with an enum, not a nullable Boolean. Null inevitably acquires several meanings: the call was skipped, the parser failed, the model was unsure, or a developer forgot to assign the field. Three decisions and specific reasons make illegal shortcuts visible in code review and test output.

The following Python module is runnable as written. Its thresholds and fixture scores are illustrative control-flow values. They are not results from an experiment and should not be copied into a production policy. Save it as moderation_policy.py.

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum
from math import isfinite
from typing import Mapping


class Decision(str, Enum):
    ALLOW = "allow"
    BLOCK = "block"
    ABSTAIN = "abstain"


@dataclass(frozen=True)
class Evidence:
    request_id: str
    model: str
    flagged: bool
    scores: Mapping[str, float]


@dataclass(frozen=True)
class Policy:
    version: str
    required_categories: tuple[str, ...]
    allow_at_most: float
    block_at_least: float

    def __post_init__(self) -> None:
        if not 0.0 <= self.allow_at_most < self.block_at_least <= 1.0:
            raise ValueError("policy thresholds must satisfy 0 <= allow < block <= 1")
        if not self.required_categories:
            raise ValueError("at least one category is required")


@dataclass(frozen=True)
class Outcome:
    decision: Decision
    reason: str
    highest_category: str | None = None
    highest_score: float | None = None


def evaluate(
    evidence: Evidence | None,
    policy: Policy,
    transport_error: str | None = None,
) -> Outcome:
    if transport_error is not None:
        return Outcome(Decision.ABSTAIN, f"classifier_unavailable:{transport_error}")
    if evidence is None:
        return Outcome(Decision.ABSTAIN, "classifier_result_missing")
    if not evidence.model:
        return Outcome(Decision.ABSTAIN, "model_id_missing")

    # A provider flag is never weakened by the custom score policy.
    if evidence.flagged:
        return Outcome(Decision.BLOCK, "provider_flagged")

    missing = [
        category
        for category in policy.required_categories
        if category not in evidence.scores
    ]
    if missing:
        return Outcome(Decision.ABSTAIN, f"missing_categories:{','.join(missing)}")

    invalid = [
        category
        for category in policy.required_categories
        if not isfinite(evidence.scores[category])
        or not 0.0 <= evidence.scores[category] <= 1.0
    ]
    if invalid:
        return Outcome(Decision.ABSTAIN, f"invalid_scores:{','.join(invalid)}")

    highest_score, highest_category = max(
        (evidence.scores[category], category)
        for category in policy.required_categories
    )
    if highest_score >= policy.block_at_least:
        return Outcome(
            Decision.BLOCK,
            "custom_block_threshold",
            highest_category,
            highest_score,
        )
    if highest_score <= policy.allow_at_most:
        return Outcome(
            Decision.ALLOW,
            "below_allow_threshold",
            highest_category,
            highest_score,
        )
    return Outcome(
        Decision.ABSTAIN,
        "boundary_band",
        highest_category,
        highest_score,
    )


def fallback_for(outcome: Outcome) -> str:
    return {
        Decision.ALLOW: "release_content",
        Decision.BLOCK: "replace_with_safe_response",
        Decision.ABSTAIN: "enqueue_for_review",
    }[outcome.decision]


if __name__ == "__main__":
    example_policy = Policy(
        version="illustrative-v1",
        required_categories=("violence", "self-harm"),
        allow_at_most=0.20,
        block_at_least=0.80,
    )
    example_evidence = Evidence(
        request_id="example-001",
        model="example-model",
        flagged=False,
        scores={"violence": 0.55, "self-harm": 0.08},
    )
    result = evaluate(example_evidence, example_policy)
    print(result.decision.value, result.reason, fallback_for(result))

The policy uses inclusive boundary semantics deliberately. A score equal to allow_at_most is allowed, while a score equal to block_at_least is blocked. Everything between those values abstains. Whether those comparisons are appropriate for a real classifier is a product decision backed by labeled cases. What matters for QA is that equality is specified and covered, rather than inherited from whichever comparison operator a developer happened to type.

Worked example: a valid but unresolved result. Imagine the provider returns a valid unflagged response. The highest relevant score is inside the application’s reviewed band. An older handler checks only flagged, sees false, and publishes. The module above returns abstain with boundary_band. The fallback then chooses the review route. Evidence for this bug is a completed classifier request, a parseable response, a known model, and a decision that disagrees with the configured band. It is not a network incident.

The trade-off is immediate. A wider band sends more work to reviewers and adds user-visible delay. A narrow band reduces that cost but automates more borderline decisions. There is no universally safe width. Build a labeled dataset from your own traffic classes, include adversarial and multilingual cases, and compare the consequences of false allows, false blocks, and delayed decisions. Record the dataset and policy versions together.

Worked example: the response never existed. A timeout takes a different route through the same decision enum. The outcome reason is classifier_unavailable rather than boundary_band. That distinction lets operations alert on availability without pretending that a reviewer can judge from missing evidence. For a high-impact action, the fallback may deny the action or hold it pending a safe retry. For a reversible, low-risk feature, a product might choose reduced capability. The application team must make that choice explicitly.

Do not catch an exception and manufacture Evidence(flagged=False, scores={}). That object claims the classifier observed the input and found no flag. A timeout proves neither claim. It will also make availability failures look like content-distribution changes in dashboards, sending the incident to the wrong team.

How to prove the fallback really contains risk

A good regression suite tests decisions, reason codes, and boundary equality. It does not call a live classifier for every pull request. Live calls add cost, latency, rate limits, external drift, and possible exposure of test content. Put the policy contract under deterministic tests, then run a smaller scheduled integration suite against the real adapter.

Pytest’s documented parametrize marker is a clean fit because each row can name a distinct failure rather than hiding several assertions in a loop. Save this next block as tests/test_moderation_policy.py. The scores remain illustrative fixtures.

import pytest

from moderation_policy import Decision, Evidence, Policy, evaluate


POLICY = Policy(
    version="illustrative-v1",
    required_categories=("violence", "self-harm"),
    allow_at_most=0.20,
    block_at_least=0.80,
)


@pytest.mark.parametrize(
    ("evidence", "transport_error", "expected_decision", "expected_reason"),
    [
        pytest.param(
            Evidence(
                request_id="clear",
                model="example-model",
                flagged=False,
                scores={"violence": 0.04, "self-harm": 0.03},
            ),
            None,
            Decision.ALLOW,
            "below_allow_threshold",
            id="clear-result-is-allowed",
        ),
        pytest.param(
            Evidence(
                request_id="boundary",
                model="example-model",
                flagged=False,
                scores={"violence": 0.55, "self-harm": 0.08},
            ),
            None,
            Decision.ABSTAIN,
            "boundary_band",
            id="boundary-result-is-reviewed",
        ),
        pytest.param(
            Evidence(
                request_id="flagged",
                model="example-model",
                flagged=True,
                scores={"violence": 0.10, "self-harm": 0.10},
            ),
            None,
            Decision.BLOCK,
            "provider_flagged",
            id="provider-flag-cannot-be-downgraded",
        ),
        pytest.param(
            None,
            "timeout",
            Decision.ABSTAIN,
            "classifier_unavailable:timeout",
            id="timeout-is-not-a-clean-result",
        ),
        pytest.param(
            Evidence(
                request_id="partial",
                model="example-model",
                flagged=False,
                scores={"violence": 0.02},
            ),
            None,
            Decision.ABSTAIN,
            "missing_categories:self-harm",
            id="partial-response-is-not-allowed",
        ),
    ],
)
def test_policy_routes_each_evidence_state(
    evidence: Evidence | None,
    transport_error: str | None,
    expected_decision: Decision,
    expected_reason: str,
) -> None:
    outcome = evaluate(evidence, POLICY, transport_error)

    assert outcome.decision is expected_decision
    assert outcome.reason == expected_reason


@pytest.mark.parametrize(
    ("score", "expected"),
    [
        pytest.param(0.20, Decision.ALLOW, id="allow-boundary-is-inclusive"),
        pytest.param(0.80, Decision.BLOCK, id="block-boundary-is-inclusive"),
    ],
)
def test_threshold_equality_is_part_of_the_contract(
    score: float,
    expected: Decision,
) -> None:
    evidence = Evidence(
        request_id=f"edge-{score}",
        model="example-model",
        flagged=False,
        scores={"violence": score, "self-harm": 0.01},
    )

    assert evaluate(evidence, POLICY).decision is expected

Run the smallest failing node first. Pytest’s output guide documents node IDs and verbose output, so the command can show the named parameter row that broke instead of dumping an entire safety suite.

set -o errexit
set -o nounset
set -o pipefail

python -m pytest \
  tests/test_moderation_policy.py::test_policy_routes_each_evidence_state \
  --verbose \
  --showlocals

Suppose a developer changes the timeout branch to allow. The useful part of the failure will contain the parameter ID timeout-is-not-a-clean-result, followed by an assertion showing Decision.ALLOW where Decision.ABSTAIN was expected. That is deterministic diagnostic output from the policy test. It points to routing logic, not classifier quality. If the same case instead fails in a live adapter test before evaluate is called, inspect the transport and schema evidence.

A failing excerpt from that deliberately mutated branch has this shape:

The fallback needs its own contract test. Replacing content with a safe response is usually synchronous and easy to assert. A review queue is harder because successful serialization is not successful acceptance. Mock the queue at the adapter boundary in unit tests and assert that the exact case ID, reason, policy version, and evidence reference are sent. In an integration environment, wait for the queue’s documented acknowledgement or observable record. Do not assert against an arbitrary sleep.

Worked example: abstention succeeds, enqueue fails. The policy result says abstain and the handler calls the queue client. The client rejects the payload because policy_version became required in a schema change. If the error handler then returns the original content, this is a fallback failure. The evidence chain looks like this:

1. Classifier request completed.
2. Policy emitted abstain with boundary_band.
3. Review submission returned a rejection.
4. User response followed the normal publish path.

Only step four turns the defect into unsafe release behavior. A test that ends at step two will report a green safety check. Add an assertion on the final response or side effect. The cost is tighter integration between the policy suite and routing layer, but that coupling reflects the real safety boundary.

An asynchronous review design also needs a pending experience. If users can repeatedly resubmit while the first item waits, the queue may fill with duplicates. If the product displays the unreviewed answer during that wait, review is not a containment measure. Tests should cover duplicate case handling, user cancellation, expired review items, and the response returned before a reviewer acts. These are workflow states, not model-evaluation metrics.

What the look-alike failures reveal

Several incidents produce the same customer report: “the classifier let this through.” Their logs separate them if the team retained evidence before reducing it to a Boolean.

Uncertainty band versus provider outage. Both can end in review. The uncertainty case has a request ID, returned model, complete scores, and boundary_band. The outage has an exception class or HTTP outcome and no valid classifier result. A sudden increase in boundary cases suggests content or model-distribution change. A sudden increase in unavailable reasons suggests dependency or networking trouble. Do not aggregate them into one abstention-rate alarm.

Unsupported modality versus genuinely low score. The moderation guide documents category_applied_input_types and notes that some categories support only particular input types. For an image-only request, a zero for a text-only category is not evidence that the image was evaluated for that category. The adapter should verify that every policy-required category applies to at least one supplied input type. The diagnostic artifact needs the input modality and applicable types, not the user’s raw image.

This near-miss matters because both cases can show a map full of small numbers. The difference is applicability. If the required category lists no applied input type, fix the product contract or choose a classifier that supports the modality. Moving the allow threshold cannot create coverage the model does not provide.

Model drift versus changed traffic. When a score distribution moves, capture the model value returned in the response and the policy version used for the decision. The OpenAI guide states that its moderation model may be upgraded and that score-based custom policies may need recalibration. If the returned model changed while the frozen replay set did not, investigate model-version effects. If the model stayed fixed but production language, region, or use case changed, the evaluation set may no longer represent traffic.

A replay result alone does not tell you which explanation is true. Compare a versioned fixture set under the old and candidate adapters. Segment by the characteristics your policy is allowed to observe, such as locale or input modality, without inventing sensitive labels. Review actual disagreements. A global average can stay flat while one high-impact category gets worse.

Parser defect versus classifier defect. A provider response can be correct while an application adapter reads the wrong key, rounds before comparing, drops slash-containing category names, or converts a missing score to zero. Store the raw provider payload only in a protected diagnostic path and only as long as policy permits. In routine CI, use sanitized fixtures that preserve the exact response shape. Assert both the adapter output and the policy output so the first divergence is obvious.

Queue backlog versus false-positive classifier. A user who waits too long may describe the content as incorrectly blocked, even when the classifier deliberately abstained. Check the reason and the age of the review item before tuning thresholds. Lowering a block or review threshold to solve queue capacity hides an operational problem inside model policy. Staff the queue, reduce the scope that needs automated classification, or design a lower-risk temporary experience.

Structured audit events make these distinctions inspectable. The next script prints one sanitized event and deliberately excludes the user input. Save it as emit_abstention_event.py beside moderation_policy.py. In a service, send the same fields through the project’s approved logger rather than relying on standard output.

import json

from moderation_policy import Evidence, Policy, evaluate, fallback_for


def emit_event(
    case_id: str,
    evidence: Evidence | None,
    policy: Policy,
    transport_error: str | None,
    fallback_status: str,
) -> None:
    outcome = evaluate(evidence, policy, transport_error)
    event = {
        "case_id": case_id,
        "classifier_request_id": evidence.request_id if evidence else None,
        "model": evidence.model if evidence else None,
        "provider_flagged": evidence.flagged if evidence else None,
        "policy_version": policy.version,
        "decision": outcome.decision.value,
        "decision_reason": outcome.reason,
        "highest_category": outcome.highest_category,
        "highest_score": outcome.highest_score,
        "fallback": fallback_for(outcome),
        "fallback_status": fallback_status,
    }
    print(json.dumps(event, sort_keys=True, separators=(",", ":")))


if __name__ == "__main__":
    illustrative_policy = Policy(
        version="illustrative-v1",
        required_categories=("violence", "self-harm"),
        allow_at_most=0.20,
        block_at_least=0.80,
    )
    illustrative_evidence = Evidence(
        request_id="example-001",
        model="example-model",
        flagged=False,
        scores={"violence": 0.55, "self-harm": 0.08},
    )
    emit_event(
        case_id="regression-boundary-001",
        evidence=illustrative_evidence,
        policy=illustrative_policy,
        transport_error=None,
        fallback_status="accepted",
    )

That example prints an illustrative boundary event with decision abstain, reason boundary_band, and fallback_status accepted. If production shows fallback_status rejected, the first investigation goes to routing or queue integration. If the status is accepted but reviewers never see the case, trace the queue consumer. If the decision is allow, compare the evidence against the policy version. Each branch has a different owner and a different corrective action.

Keep operational counts separate:

• Decision counts tell you how much traffic was allowed, blocked, or deferred.
• Reason counts tell you whether deferral came from boundaries, missing evidence, invalid responses, or dependency failures.
• Fallback-status counts tell you whether containment accepted the work.
• Review outcomes tell you how often human labels disagree with the automated boundary decision.
• Labeled replay results tell you unsafe-allow and false-block behavior on a known dataset.

None of those counts proves safety by itself. They answer different questions. Publish a denominator and dataset version with every rate. If traffic volume doubles, an abstention count can double while the rate stays stable. If the mix changes, the same overall rate can hide a category-specific regression.

How to roll the policy into an existing suite

Dropping a new abstain value into a mature system can break callers that assume two states. Treat the change as a migration across the classifier adapter, product router, analytics schema, queue, and release checks.

First, freeze the current behavior in characterization tests. Include a provider-flagged case, a clearly unflagged case, a timeout, a malformed response, and exact threshold boundaries. The purpose is not to approve old behavior. It is to show every intentional change in the diff and prevent unrelated routing from moving at the same time.

Second, introduce the three-state type without changing user behavior. In shadow mode, calculate the new outcome and emit its reason, but continue using the existing route. Never call the real review side effect from shadow traffic. Compare old and new decisions on a versioned sample, then inspect disagreements. This stage reveals how often the old Boolean conflated missing and negative evidence.

Third, label a sample of the boundary band. Use reviewers who can see the source material needed for the decision and give them an escalation path. Record disagreement rather than forcing consensus into the first label. Reviewers can share the same bias, misunderstand a policy category, or become inconsistent under queue pressure. Quality-check the human process as you would any other oracle.

Fourth, select the policy from the labeled set and business consequences. Do not optimize only for overall accuracy. An unsafe allow in a high-impact flow can cost far more than a temporary hold, while a false block in a crisis-support flow can also cause harm. Evaluate category-specific errors, languages, input types, and adversarial variants. Write down which errors the policy prioritizes.

Fifth, canary the route on a small, bounded flow with a staffed fallback. Keep a kill switch that returns to the previous safe behavior, not to unrestricted publish. Watch queue acceptance and age as well as classifier outcomes. A policy that produces excellent classifications but overwhelms reviewers is not ready for wider use.

Finally, make deterministic policy tests a pull-request gate and run live classifier replays on a schedule appropriate to provider drift and release risk. CI should never require a secret for the pure policy suite. The workflow below installs pytest, runs the local safety tests, and uploads JUnit XML even after a failure. In a real repository, replace the unpinned install with the project’s locked dependency command.

name: safety-classifier-policy

on:
  pull_request:
    paths:
      - "moderation_policy.py"
      - "tests/test_moderation_policy.py"
      - ".github/workflows/safety-classifier-policy.yml"

permissions:
  contents: read

jobs:
  policy-contract:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - name: Check out the repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install the test runner
        run: python -m pip install pytest

      - name: Run classifier policy tests
        run: |
          mkdir -p reports
          python -m pytest tests/test_moderation_policy.py \
            --verbose \
            --junitxml=reports/safety-classifier.xml

      - name: Upload the safety test report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: safety-classifier-report
          path: reports/safety-classifier.xml

Pytest documents the junitxml option and its normal exit codes. Preserve the test process exit status so a failed case fails the job. Uploading a report is diagnostic support, not a replacement for the gate. Also distinguish exit code 5, no tests collected, from a pass. A path filter or test discovery mistake must not turn an empty safety lane green.

Keep the scheduled live suite separate. It may need credentials, external network access, retry rules, and a budget. Pin or record the classifier model where the provider offers that control. Save sanitized response fixtures when a disagreement is approved for regression coverage. Do not make pull requests flaky by depending on live scores that can move independently of the code under review.

The migration cost is broader than adding one enum member. Analytics consumers may reject the new value. Dashboards may omit it. Database constraints may require a migration. A mobile client may treat an unknown decision as allow. Contract-test every consumer, and choose an unknown-value behavior that does not release unreviewed content in a high-impact path.

Where abstention causes more harm than it prevents

Deferral is a tool, not a default answer to every imperfect classification. It adds latency, reviewer exposure, queue infrastructure, policy complexity, and a new denial-of-service surface. An attacker may deliberately produce borderline content to exhaust a review team. If the queue is unbounded and the pending path holds expensive resources, a cautious classifier can become an availability incident.

Do not use an uncertainty band to weaken a definite block. When a provider flag or deterministic product rule says the action is forbidden, routing it to routine review may create an override path nobody intended. Reserve exceptional overrides for an explicitly authorized process with audit evidence.

Do not use model uncertainty for conditions a deterministic check can answer. Missing authentication, an invalid capability token, an unauthorized tool, an oversized upload, or a disallowed file type should follow ordinary validation and access-control rules. Sending those cases to a classifier makes a crisp boundary slower and less predictable.

Do not defer low-risk, reversible presentation choices merely because a score is not extreme. A search suggestion, draft label, or optional personalization feature may be able to degrade gracefully without exposing harmful output or blocking a user. The right fallback might be to omit the feature, not create a human-review ticket. Tie the response to the consequence of a wrong decision.

Do not add review when nobody owns the queue. An unstaffed queue is delayed failure. Define service hours, expiry behavior, escalation, and what the user sees while waiting before enabling the route. Test the oldest-item condition and queue rejection. Measure capacity during canary rollout instead of discovering it after launch.

Do not tune thresholds to repair malformed data. If the adapter drops categories, normalizes the wrong text, truncates before classification, or sends an unsupported modality, fix the evidence path. Threshold movement cannot recover information that was never classified.

Do not treat a reviewer as an infallible oracle. Human decisions can drift across shifts, languages, policy updates, and emotionally difficult material. Reviewers may develop automation bias when the model score is visible. Consider hiding the suggested decision during an independent labeling exercise, then reveal it for adjudication. Protect reviewers from unnecessary exposure and provide a route for difficult cases.

Do not expose thresholds or reason details in a way that helps users probe the exact boundary. User-facing messages can explain that content requires review without returning category scores or internal policy values. Internal evidence should remain available to authorized investigators.

Finally, do not store raw prompts and outputs merely because an abstention occurred. Diagnostic value must be weighed against privacy, security, and retention obligations. A case ID, model, policy, category evidence, and fallback result often localize an engineering defect without copying the content into every log sink. When raw material is necessary for review, put it in the approved restricted store and link to it by reference.

A well-designed abstention path is intentionally inconvenient. It spends time or capacity where automation lacks enough evidence. If the team cannot name that cost, observe the fallback, and assign an owner, the new third state will become another silent pass with a better label.