Calibrating Guardrail Thresholds Against Over-Refusal

Guardrail threshold calibration is a risk decision, not a hunt for the highest aggregate score. Lowering a threshold may catch more unsafe cases while refusing legitimate work; raising it may improve utility while allowing severe misses. The suite must show both error distributions and who bears their cost.

The OpenAI Agents SDK guardrail guide defines tripwires and explains where input, output, and function-tool guardrails run. It also distinguishes parallel input checks, where agent work may already begin, from blocking checks that complete before agent execution. Threshold quality and execution placement must be evaluated together.

Define the decision and the error costs

State what a positive classification means. In this guide, positive means the guardrail selects a restrictive action for an unsafe case. A true positive restricts unsafe content; a false positive restricts safe content and creates over-refusal; a false negative allows an unsafe case; a true negative allows a safe case.

Do not hide several actions behind one "blocked" label. The policy may allow, allow with warning, transform, request clarification, require approval, route to a safer tool, escalate for review, or refuse. Each has different user and safety consequences. Calibration chooses score regions for those actions, not merely a boolean.

Create a severity table before seeing candidate results. A miss involving an irreversible tool action may carry a harder constraint than a miss in low-impact generated prose. Likewise, refusing an accessibility request, a benign educational discussion, or an urgent support workflow can have unequal user cost.

Build an adjudicated calibration corpus

Use policy-authored examples, historical incidents with appropriate privacy treatment, safe near-neighbors, adversarial paraphrases, and ordinary traffic slices. Unsafe examples alone cannot measure over-refusal. Random safe requests alone rarely stress the semantic boundary.

Label category, severity, expected action, acceptable alternative actions, rationale, ambiguity, language, conversation dependence, tool availability, and source. Give reviewers the full evidence the guardrail receives. If the online guardrail sees only the latest user message, do not label with hidden history and then call its failure a model defect.

Resolve disagreement through a recorded adjudication process. Keep difficult disagreements in the set with an ambiguity marker; deleting them creates artificial confidence. Split threshold-selection data from a locked confirmation set so repeated tuning does not optimize every decision to known examples.

Preserve raw scores and provenance

Store category scores before thresholding, the thresholded action, guardrail revision, policy revision, input transformation, execution mode, latency, error state, and case ID. A missing score due to timeout cannot be silently treated as safe. Give dependency failure its own result and a fail-open or fail-closed policy approved for that operation.

Scores from different categories or classifier revisions may not be comparable. A value of 0.8 is not automatically an 80 percent probability and does not carry the same meaning across models. Calibration evidence must be refreshed when prompts, labels, model, preprocessing, language mix, or traffic source changes.

Keep the original score at enough precision to rerun threshold analysis. Do not store only the final blocked flag; that forces expensive re-execution and makes historical policy reconstruction impossible.

Sweep thresholds with complete confusion counts

For each candidate threshold, count true positives, false positives, true negatives, and false negatives. Calculate unsafe recall, block precision, safe false-refusal rate, and workload for review or clarification. Always print denominators, especially for rare high-severity categories.

This Python example is deliberately framework-neutral:

def metrics(rows: list[dict], threshold: float) -> dict:
    tp = fp = tn = fn = 0
    for row in rows:
        restrict = row["score"] >= threshold
        unsafe = row["label"] == "unsafe"
        tp += int(restrict and unsafe)
        fp += int(restrict and not unsafe)
        tn += int(not restrict and not unsafe)
        fn += int(not restrict and unsafe)

    return {
        "threshold": threshold,
        "tp": tp, "fp": fp, "tn": tn, "fn": fn,
        "unsafe_recall": tp / (tp + fn) if tp + fn else None,
        "block_precision": tp / (tp + fp) if tp + fp else None,
        "safe_false_refusal_rate": fp / (fp + tn) if fp + tn else None,
    }

Use confidence intervals or resampling appropriate to the sampling design. Repeated variants from one seed case are correlated and should not masquerade as independent coverage. A threshold with apparently perfect critical recall on a tiny set is uncertain, not proven safe.

Examine false refusals as product failures

Read false positives, not just their count. Group them by intent, risk category, language, dialect, user role, response format, conversation length, quoted material, and tool request. Over-refusal often clusters where policy examples are sparse or where contextual discussion resembles prohibited intent lexically.

Measure the full handling outcome. A clarification that lets the user proceed is less harmful than a terminal refusal, but repeated clarification can still make a workflow unusable. Record extra turns, abandonment where legitimately observable, human-review load, and whether the safe task eventually completes. Do not invent causal claims from a simple correlation.

Build contrast sets around each false refusal: preserve the safe intent while changing superficial words, then preserve risky intent while removing obvious trigger terms. A robust threshold should rely on policy-relevant features, not a fragile phrase list.

Protect critical slices without one global cutoff

A global threshold can satisfy aggregate metrics while failing a small severe slice. Establish constraints per risk category, severity, language, and tool class. For deterministic invariants such as forbidden tool arguments, tenant mismatch, or missing approval, use code checks rather than asking a probabilistic score to carry the whole boundary.

Category-specific thresholds are justified only when categories are reliably identified and have enough adjudicated evidence. Otherwise complexity can hide routing errors. Track the first-stage category decision and the category threshold separately.

Consider a three-zone policy: clear allow, uncertain review or clarification, and clear restriction. The middle zone increases operational work but makes uncertainty visible. Set its width from evidence and staffing constraints, then test queue overflow and reviewer disagreement.

Test where the guardrail executes

Agent-level input guardrails, final-output guardrails, and function-tool guardrails protect different workflow boundaries. The SDK documentation notes that input guardrails run for the first agent in a chain and output guardrails for the final-output agent, while tool guardrails wrap each supported custom function-tool invocation. A multi-agent test must verify coverage at every actual effect boundary.

Parallel input execution can reduce added latency, but the agent may consume tokens or execute tools before a tripwire is observed. For risky tools, cancellation after dispatch is not prevention. Test with a barrier tool that records whether execution began before the guardrail decision. Use blocking mode or a separate pre-effect control where policy requires no action before classification.

Output checks happen after generation, so they cannot recover generation cost and may need a safe replacement response. Inject guardrail timeout, exception, malformed result, and disagreement between input and output checks. Each state needs a deterministic handling policy.

Choose and document an action policy

The following values are illustrative thresholds only, not recommendations:

{
  "policyLabel": "illustrative thresholds only",
  "category": "restricted-tool-intent",
  "allowBelow": 0.28,
  "reviewFrom": 0.28,
  "restrictFrom": 0.73,
  "criticalDeterministicVeto": true,
  "timeoutDisposition": "deny-tool-and-offer-safe-path",
  "minimumReviewSamplePerSlice": "set-by-risk-owner"
}

Version the score producer and action policy independently. The same scores may be replayed under a candidate policy; a new classifier requires fresh score collection. Record approver, supported population, excluded slices, rollback trigger, and expiry date for the calibration decision.

Validate offline and in production shadow mode

The LangSmith evaluation concepts guide distinguishes offline evaluation on curated datasets from online evaluation on production runs and threads. Use offline sets for controlled threshold sweeps and locked regression cases. Use policy-eligible shadow scoring to discover current slices and score drift without changing the user response.

Compare candidate and current policy on the same events. Review all critical disagreements, a stratified sample of ordinary disagreements, and a random sample of agreements to detect shared blind spots. Protect sensitive inputs and do not retain them merely because evaluation is useful.

Canary the action policy with immediate rollback. Monitor safe refusals, unsafe misses from reviewed incidents, uncertainty-zone volume, classifier errors, latency, tool starts before tripwire, and slice score distributions. Production reports supplement adjudicated tests; they do not magically provide ground-truth labels.

Use a failure model for release decisions

Also test threshold equality, floating-point serialization, multiple category scores, ties, contradictory guardrails, retries, and policy rollback. Boundary values deserve explicit fixtures because > versus >= can change the action.

Execute the calibration plan

1. Define positive class, action set, severity, false-negative harm, and false-refusal harm before comparing thresholds.
2. Build adjudicated unsafe, safe near-boundary, ordinary, multilingual, conversational, tool, and adversarial cases with a locked confirmation split.
3. Retain raw scores, missing results, provenance, revisions, and execution placement for every case.
4. Sweep thresholds with full confusion counts, uncertainty, severity, and slice constraints; inspect false refusals qualitatively.
5. Add deterministic vetoes and guardrails at the actual side-effect boundaries instead of relying on one classifier.
6. Shadow the candidate, review disagreements, canary the versioned action policy, and keep rollback immediate.
7. Block release on critical misses, unhandled classifier failure, material slice over-refusal, pre-tripwire side effects, or untraceable policy decisions.

Good calibration does not promise zero mistakes. It makes the allowed tradeoff explicit, tests the people and workflows harmed on both sides, and prevents a convenient average from deciding a safety policy by accident.