Catch the agent failures that averages hide

An illustrative release candidate clears its average eval threshold, yet one policy-sensitive case calls the same tool six times before timing out. The dashboard says pass because common cases dominate the mean. A QA engineer should stop promotion long enough to learn whether that lonely result is bad data, a rare product defect, or evidence that the runs were grouped incorrectly.

See what the average has hidden

A mean answers a narrow question: what value do these observations balance around? It does not tell you whether one run is far from its peers, whether the distribution has two distinct clusters, or whether a low-volume safety slice failed. An agent can improve its average answer score while introducing a retry loop that affects only one tool path. It can also keep its median latency steady while a few calls wait on a dependency until the request budget is exhausted.

Outlier analysis starts with a comparison set. An observation is unusual only relative to some collection of observations that represents its expected context. Mixing unrelated work destroys that context. A retrieval question, a browser task, and an account-maintenance request can have different tool counts and duration even when all three behave correctly. Put them in one distribution and the most complex capability may look anomalous simply because it does more work.

Build cohorts around factors that plausibly change the metric. Capability, risk class, tool path, model and prompt revision, evaluator revision, region, and runtime class are useful candidates. Do not split on every available field, because a cohort of one cannot describe normal variation. Start with the coarsest defensible group, plot or list the observations, and split when the evidence shows a mixture. The grouping decision is part of the test design, not an innocent reporting choice.

The raw unit matters too. One row should represent one defined attempt. If the report stores the best of three retries, the tail has already been removed. If it stores an average across retries, a transient failure has been diluted. Keep the attempt identifier, case identifier, input revision, start and end times, tool events, final result, and evaluator output. Summaries should be derived from those rows, never substituted for them.

NIST defines an outlier as an observation an abnormal distance from other values in a sample, while stressing that normal behavior must first be characterized. That wording is useful for agent testing because it does not equate unusual with wrong. A trace can be statistically ordinary and still violate policy. Another trace can be statistically extreme because a telemetry field used the wrong unit, even though the agent behaved correctly.

Keep three questions separate during a release review:

1. Is this observation unusual relative to comparable runs?
2. Is the observation valid, or did collection and evaluation produce it incorrectly?
3. Does the underlying behavior violate a product, safety, cost, or reliability requirement?

The first is statistical labeling. The second is evidence validation. The third is a release decision. A modified z-score can help with the first question. It cannot answer the other two without trace evidence and an explicit policy.

That separation also explains why a rare severe event is not merely an outlier. Suppose one test causes an unauthorized side effect. Its severity comes from the side effect, not its distance from a median. The event blocks if the approved policy says any unauthorized side effect blocks. Whether the same event appears once, ten times, or in a zero-MAD binary series changes the investigation, but not the contract.

Conversely, a long but correct run may deserve investigation without proving a release defect. Perhaps the test runner shared a saturated host. Perhaps the fixture forced a larger document than its cohort. Perhaps a real model change made one reasoning path much longer. The evidence has to distinguish these causes before the team tunes a threshold or edits a prompt.

A useful release report therefore carries more than an overall score. It retains each raw observation, its cohort keys, the method and parameters used to label candidates, the reason any candidate was excluded or retained, and the final disposition. That record lets another engineer reproduce the label and challenge the decision. A screenshot of a red dot cannot do that.

Choose a statistical rule the data can support

Mean and standard deviation are tempting because most dashboards already calculate them. They are a poor first choice when the point you are looking for can pull both quantities toward itself. A single extreme duration increases the mean and the standard deviation, which can make its own standardized distance look less impressive. Small samples add another problem: an apparent shape may depend heavily on one or two observations.

Median absolute deviation, usually shortened to MAD, is a practical labeling method for one-dimensional release metrics. First compute the median, m, of the values. Then take each absolute distance from that median and compute the median of those distances. In notation, MAD = median(|x_i - m|). NIST's measures-of-scale guidance describes why MAD and interquartile range are less affected by extreme tail values than standard deviation.

For a nonzero MAD, calculate the modified z-score 0.6745 * (x_i - m) / MAD. NIST's outlier-detection page reports the Iglewicz and Hoaglin recommendation to label observations with an absolute modified z-score above 3.5 as potential outliers. The words “label” and “potential” matter. The result nominates a run for investigation. It does not establish root cause, defect severity, or release quality.

Direction should match the metric. High latency, tool attempts, and cost are normally investigated on the upper side. A success score or groundedness score may be investigated on the lower side. A metric such as a signed calibration error may need both tails. Using an absolute value for every metric creates noise by flagging unusually good outcomes that pose no release risk. Using only the upper tail for every metric can miss the failure entirely.

The observations must also be meaningfully comparable. A one-dimensional MAD calculation does not adjust for prompt length, tool choice, region, or cache state. If those factors create expected differences, stratify or model them before applying the label. Repeated calls that share the same outage are not independent evidence simply because they have different attempt IDs. Keep the shared incident or batch identifier so reviewers can see that dependence.

Agent measurements often violate the clean assumptions behind formal outlier tests. Durations can be right-skewed. Tool counts are discrete. Scores are bounded and frequently tied. NIST notes that distributional assumptions affect outlier identification and that non-normality can be mistaken for outliers. Treat modified z-scores here as a robust exploratory label, not a hypothesis test with a claimed false-positive rate. If you need a formal statistical conclusion, involve someone who can justify the model for the actual data.

Small samples need an explicit inconclusive state. Three fast runs and one slow run do not provide a stable picture of a tail. There is no universally correct minimum, so the code below uses twelve as an illustrative release-policy setting rather than a scientific constant. A team may choose a different number based on repetition cost, expected variability, and risk. The important behavior is that the gate does not call a tiny sample clean.

Zero MAD needs the same honesty. It occurs when enough observations tie at the median that the median absolute deviation is zero, which is easy with repeated discrete values. Dividing by zero is invalid. Replacing MAD with an arbitrary epsilon can turn a one-unit difference into an enormous score whose size reflects the epsilon, not evidence. Return a degenerate state, list values that differ from the median for inspection, and use a domain rule or gather richer data.

The following standard-library script performs those checks. Save it as outlier_gate.py. It reads a JSON document containing observations, an evidence_complete boolean, and a list of hard_failures. Exit code 0 means the statistical evidence is clear, 2 means review is required, and 3 means an independent release contract failed. Those codes describe workflow state, not a universal judgment of model quality.

from __future__ import annotations

import argparse
import json
import math
import statistics
import sys
from pathlib import Path
from typing import Any

SCALE = 0.6745


def analyze(
    observations: list[dict[str, Any]],
    direction: str,
    threshold: float,
    min_sample: int,
) -> dict[str, Any]:
    if direction not in {"high", "low", "both"}:
        raise ValueError("direction must be high, low, or both")
    if not math.isfinite(threshold) or threshold <= 0:
        raise ValueError("threshold must be positive and finite")
    if min_sample < 3:
        raise ValueError("min_sample must be at least 3")

    rows: list[dict[str, Any]] = []
    for item in observations:
        value = float(item["value"])
        if not math.isfinite(value):
            raise ValueError(f"observation {item['id']} is not finite")
        rows.append({"id": str(item["id"]), "value": value})

    if len(rows) < min_sample:
        return {
            "state": "insufficient_sample",
            "sample_size": len(rows),
            "minimum_sample": min_sample,
            "candidates": [],
        }

    values = [row["value"] for row in rows]
    center = statistics.median(values)
    deviations = [abs(value - center) for value in values]
    mad = statistics.median(deviations)

    if mad == 0:
        return {
            "state": "degenerate",
            "sample_size": len(rows),
            "median": center,
            "mad": mad,
            "different_from_median": [
                row["id"] for row in rows if row["value"] != center
            ],
            "candidates": [],
        }

    scored = []
    for row in rows:
        score = SCALE * (row["value"] - center) / mad
        is_candidate = (
            (direction == "high" and score > threshold)
            or (direction == "low" and score < -threshold)
            or (direction == "both" and abs(score) > threshold)
        )
        if is_candidate:
            scored.append(
                {
                    "id": row["id"],
                    "value": row["value"],
                    "modified_z": round(score, 4),
                }
            )

    return {
        "state": "candidates_found" if scored else "clear",
        "sample_size": len(rows),
        "median": center,
        "mad": mad,
        "threshold": threshold,
        "direction": direction,
        "candidates": scored,
    }


def decide(
    analysis: dict[str, Any],
    hard_failures: list[dict[str, Any]],
    evidence_complete: bool,
) -> dict[str, Any]:
    if hard_failures:
        return {
            "status": "block",
            "reason": "independent release contract failed",
            "hard_failures": hard_failures,
        }
    if not evidence_complete:
        return {"status": "review_required", "reason": "evidence is incomplete"}
    if analysis["state"] != "clear":
        return {
            "status": "review_required",
            "reason": f"statistical state is {analysis['state']}",
        }
    return {"status": "clear", "reason": "no unresolved gate evidence"}


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("input", type=Path)
    parser.add_argument("--direction", choices=("high", "low", "both"), required=True)
    parser.add_argument("--threshold", type=float, default=3.5)
    parser.add_argument("--min-sample", type=int, default=12)
    args = parser.parse_args()

    payload = json.loads(args.input.read_text(encoding="utf-8"))
    analysis = analyze(
        payload["observations"],
        args.direction,
        args.threshold,
        args.min_sample,
    )
    decision = decide(
        analysis,
        list(payload.get("hard_failures", [])),
        payload.get("evidence_complete") is True,
    )
    print(json.dumps({"analysis": analysis, "decision": decision}, indent=2))
    return {"clear": 0, "review_required": 2, "block": 3}[decision["status"]]


if __name__ == "__main__":
    sys.exit(main())

This implementation deliberately refuses two seductive shortcuts. It does not delete candidate points and recompute until the report turns green. That iterative pattern can mask several tail values. It also does not fall back from zero MAD to standard deviation without telling the reviewer. Changing the estimator changes the meaning of the gate and belongs in a reviewed policy revision.

The input below is an illustrative fixture, not collected performance data. Save it as fixtures/candidate-tool-duration.json. The block uses JSON syntax, which is also valid YAML, so the standard-library JSON parser can read it without another dependency.

{
  "metric": "tool_duration_ms",
  "cohort": {
    "capability": "inventory_lookup",
    "runtime": "isolated_linux"
  },
  "evidence_complete": true,
  "hard_failures": [],
  "observations": [
    {"id": "attempt-00", "value": 98},
    {"id": "attempt-01", "value": 100},
    {"id": "attempt-02", "value": 101},
    {"id": "attempt-03", "value": 99},
    {"id": "attempt-04", "value": 102},
    {"id": "attempt-05", "value": 97},
    {"id": "attempt-06", "value": 103},
    {"id": "attempt-07", "value": 100},
    {"id": "attempt-08", "value": 101},
    {"id": "attempt-09", "value": 99},
    {"id": "attempt-10", "value": 104},
    {"id": "attempt-11", "value": 280}
  ]
}

For that fixture, the script reports candidates_found, a median of 100.5, a MAD of 1.5, and attempt-11 as the only candidate. Its modified z-score is 80.7152. Those figures are deterministic outputs from the illustrative values above, not performance measurements from an agent experiment.

IQR is a reasonable alternative for exploratory labeling. Compute the first and third quartiles, subtract them to get IQR, then apply a fixed fence convention such as Q3 + 1.5 * IQR for a high-side label. NIST documents those box-plot fences, but software packages can calculate quartiles differently for small samples. Freeze the quantile convention in code and tests. Switching between MAD and IQR after seeing which one passes the candidate is threshold shopping.

Diagnose the tail before changing the gate

A candidate label should lead to a case ID, not a vague warning that “latency is anomalous.” Start with the exact input row and reconstruct the run from retained evidence. Check the cohort fields first. A mislabeled capability or runtime class can make a healthy observation look distant from unrelated peers.

Then check the measurement path. Confirm that all values use the same unit and clock source, that start and end events exist, and that a retry was not counted both as a child attempt and as part of its parent. Inspect the raw value before any dashboard conversion. A duration of 7.4 can mean seconds in one producer and milliseconds in another; a chart that assumes one unit can create an impressive but meaningless tail.

Next inspect the trajectory. Find the first event where the candidate differs from a normal peer in the same cohort. Useful evidence includes the selected tool, normalized arguments, response status, retry reason, backoff interval, evaluator input, evaluator output, and termination reason. Redact secrets and personal data, but keep stable hashes or approved identifiers that let investigators join the records. A final timeout message alone cannot show whether the agent looped, a tool stalled, or the evaluator waited on a missing event.

Run the same fixture once under the original conditions before changing anything. This checks basic reproducibility. Then vary one suspected cause at a time: pin the dependency response, move to an isolated runner, or restore the baseline prompt. Do not repeatedly rerun until a pass appears and then discard the failures. Each attempt is evidence about variation and needs its own row.

The script's states make the diagnostic branch explicit. candidates_found means one or more observations crossed the configured modified z-score threshold. insufficient_sample means the method was not applied. degenerate means MAD was zero and the score was undefined. clear means no candidate crossed this statistical rule in this cohort; it does not mean every product requirement passed.

This shell sequence preserves the report even when the command places promotion on hold. It also prints the JSON with Python's built-in formatter, so a reviewer sees the case IDs and calculation inputs rather than only a failing job name.

set +e
mkdir -p artifacts
python outlier_gate.py fixtures/candidate-tool-duration.json \
  --direction high \
  --threshold 3.5 \
  --min-sample 12 \
  > artifacts/tool-duration-outliers.json
gate_status=$?
set -e

python -m json.tool artifacts/tool-duration-outliers.json
exit "$gate_status"

Read an ordinary pytest failure just as concretely. Pytest rewrites plain assertions to display compared values. If a test expected review_required and received clear, the report should show those strings and the failing line. That is stronger evidence than a test whose assertion repeats a constant already embedded in the fixture.

Ask what input mutation makes each oracle fail. Increasing the last duration should make the high-tail test cross the threshold. Replacing it with a value near the median should remove the candidate and change the decision. Adding a policy failure should produce block even when every duration is identical. If none of those changes affect an assertion, the test is not protecting the behavior described by the article.

Do not “repair” the report by widening the cohort or raising 3.5 after inspecting the release candidate. A parameter change can be valid when accumulated evidence shows the original method does not fit the data. Make that change on historical or held-out observations, record the rationale, and apply it to baseline and candidate alike. A threshold selected to rescue one build is not a gate.

Plotting remains useful even when CI uses a numeric label. A sorted dot plot can reveal two clusters, a long continuous tail, or values rounded into ties. Those shapes demand different treatment. MAD reduces the influence of extremes on center and spread, but it cannot tell you why the shape exists. The plot guides the investigation; the retained rows make it auditable.

Separate product defects from convincing near-misses

Consider an illustrative tool-duration fixture with twelve comparable attempts. Eleven complete within a narrow band, while inventory-017-attempt-1 takes much longer and records repeated calls to the same lookup tool. The modified z-score labels that attempt. The label becomes product evidence only after the trajectory shows that the model received a successful tool response, ignored it, issued the equivalent request again, and exhausted the run budget.

The fix in that case might be a clearer tool-result contract, a deterministic duplicate-call guard, or a corrected termination condition. Each option costs something. A duplicate-call guard can reject a legitimate second query whose arguments were normalized too aggressively. A prompt change can alter unrelated behavior and requires broad reevaluation. A termination limit prevents an endless loop but may cut off a difficult case that would have recovered on the next step. The release owner chooses among those costs using the trace, not the z-score.

Now take a near-miss with almost identical output. One duration is far above its cohort, but the trace contains a single tool call and an ordinary response. The raw events reveal that one producer wrote microseconds into a field documented internally as milliseconds. The agent did not regress. The telemetry contract did. The correct action is to repair or quarantine the invalid measurement, backfill only when conversion can be proved, and rerun the analysis. Deleting the point without recording the reason would hide a broken evidence pipeline.

A second near-miss comes from a mixed population. Suppose the same capability can use a local cache or a remote document service. The sorted values form two stable groups, and the “outliers” all used the remote path. Those runs may be normal for that path. Add the dependency path to the cohort key, verify that each new group has enough observations, and analyze them separately. If the release requirement applies to end-to-end latency regardless of path, test that explicit limit as a separate service-level gate. Statistical regrouping must not erase a user-facing deadline.

Policy events create the opposite trap. An illustrative dataset records whether an account-deletion action ran without the required review: eleven zeros and one one. MAD is zero, so a modified z-score is undefined. Calling the one an ordinary value because the detector returned no score would be indefensible. The approved contract should inspect the event sequence and block on any confirmed bypass. Statistics can report frequency and uncertainty later, but it is not the oracle for the forbidden transition.

The tests below exercise those distinctions. They use values created solely to test code branches, not measurements from a real evaluation. Each assertion can fail when the implementation or input changes: lowering the extreme value removes its label, silently passing zero MAD changes the decision, and removing the hard failure changes block to another state.

from outlier_gate import analyze, decide


def rows(values: list[float]) -> list[dict[str, object]]:
    return [
        {"id": f"attempt-{index:02d}", "value": value}
        for index, value in enumerate(values)
    ]


def test_high_tail_value_requires_review() -> None:
    values = [98, 100, 101, 99, 102, 97, 103, 100, 101, 99, 104, 280]
    analysis = analyze(rows(values), "high", threshold=3.5, min_sample=12)

    assert analysis["state"] == "candidates_found"
    assert [item["id"] for item in analysis["candidates"]] == ["attempt-11"]
    assert decide(analysis, [], evidence_complete=True)["status"] == "review_required"


def test_near_median_replacement_clears_the_statistical_rule() -> None:
    values = [98, 100, 101, 99, 102, 97, 103, 100, 101, 99, 104, 102]
    analysis = analyze(rows(values), "high", threshold=3.5, min_sample=12)

    assert analysis["state"] == "clear"
    assert analysis["candidates"] == []
    assert decide(analysis, [], evidence_complete=True)["status"] == "clear"


def test_zero_mad_is_inconclusive_instead_of_clean() -> None:
    values = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 7]
    analysis = analyze(rows(values), "high", threshold=3.5, min_sample=12)

    assert analysis["state"] == "degenerate"
    assert analysis["different_from_median"] == ["attempt-11"]
    assert decide(analysis, [], evidence_complete=True)["status"] == "review_required"


def test_policy_failure_blocks_even_when_metric_is_typical() -> None:
    analysis = analyze(rows([10, 11, 9, 10, 12, 8, 10, 11, 9, 10, 12, 8]), "high", 3.5, 12)
    failures = [
        {
            "case_id": "account-close-017",
            "rule": "review_must_precede_account_closure",
            "evidence": ["request_received", "close_account"],
        }
    ]

    decision = decide(analysis, failures, evidence_complete=True)

    assert analysis["state"] == "clear"
    assert decision["status"] == "block"
    assert decision["hard_failures"][0]["case_id"] == "account-close-017"

There is another product failure that a univariate detector can miss: a coherent shift in the whole cohort. If every candidate duration gets slower by roughly the same amount, none may be distant from the candidate median. Outlier analysis asks about unusual points within a set, not whether the set moved relative to baseline. Pair it with a baseline comparison, a reviewed absolute requirement, or an interval estimate suited to the metric. Do not claim that an empty candidate list proves no regression.

The reverse is also possible. One run can be unusual within both baseline and candidate because the fixture intentionally exercises a large input. If its behavior and requirement are stable, it belongs in a named boundary slice rather than in the ordinary cohort. Preserve it as coverage. Do not delete the hard case merely to make the distribution easier to summarize.

Put anomaly evidence into a release decision

A defensible gate has at least three outcomes. clear means required evidence exists, independent contracts passed, and this outlier rule found no unresolved candidates. review_required means promotion pauses because the sample is too small, MAD is zero, evidence is missing, or a candidate needs disposition. block means a separate release contract has failed, such as an unauthorized action or a confirmed deadline breach. Teams may use different labels, but they need the distinction.

This design prevents anomaly detection from masquerading as a quality score. A statistical candidate does not subtract arbitrary points from “agent quality.” It opens an investigation linked to the raw run. Once triaged, the release record should mark the result as a confirmed product defect, invalid measurement, expected boundary case, cohort error, dependency incident, or unresolved. Add another disposition only when it carries a clear next action.

Promotion can resume after an invalid measurement is corrected and the affected evidence is regenerated. A dependency incident may require a rerun on a healthy environment plus a separate reliability decision about dependency behavior. A confirmed product defect follows the relevant severity and waiver policy. An expected boundary case may be moved into a named cohort, but the new grouping must be applied consistently to historical baseline data.

CI should retain the report even when the analysis command exits nonzero. The following GitHub Actions job uses documented action versions and uploads the JSON on both passing and failing runs. It assumes the script, fixture, and tests shown above are committed in the repository. The fixture values remain explicitly illustrative until a team replaces them with versioned evaluation evidence.

name: Agent release evidence

on:
  pull_request:
  workflow_dispatch:

jobs:
  outlier-evidence:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install test dependency
        run: python -m pip install pytest
      - name: Prove the gate behavior
        run: python -m pytest tests/test_outlier_gate.py -q
      - name: Analyze comparable candidate runs
        run: >-
          python outlier_gate.py fixtures/candidate-tool-duration.json
          --direction high
          --threshold 3.5
          --min-sample 12
          > outlier-report.json
      - name: Retain gate evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: agent-outlier-report
          path: outlier-report.json
          if-no-files-found: error

GitHub's Python CI documentation recommends setup-python for consistent Python versions on hosted runners. Consistency matters here because a gate should not pick up an unreviewed runtime change from the runner image. Pin application dependencies as well. The example installs only pytest because the analyzer itself uses the Python standard library.

Roll this gate out in observation mode first. For several representative candidate and baseline runs, calculate labels and require engineers to disposition them without stopping promotion. This period reveals bad cohort keys, missing events, unit disagreements, and frequent zero-MAD slices. It does not prove future accuracy, but it exposes whether the evidence needed for a decision is available.

Freeze the first enforcement policy before using it on a release. Record the cohort definition, metric direction, minimum sample, threshold, code revision, and owners who can approve a change. Run the same policy on baseline and candidate evidence. Keep a separate policy for deterministic contracts so a statistical configuration edit cannot weaken a safety rule by accident.

Move one risk area at a time from observation to enforced review. Start where trace completeness is high and investigators understand normal behavior. A global rollout creates a queue of unlabeled red jobs that teaches developers to rerun or override the gate. A narrow rollout lets the team measure operational cost such as review time and additional repetitions without pretending those costs are model performance measurements.

Every override needs a case-specific reason, owner, and expiry. “Known outlier” is not enough. State whether the evidence was invalid, the cohort was wrong, the behavior was accepted under a named exception, or the dependency incident is tracked elsewhere. Preserve the original report and the post-correction report. Rewriting the artifact removes the audit trail that explains why promotion continued.

Expect real costs. Retaining attempt-level traces uses storage and requires careful redaction. Repetitions consume model and tool capacity. Small cohorts stay inconclusive longer. Manual triage adds release latency. More cohort keys reduce false comparisons but can starve each group of data. Those costs are a reason to scope the gate by risk and improve evidence collection, not a reason to collapse everything back into one average.

The method also creates a maintenance obligation. New tools, routing policies, model versions, and runner classes can change a distribution legitimately. Schedule reviews based on system changes and accumulated dispositions, not only when a candidate fails. Historical decisions should remain reproducible with the policy version that made them, even after the current policy evolves.

Know when outlier detection is the wrong tool

Skip statistical labeling when a direct requirement answers the release question. If a production change must never execute before human review, test the order of those events. If a run must finish within a contractual deadline, compare the measured duration with that deadline after validating the measurement. A median-based rule can supplement those checks, but it should not replace them with a relative standard that moves as the product changes.

Do not apply MAD to categorical outcomes such as refusal reason, tool name, or final state. Encoding categories as numbers does not create meaningful distance. Test allowed sets, state transitions, schema contracts, and frequencies with methods suited to categorical data. A refusal code numbered 9 is not farther from code 2 in any useful statistical sense than code 3 is.

Avoid the method when the cohort is intentionally heterogeneous and cannot be split without losing the release question. An end-to-end user journey may contain fast cache hits and slower remote searches by design. If users care about a percentile or deadline across that mixture, test that service objective directly. If engineers need component diagnosis, analyze the named paths separately after the end-to-end gate has been evaluated.

Very small slices should remain visible but inconclusive. Safety and boundary cases are often rare precisely because the dataset curates them rather than samples a population. Give those cases explicit expectations. Running a modified z-score over four handpicked cases adds mathematical decoration without reliable characterization of normal behavior.

Nonstationary systems need different treatment too. If traffic, tool providers, or model routing changed during collection, the observations may not share one stable process. A point from before the change can look unusual after it even when both regimes are healthy. Segment at the known change, use time-aware monitoring, and compare like with like. Do not let a wider rolling window blur a release-caused shift.

Multivariate failures can hide in ordinary one-dimensional values. Tool count and duration may each look normal while their combination is strange for a capability. A simple MAD gate will miss that relationship. Start with trace-based rules when the risky combination is known. Use a reviewed multivariate method only when the team can explain its assumptions, validate it on relevant evidence, and give investigators enough detail to reproduce a label.

Heavy tails deserve special caution. If rare long runs are a stable feature of the process, a symmetric modified z-score threshold can produce a stream of unsurprising labels. Inspect the shape and consider a transformation only when the transformed model is justified. NIST notes that a log transform can be appropriate for lognormal data, but positive values alone do not prove lognormality. A quantile-based operational objective may match the release question better.

Do not use an outlier detector to make a weak suite look sophisticated. If every observation is poor, none has to be unusual. If the evaluator rewards plausible text while ignoring forbidden tool actions, better statistics preserve the wrong oracle. Fix coverage, contracts, and evidence first. Anomaly analysis earns its place only after a labeled run leads an engineer to a decision they could defend from the underlying trace.

Finally, resist automatic deletion. NIST advises investigating outliers carefully because they may reveal useful process or data-collection information. In agent testing, the farthest point may be the only fixture that exposes a retry loop, a broken clock conversion, or a new routing path. Preserve it, explain it, and let an explicit release policy decide what happens next.