When a voice agent talks too soon, too late, or over the caller

A caller finishes saying their account number, waits, says "hello?", and the agent begins answering over that second prompt. The backend dashboard shows a quick model response, so nobody owns the delay. Listening to the recording makes the failure obvious, but it does not say whether endpoint detection, synthesis, transport, or playback caused it. A useful test has to preserve that distinction.

What makes a fast backend feel slow

People experience a voice turn as one continuous exchange. The implementation is a chain of decisions and buffers. Audio arrives from the caller, speech activity is classified, a turn is committed, recognition produces usable text, the application decides what to do, response audio is synthesized, packets travel back, a client buffer releases them, and a speaker finally plays them. Measuring only one link can make a broken call look healthy.

Time to first model token is the classic misleading metric. It can be valuable when comparing model execution, but the caller does not hear a token. A text token may arrive while the system is still waiting for a speech endpoint, while synthesis is accumulating enough text to sound natural, or while audio is queued behind already buffered media. Calling that number "voice latency" gives the fastest component credit for work that remains unfinished.

Start with the two acoustic boundaries the user can perceive. The first is the end of the caller's intended utterance. The second is the first assistant sample that becomes audible on the caller's output path. The difference is the response gap. It is positive when there is silence between speakers and negative when the assistant starts before the caller is done. That signed value exposes both sluggishness and premature interruption, which are different bugs hidden by an absolute duration.

Those boundaries need operational definitions. "Caller finished" might mean the final active audio frame in a labeled recording, the speech detector's end event, or the moment a human annotator believes the thought is complete. They are not interchangeable. The final active frame is useful as an offline reference. The detector event shows what the running system believed. A semantic annotation can identify a pause that sounded like an endpoint but was actually inside a sentence. Store all three when the test needs to assess detector quality.

The same discipline applies to the assistant boundary. A synthesis service reporting its first chunk does not prove that sound played. A packet received by the browser does not prove that the jitter buffer released it. An application callback that schedules a buffer does not prove that the audio device rendered it. In a controlled test, assistant_playout_started can come from the output capture crossing its calibrated activity threshold. Production instrumentation may reach only scheduling, in which case speaker_first_sample_scheduled is more honest than audio_heard.

Turn-taking adds a second axis beyond response delay. When the caller begins speaking during an assistant response, many products promise some form of barge-in. The relevant interval starts at the caller's new audible speech and ends when assistant playout stops. A quick cancellation acknowledgement from the server is not the result. Buffered sound may continue on the client, so the caller can still hear a sentence after the backend says cancellation succeeded.

Do not combine early interruption and failed barge-in into one "overlap" score. In an early-interruption case, the assistant starts while the caller owns the floor. In a barge-in case, the assistant owned the floor first and the caller deliberately takes it. The same waveform can contain simultaneous speech, but the expected behavior depends on which transition happened. Every fixture needs the intended floor owner and the event that changed it.

Clock choice matters before any threshold does. Within one browser or worker, performance.now() is tied to a monotonic time origin, so system clock corrections do not make a duration run backward. MDN's performance.now() reference documents both that monotonic property and the fact that the value is relative to a time origin. That makes it suitable for intervals recorded in one execution context, not for subtracting a service timestamp from an unrelated browser timestamp. For a distributed path, either calculate component durations where each clock is local or use tracing infrastructure with an explicit clock-synchronization model.

Sequence numbers are a useful companion to timestamps. They tell you that the turn was committed before synthesis started even when events came from different machines. They do not recover an accurate cross-machine duration. Treat sequence as ordering evidence and time as duration evidence. Mixing those jobs produces precise-looking numbers that have no trustworthy zero point.

One number should never carry the release decision by itself. A short response gap can be terrible if the agent clips the caller's final word. A longer gap can be correct when a payment tool must finish or a safety confirmation must be spoken in full. Keep endpointing delay, application work, synthesis startup, delivery delay, and audible gap as separate fields. Then attach policy to the scenario rather than pretending every turn has identical work.

Record the boundaries that can prove the fault

A testable event record needs a session identifier, turn identifier, event name, timestamp, and the component that observed it. Add a sequence number when events cross processes. Avoid storing raw audio or transcript text merely because the timing test can access it. Timing evidence usually needs event metadata and a separately controlled audio fixture, which is easier to retain safely than a production conversation.

Use names that describe facts, not conclusions. caller_speech_end_observed says a detector emitted a boundary. turn_committed says the orchestrator accepted it. tts_first_chunk_received says audio reached a client boundary. speaker_first_sample_scheduled says the output path accepted the first sample. A field called latency_problem skips all of the evidence needed to challenge the label.

The reference speech end belongs in recorded-fixture tests. It should come from a reviewed annotation or a deterministic signal you generated, not from the same detector being evaluated. If both the expected boundary and actual boundary come from one detector event, the endpointing assertion can never expose that detector's mistake. This is the voice equivalent of computing expected and actual values with the same buggy function.

The following TypeScript test uses application-owned event names. Its limits are explicitly an illustrative policy for the fixture, not measurements from a live system. The three assertions can fail independently: slow endpoint commitment, slow end-to-end playout, and assistant speech before the reviewed end of the caller's utterance. Changing the evaluator so it ignores any one of those conditions makes the corresponding test fail.

import assert from "node:assert/strict";
import test from "node:test";

type EventName =
  | "caller_speech_end_observed"
  | "turn_committed"
  | "response_requested"
  | "tts_first_chunk_received"
  | "assistant_playout_started";

type TurnEvent = {
  name: EventName;
  atMs: number;
};

type TurnCase = {
  id: string;
  referenceCallerEndMs: number;
  events: TurnEvent[];
  policy: {
    maxEndpointDelayMs: number;
    maxAudibleGapMs: number;
    maxEarlyOverlapMs: number;
  };
};

type Evaluation = {
  endpointDelayMs: number;
  audibleGapMs: number;
  earlyOverlapMs: number;
  failures: string[];
};

function eventTime(turn: TurnCase, name: EventName): number {
  const matches = turn.events.filter((event) => event.name === name);
  assert.equal(matches.length, 1, `${turn.id}: expected one ${name} event`);
  assert.ok(Number.isFinite(matches[0].atMs), `${turn.id}: ${name} must be finite`);
  return matches[0]!.atMs;
}

function evaluateTurn(turn: TurnCase): Evaluation {
  const observedEnd = eventTime(turn, "caller_speech_end_observed");
  const committed = eventTime(turn, "turn_committed");
  const requested = eventTime(turn, "response_requested");
  const firstChunk = eventTime(turn, "tts_first_chunk_received");
  const firstSample = eventTime(turn, "assistant_playout_started");

  assert.ok(committed >= observedEnd, `${turn.id}: commit precedes observed endpoint`);
  assert.ok(requested >= committed, `${turn.id}: response precedes turn commit`);
  assert.ok(firstChunk >= requested, `${turn.id}: TTS chunk precedes response request`);
  assert.ok(firstSample >= firstChunk, `${turn.id}: playout precedes received audio`);

  const endpointDelayMs = committed - turn.referenceCallerEndMs;
  const audibleGapMs = firstSample - turn.referenceCallerEndMs;
  const earlyOverlapMs = Math.max(0, turn.referenceCallerEndMs - firstSample);
  const failures: string[] = [];

  if (endpointDelayMs > turn.policy.maxEndpointDelayMs) {
    failures.push(`endpointing ${endpointDelayMs}ms > ${turn.policy.maxEndpointDelayMs}ms`);
  }
  if (audibleGapMs > turn.policy.maxAudibleGapMs) {
    failures.push(`audible gap ${audibleGapMs}ms > ${turn.policy.maxAudibleGapMs}ms`);
  }
  if (earlyOverlapMs > turn.policy.maxEarlyOverlapMs) {
    failures.push(`early overlap ${earlyOverlapMs}ms > ${turn.policy.maxEarlyOverlapMs}ms`);
  }

  return { endpointDelayMs, audibleGapMs, earlyOverlapMs, failures };
}

const policy = {
  maxEndpointDelayMs: 350,
  maxAudibleGapMs: 800,
  maxEarlyOverlapMs: 40,
};

test("identifies slow endpoint commitment", () => {
  const result = evaluateTurn({
    id: "slow-endpoint",
    referenceCallerEndMs: 2_000,
    policy,
    events: [
      { name: "caller_speech_end_observed", atMs: 2_000 },
      { name: "turn_committed", atMs: 2_520 },
      { name: "response_requested", atMs: 2_540 },
      { name: "tts_first_chunk_received", atMs: 2_650 },
      { name: "assistant_playout_started", atMs: 2_720 },
    ],
  });
  assert.deepEqual(result.failures, ["endpointing 520ms > 350ms"]);
});

test("identifies a slow first audible response after prompt endpointing", () => {
  const result = evaluateTurn({
    id: "slow-playout",
    referenceCallerEndMs: 2_000,
    policy,
    events: [
      { name: "caller_speech_end_observed", atMs: 2_000 },
      { name: "turn_committed", atMs: 2_180 },
      { name: "response_requested", atMs: 2_200 },
      { name: "tts_first_chunk_received", atMs: 2_700 },
      { name: "assistant_playout_started", atMs: 2_850 },
    ],
  });
  assert.equal(result.endpointDelayMs, 180);
  assert.deepEqual(result.failures, ["audible gap 850ms > 800ms"]);
});

test("identifies assistant speech before the reviewed caller end", () => {
  const result = evaluateTurn({
    id: "early-answer",
    referenceCallerEndMs: 2_300,
    policy,
    events: [
      { name: "caller_speech_end_observed", atMs: 2_050 },
      { name: "turn_committed", atMs: 2_080 },
      { name: "response_requested", atMs: 2_100 },
      { name: "tts_first_chunk_received", atMs: 2_160 },
      { name: "assistant_playout_started", atMs: 2_190 },
    ],
  });
  assert.deepEqual(result.failures, ["early overlap 110ms > 40ms"]);
});

The first case is deliberately subtle. Its total audible gap still fits the illustrative end-to-end limit, but the endpointing slice is too large. That points the investigation toward speech-end detection or turn commit instead of the model. If the test asserted only total gap, a faster synthesis change could hide the endpoint regression and the user would still experience an awkward pause in less favorable turns.

The second case is the mirror image, and it exists because a guard that no fixture exercises is not a guard. Endpointing lands at 180 milliseconds, comfortably inside its own limit, so the detector is not the problem. Synthesis then takes half a second to produce a first chunk and playout starts at 2,850 milliseconds, which is 850 milliseconds of silence measured from the reviewed caller end. Only the end-to-end limit catches that. Without this fixture the audible-gap comparison is code nothing can prove correct: delete the whole branch and the suite stays green, which means the release gate would silently stop noticing dead air. Check that for yourself by removing each of the three comparisons in turn and rerunning. Each deletion should turn exactly one test red. If one of them costs nothing, the limit it enforces is decoration.

The third case preserves two views of caller speech. The running detector reports an end at 2,050 milliseconds, while the reviewed fixture says the caller continues until 2,300 milliseconds. The assistant begins at 2,190 milliseconds. A test based only on the detector's own event would call that a healthy positive gap; the independent annotation exposes 110 milliseconds of premature speech.

Keep raw events with the calculated result. A failure that prints only audibleGapMs=910 forces the investigator to reconstruct the path from logs. A useful failure prints the event pair, turn ID, observer, and local duration that crossed policy. For a distributed call, include trace and media identifiers but do not subtract unrelated host clocks inside the assertion.

Threshold equality is a product choice and deserves a boundary test. If 800 milliseconds is allowed, code > and add a case at exactly 800. If 800 should fail, code >= and name that convention in the policy. Leaving the comparison implicit creates release arguments after the result, when the contract should have settled them before execution.

Reproduce three failures without guessing

The first worked failure is late endpointing. Use a recording with a clean final word followed by controlled silence. Label the final active caller frame and replay the same audio into the endpoint detector. If turn_committed consistently lands late while response processing and synthesis remain within their own budgets, the detector or its surrounding debounce logic owns the gap. A model swap is not a relevant fix.

Look at the waveform before reducing a silence setting. A long tail from room reverberation, keyboard noise, or automatic gain control can keep a detector active. That produces the same late commit as a conservative endpoint policy, but the correction is different. A clean lab fixture that passes while a reverberant fixture fails suggests input classification. Both clean and noisy fixtures failing by the configured hold duration suggests policy or timer behavior.

The trade-off is conversational safety. Shortening an end-of-speech hold reduces silence, but it also makes hesitation, serial numbers, and self-corrections easier to cut off. Test phrases that contain deliberate internal pauses: "the code is four seven... no, four nine two" and multi-part dates are more useful than a steady sentence. The negative fixture should prove that the agent does not answer after the pause when the caller resumes within the accepted continuation window.

The second failure is an early answer caused by a false endpoint. Transcript review often misses it because a recognizer may later revise the text into a complete sentence. The final transcript looks coherent even though the assistant began over the last clause. Evidence lives in the audio boundary and event chronology: the detector emits end, turn commit follows, the caller waveform continues, and assistant playout starts before the reviewed caller end.

Fixing this with a blanket longer hold can make every turn feel slow. A better rollout starts by classifying the shapes that caused false endpoints. Numbers, addresses, hesitant speech, second-language speech, and noisy mobile input may need different coverage even if the product uses one detector configuration. Add fixtures before adjusting policy, then compare both false-end and late-end rates. An endpoint change that fixes interruption while doubling dead air has exchanged one defect for another.

The third failure is broken barge-in. Begin with assistant audio that is long enough to interrupt, then inject caller speech at a known point. Mark four events: caller speech begins, the client sends the interrupt signal, the response producer stops adding audio, and local playout becomes silent. The user-facing stop delay ends at local silence, not at the server's cancellation log.

A common near-miss is echo. The test microphone captures the assistant speaker, the speech detector treats that echo as caller speech, and the agent cancels itself. From an event-only trace it looks like successful, implausibly fast barge-in. A duplex capture tells a different story: there is energy on the microphone channel, but no independent caller source was injected. Run a control with assistant playback and no caller audio. If that control emits caller-speech events, fix echo handling or the test rig before evaluating interruption latency.

Another near-miss is queued playback. The server may stop generation promptly while the browser already holds audio frames. The cancellation span is green, but recorded output continues. The correct client behavior depends on the product: it may clear queued audio immediately, fade it to avoid a click, or finish a short phoneme. Whatever the choice, assert the output consequence. Do not award a pass because an upstream component accepted a command.

Waveform evidence makes these cases reviewable without trusting a service label. The next Python program analyzes a stereo PCM WAV capture with caller audio on the left and assistant output on the right. It reports the last active caller window, first active assistant window, and the resulting gap or overlap. The amplitude threshold is a fixture parameter, not a universal speech detector. Calibrate it against silence and known speech in your capture rig, and keep that calibration with the fixture.

#!/usr/bin/env python3
import argparse
import json
import math
import struct
import sys
import wave
from pathlib import Path


def rms(samples: tuple[int, ...]) -> float:
    if not samples:
        return 0.0
    return math.sqrt(sum(sample * sample for sample in samples) / len(samples))


def active_windows(path: Path, window_ms: int, threshold: float):
    with wave.open(str(path), "rb") as source:
        if source.getnchannels() != 2:
            raise ValueError("expected stereo WAV: caller left, assistant right")
        if source.getsampwidth() != 2 or source.getcomptype() != "NONE":
            raise ValueError("expected uncompressed 16-bit PCM WAV")

        rate = source.getframerate()
        frames_per_window = max(1, round(rate * window_ms / 1000))
        raw = source.readframes(source.getnframes())
        values = struct.unpack(f"<{len(raw) // 2}h", raw)

    caller_active: list[int] = []
    assistant_active: list[int] = []
    sample_stride = frames_per_window * 2

    for window_index, offset in enumerate(range(0, len(values), sample_stride)):
        block = values[offset : offset + sample_stride]
        caller = block[0::2]
        assistant = block[1::2]
        if rms(caller) >= threshold:
            caller_active.append(window_index)
        if rms(assistant) >= threshold:
            assistant_active.append(window_index)

    if not caller_active:
        raise ValueError("no caller activity crossed the configured threshold")
    if not assistant_active:
        raise ValueError("no assistant activity crossed the configured threshold")

    caller_end_ms = (caller_active[-1] + 1) * window_ms
    assistant_start_ms = assistant_active[0] * window_ms
    signed_gap_ms = assistant_start_ms - caller_end_ms
    return {
        "callerEndMs": caller_end_ms,
        "assistantStartMs": assistant_start_ms,
        "gapMs": max(0, signed_gap_ms),
        "overlapMs": max(0, -signed_gap_ms),
        "windowMs": window_ms,
        "rmsThreshold": threshold,
    }


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("capture", type=Path)
    parser.add_argument("--window-ms", type=int, default=20)
    parser.add_argument("--rms-threshold", type=float, required=True)
    parser.add_argument("--max-gap-ms", type=int, required=True)
    parser.add_argument("--max-overlap-ms", type=int, required=True)
    args = parser.parse_args()

    result = active_windows(args.capture, args.window_ms, args.rms_threshold)
    print(json.dumps(result, indent=2))

    violations = []
    if result["gapMs"] > args.max_gap_ms:
        violations.append(f'gap {result["gapMs"]}ms > {args.max_gap_ms}ms')
    if result["overlapMs"] > args.max_overlap_ms:
        violations.append(
            f'overlap {result["overlapMs"]}ms > {args.max_overlap_ms}ms'
        )
    if violations:
        print("; ".join(violations), file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

This analyzer is intentionally narrower than voice activity detection in production. It works well with controlled, separated channels and generated or reviewed fixtures. It is not a safe way to infer who spoke in a mixed production recording. Compression noise, crosstalk, music, and gain changes can cross a simple RMS threshold. When those conditions are part of the requirement, use labeled channels or a validated speech detector and retain its confidence separately.

Run the analyzer on at least three audio shapes. A clean handoff checks ordinary response gap. A false-end fixture puts a pause inside the caller's sentence and checks for early overlap. A barge-in fixture starts caller audio during a long assistant response and needs a different assertion that measures when assistant energy ends after caller onset. Do not force the first-assistant-start script to answer the barge-in question; add an end-of-assistant calculation because the event under test is different.

Tell endpointing delay from transport and playback delay

Two calls can produce the same silent gap for unrelated reasons. In the first, the detector waits too long before committing the turn. In the second, the turn commits promptly but returning audio sits in a network or client buffer. The audio alone establishes impact. The component timeline establishes ownership.

Investigate from the caller boundary forward. Compare the reviewed caller end with the detector end. Then compare detector end with turn commit, turn commit with response request, request with the first synthesis audio received, and received audio with local playout. Stop at the first interval that changed against a matched passing run. Later intervals can move as a consequence, so blaming the largest final timestamp often sends the defect to the wrong team.

Use a matched run rather than a global average. Hold the audio fixture, response content, browser family, media path, and network profile constant. If the endpointing slice changes while WebRTC counters remain similar, inspect speech processing. If the application and synthesis slices remain similar while the receive-to-playout slice grows, inspect transport and buffering. If every slice is stable but the waveform shows a later first sample, inspect capture and output instrumentation.

For browser calls carried through RTCPeerConnection, getStats() returns an RTCStatsReport. The WebRTC statistics specification defines inbound RTP counters including jitterBufferDelay and jitterBufferEmittedCount. Both are cumulative. The specification defines average jitter-buffer delay as their ratio, which means a test comparing a particular interval should subtract two snapshots before dividing. Reading the lifetime ratio after one bad turn can dilute the event with the rest of the call.

This browser-side TypeScript helper records deltas for every inbound audio report instead of assuming the call contains only one. It also keeps the stats object's identifier so an investigator can match the same monitored stream across snapshots. Missing optional counters stay missing; the code does not silently replace absent evidence with zero.

type AudioSnapshot = {
  id: string;
  sampledAt: number;
  jitterBufferDelay?: number;
  jitterBufferEmittedCount?: number;
  concealedSamples?: number;
  packetsLost?: number;
};

type AudioWindow = {
  id: string;
  elapsedMs: number;
  averageJitterBufferDelayMs?: number;
  concealedSamplesDelta?: number;
  packetsLostDelta?: number;
};

export async function snapshotInboundAudio(
  connection: RTCPeerConnection,
): Promise<AudioSnapshot[]> {
  const report = await connection.getStats();
  const snapshots: AudioSnapshot[] = [];

  report.forEach((raw) => {
    if (raw.type !== "inbound-rtp" || raw.kind !== "audio") return;
    snapshots.push({
      id: raw.id,
      sampledAt: raw.timestamp,
      jitterBufferDelay:
        typeof raw.jitterBufferDelay === "number" ? raw.jitterBufferDelay : undefined,
      jitterBufferEmittedCount:
        typeof raw.jitterBufferEmittedCount === "number"
          ? raw.jitterBufferEmittedCount
          : undefined,
      concealedSamples:
        typeof raw.concealedSamples === "number" ? raw.concealedSamples : undefined,
      packetsLost: typeof raw.packetsLost === "number" ? raw.packetsLost : undefined,
    });
  });

  return snapshots;
}

export function compareAudioSnapshots(
  before: AudioSnapshot,
  after: AudioSnapshot,
): AudioWindow {
  if (before.id !== after.id) throw new Error("cannot compare different RTP streams");

  const delayDelta =
    before.jitterBufferDelay === undefined || after.jitterBufferDelay === undefined
      ? undefined
      : after.jitterBufferDelay - before.jitterBufferDelay;
  const emittedDelta =
    before.jitterBufferEmittedCount === undefined ||
    after.jitterBufferEmittedCount === undefined
      ? undefined
      : after.jitterBufferEmittedCount - before.jitterBufferEmittedCount;

  return {
    id: before.id,
    elapsedMs: after.sampledAt - before.sampledAt,
    averageJitterBufferDelayMs:
      delayDelta === undefined || emittedDelta === undefined || emittedDelta <= 0
        ? undefined
        : (delayDelta / emittedDelta) * 1000,
    concealedSamplesDelta:
      before.concealedSamples === undefined || after.concealedSamples === undefined
        ? undefined
        : after.concealedSamples - before.concealedSamples,
    packetsLostDelta:
      before.packetsLost === undefined || after.packetsLost === undefined
        ? undefined
        : after.packetsLost - before.packetsLost,
  };
}

Do not turn these counters into a universal diagnosis. A larger interval average for jitter-buffer delay supports a buffering investigation; it does not prove why the network varied or that buffering alone caused the audible gap. Concealed samples support an audio-quality investigation, not a direct latency calculation. Packet loss can matter without being the only cause. Keep the stats snapshot beside, not in place of, the event and audio timelines.

The exact diagnostic output should name missing evidence. If jitterBufferDelay is absent, report it as unavailable for that stream and browser. Zero would mean the implementation observed the counter and its value was zero, which is a different fact. Likewise, a snapshot with no increase in jitterBufferEmittedCount cannot produce an interval average. Returning undefined prevents a division by zero from becoming Infinity or a misleading pass.

Client scheduling deserves its own check. A page can receive audio promptly and schedule it late because the main thread is busy, the audio context is not running, or the application serializes chunks incorrectly. Compare the time of first chunk receipt with the nearest playout event your client can observe. Then confirm impact with an output capture. A network fix cannot repair a queue that grows after packets arrive.

Cold-start behavior is another look-alike. The first synthesis request after deployment or idle time may take a different path from warmed requests. Do not hide it by retrying the same test until it passes. Label cold and warm scenarios, control their setup, and report them separately. If production traffic routinely encounters idle instances, the cold path belongs in release evidence. If the test environment alone tears down resources between cases, the extra delay is infrastructure evidence and should not be filed as a product regression without reproduction.

Tool-backed turns also resemble latency regressions. A balance lookup is expected to perform work that a greeting does not. Compare like with like: same intent class, same tool response fixture, same audio, and same response-length policy. The correct observation may be a brief acknowledgement followed by the result, not an unrealistically short silence while the tool runs. Tests should enforce the product's conversational design, not reward the system for speaking before it has an answer.

Roll the checks into CI without hiding variation

Begin the rollout with event-schema tests. They are fast and can prove required events are present once, ordered according to your contract, and linked to the correct turn. They also catch accidental clock mixing when metadata identifies the observer and time origin. These tests do not need live speech services, so they belong in every change set that touches orchestration, synthesis plumbing, or the client player.

Add recorded-audio replay next. Keep the first set small and intentional: clean handoff, internal pause, trailing low-energy word, background noise, and caller barge-in. A fixture earns its place by representing a distinct decision boundary. Fifty near-identical sentences make failures expensive to review without expanding the behavioral surface.

Pin the fixture file, annotation, expected floor owner, and threshold policy together. If an annotator moves the caller end, that is a test-data change and should be reviewed as such. If a product owner changes the maximum response gap, that is a policy change. If the event timeline shifts with neither changing, that is a candidate implementation regression. Versioning those inputs stops a policy edit from appearing as a sudden performance improvement.

Use distributions for repeated environments, but retain individual failures. A median can stay flat while a tail of calls becomes painful. A high percentile can reveal the tail, but it still cannot tell you whether the worst turns are all one intent or one network profile. Report scenario, environment, and component slice alongside the percentile. Never average negative early-overlap values with positive silence values because they can cancel into a reassuring number that represents neither experience.

Choose the release gate from data you actually collected. The values below are illustrative wiring and must be replaced with limits approved for the product and capture rig. The workflow runs deterministic unit tests, analyzes one controlled stereo capture, preserves the output even on failure, and relies only on a standard Python interpreter for the analyzer. It does not call a live vendor service, so a third-party outage cannot turn a code review into a random red build.

name: voice-turn-contract

on:
  pull_request:
    paths:
      - "src/voice/**"
      - "tests/voice/**"
      - "scripts/analyze_stereo_turn.py"

jobs:
  recorded-turns:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
      - name: Run turn evaluator tests
        run: node --test "tests/voice/**/*.test.ts"
      - name: Run event and fixture tests
        run: python -m unittest discover -s tests/voice -p "test_*.py"
      - name: Check first audible response
        shell: bash
        run: |
          set -o pipefail
          mkdir -p artifacts
          python scripts/analyze_stereo_turn.py \
            tests/voice/fixtures/clean-handoff.wav \
            --rms-threshold 700 \
            --max-gap-ms 800 \
            --max-overlap-ms 40 \
            | tee artifacts/clean-handoff.json
      - name: Preserve timing evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: voice-turn-evidence
          path: artifacts/

Both test steps are present because the suite spans two runtimes, and a gate that runs only one of them is a gate with a hole in it. The turn evaluator is TypeScript on the node:test runner, while the waveform fixtures are Python. A workflow carrying only the unittest step would never execute the evaluator at all, and the failure would not even be loud: on Python 3.12 an empty discovery exits with status 5, so a job that also lost its Python tests would at least go red, but a job that quietly skips the TypeScript half looks perfectly healthy. Node 22 runs the .ts files directly through its own type stripping, so no build step is required. Confirm the step by breaking one assertion on purpose and watching the workflow fail before you trust it.

The workflow's file filter is deliberately narrow, but teams should broaden it to any code that can influence the path. Prompt changes can alter how soon synthesizeable text becomes available. Codec, transport, and player changes can alter delivery. Speech-detector configuration can alter endpointing. A filter that ignores those files makes the suite look stable by not running when risk changes.

Keep live end-to-end probes outside the mandatory pull-request gate until their dependencies are controlled. They are useful for scheduled checks against a staging deployment, especially when the real media path differs from replay. They are also exposed to shared-network noise, service quotas, region routing, and transient provider failures. Record those conditions and route inconclusive infrastructure results to review instead of automatically blaming the change under test.

Retries need an explicit purpose. Repeating a noisy probe can estimate variability when every attempt is retained. Retrying only to turn red into green erases the tail behavior voice users notice. Report attempt count, all component timings, and whether a release rule evaluates the first attempt, worst attempt, or a declared distribution statistic. The choice should match the user risk, not the desire for a green dashboard.

Roll out new limits in shadow mode. Capture results without blocking, inspect failures, and verify the test rig against recordings. Then enable blocking for stable, deterministic fixtures. Keep noisy networks and live services informational until the team has enough evidence to set an environment-specific policy. Shadowing costs calendar time, but it is cheaper than training developers to ignore a flaky gate.

When a limit changes, replay the frozen corpus under both policies. That shows exactly which turns move from pass to fail. Review those audio files rather than accepting a percentage alone. A threshold that rejects a reasonable reflective pause or accepts clipped final digits needs more work even if its aggregate pass rate looks attractive.

Know when not to tighten the threshold

Do not reduce endpoint delay merely because a competitor demo sounds faster. A demo question is often short, clean, and rehearsed. Real callers pause to recall names, spell addresses, breathe between number groups, or correct themselves. If the product cannot preserve those turns, a lower latency figure is an expensive way to make the agent less usable.

Avoid a strict first-audio gate when the scenario requires a completed side effect before speaking. A refund, transfer, or identity check may need confirmed state. The conversational design can still reduce uncertainty with an acknowledgement, but that acknowledgement becomes its own content and timing contract. Forcing the final answer to start early risks confidently reporting work that has not completed.

Do not use transcript timing as a substitute for acoustic timing when overlap is the concern. Text can arrive in chunks, be revised, and be timestamped at recognition boundaries. None of that establishes when a speaker emitted the final phoneme or when the other side heard audio. Transcript evidence is excellent for whether the agent understood the last clause. It cannot, by itself, prove that the agent waited for it.

Skip waveform thresholding on a mixed production recording when you cannot separate caller, assistant, and echo. A single energy trace cannot reliably attribute simultaneous sound. Use separated media tracks, controlled injection, or human review. Presenting a precise overlap duration from an unattributed mix creates confidence the evidence does not deserve.

Do not apply browser WebRTC counters to a media path that does not use RTCPeerConnection. The names may be familiar, but evidence from a different transport model will not appear. Instrument the actual client boundary instead. Even on WebRTC, treat optional stats as optional and verify availability in the browsers that matter before making a gate depend on them.

Avoid comparing absolute timestamps from separate hosts unless the system has a documented synchronization guarantee and an uncertainty small enough for the limit. Wall-clock subtraction can make a healthy component look negative or slow. Local durations plus trace ordering are less glamorous, but they are defensible. If precise cross-host timing is essential, test the clock pipeline as part of the measurement system.

Do not optimize an average while leaving scenario coverage unchanged. Greetings dominate many datasets and are cheap to answer. Tool calls, noisy speech, hesitant speech, and barge-in can regress while the overall mean improves. Segment first, then decide where speed is required and where conversational correctness deserves more time.

A fixed limit is also the wrong tool for exploratory perceptual work. Two responses with the same silence can feel different because one gives an immediate acknowledgement, one begins with a filler, and one starts a complete answer. Human listening sessions can help define the product contract, but they should not overwrite deterministic timing records. Use them to decide what to measure and which scenarios deserve separate policies.

Every latency fix charges something. A shorter endpoint hold increases false ends. More lookahead protects the caller but adds silence. Smaller playback buffers can reduce delay but tolerate less network variation. Streaming smaller synthesis chunks can start audio sooner but increase coordination and may expose awkward prosody boundaries. Aggressive barge-in can feel responsive but truncate important disclosures. Name the cost in the change review and add the fixture that would catch it.

The safest release decision preserves both sides of the conversation. Require a response gap appropriate to the scenario, a limit on uninvited overlap, and a barge-in stop contract where interruption is supported. When one fails, route the defect to the earliest changed boundary with audio and event evidence attached. That gives the engineer something to reproduce instead of one red number labeled "slow."