The transcript was close, but the action was wrong

A caller says, “Cancel tomorrow's delivery,” but the transcript reads, “Cancel today's delivery.” The intent classifier correctly returns cancel_delivery for the words it received, and a coarse intent dashboard stays green. The customer still loses the wrong package.

A useful evaluation follows what reached the recognizer, which words it produced, how those words became an intent and entities, and which action the agent finally took.

Follow the signal through every boundary

A voice request crosses more systems than a typed message. The microphone and browser or telephony client capture a waveform. Encoding, transport, jitter handling, resampling, echo cancellation, noise suppression, and voice activity detection may alter what reaches automatic speech recognition (ASR). The recognizer emits text, sometimes with timing or confidence metadata. Normalization changes that text. An intent model or rules engine chooses a label and extracts entities. An orchestrator turns the result into a tool call or spoken response.

Put an observation point at each boundary:

1. Capture the audio that the client says it sent.
2. Capture the audio bytes received by the ASR service or adapter.
3. Save the recognizer's raw transcript before cleanup.
4. Save the normalized text actually passed to intent classification.
5. Record intent, entities, conversation state, and taxonomy version.
6. Record the action request and the tool result.

Those artifacts create ownership. If the client recording contains “tomorrow” but the ASR input contains a dropout over that syllable, the recognizer is not the first broken component. If both audio files are clear and the transcript says “today,” the recognition path deserves attention. If the transcript is correct and the entity is wrong, audio tuning will not fix it.

Browser capture settings matter when the product uses web audio. The W3C Media Capture and Streams specification defines constrainable audio properties including sample rate, channel count, echo cancellation, automatic gain control, noise suppression, and latency. The actual settings and support depend on the user agent and device. Record what was requested and what was actually applied rather than assuming a constraint took effect.

For WebRTC calls, network evidence can help separate transport damage from language errors. The W3C statistics specification defines inbound RTP fields such as packets received, packets lost, jitter, packets discarded, and concealed audio samples. Those are cumulative transport or media-pipeline observations, not a direct measure of transcription quality. Correlate deltas over the tested utterance with the captured audio. A lifetime counter from the whole call can mislead.

Offline replay and live-call evaluation answer different questions. Offline replay sends a stable audio file to the recognition pipeline and is good for ASR, normalization, and intent regression. A live call exercises microphone permissions, devices, codecs, packet timing, echo, turn taking, and capture boundaries. A suite needs both, but a failure must say which path ran.

Audio format is part of the case contract. Store container, codec, sample rate, sample width when relevant, channel count, duration, and digest. A .wav extension does not prove PCM content, and PCM WAV is not representative of every production codec. The MDN audio-codec guide documents that codecs and containers are separate concerns, which is exactly why “we tested a WAV” is not enough context for a telephony incident.

Measure words without losing business meaning

Word error rate (WER) is the edit distance between a reference word sequence and a hypothesis, divided by the number of reference words. Substitutions, deletions, and insertions contribute to the distance. It is useful because it shows recognition movement over a corpus and supports analysis by acoustic slice.

WER is not intent accuracy. “Do cancel my booking” and “don't cancel my booking” differ by one short token and demand opposite behavior. “Transfer fifteen dollars” and “transfer fifty dollars” can have a low WER while changing money movement. Report critical-token and entity correctness next to the aggregate.

Normalization policy changes the score. Case folding may be safe for English intent classification. Removing apostrophes can make we'll and well collide. Converting spoken number words to digits can help entity comparison, but only if the same deterministic rule is applied to reference and hypothesis. Punctuation may be absent from streaming transcripts. Document the transformation and keep the raw strings.

This implementation computes token-level edit distance after a deliberately small normalization step. It uses only the Python standard library and makes the empty-reference behavior explicit:

from dataclasses import dataclass
import re
import unicodedata


@dataclass(frozen=True)
class WerResult:
    reference_words: int
    edits: int
    rate: float


def words(text: str) -> list[str]:
    normalized = unicodedata.normalize("NFKC", text).casefold()
    normalized = re.sub(r"[^\w']+", " ", normalized, flags=re.UNICODE)
    return normalized.split()


def word_error_rate(reference: str, hypothesis: str) -> WerResult:
    expected = words(reference)
    actual = words(hypothesis)
    previous = list(range(len(actual) + 1))

    for row, expected_word in enumerate(expected, start=1):
        current = [row]
        for column, actual_word in enumerate(actual, start=1):
            substitution = previous[column - 1] + (
                expected_word != actual_word
            )
            deletion = previous[column] + 1
            insertion = current[column - 1] + 1
            current.append(min(substitution, deletion, insertion))
        previous = current

    edits = previous[-1]
    if not expected:
        rate = 0.0 if not actual else float("inf")
    else:
        rate = edits / len(expected)
    return WerResult(len(expected), edits, rate)


if __name__ == "__main__":
    result = word_error_rate(
        "Cancel tomorrow's delivery",
        "Cancel today's delivery",
    )
    print(result)

That code reports a rate for one utterance, but release reporting should also sum edits and reference words across a defined corpus. An unweighted average of utterance-level WER gives a two-word clip the same influence as a twenty-word request. Keep both per-case results and a documented corpus aggregation.

Reference transcription needs a style guide. Decide how to label filled pauses, contractions, partial words, background speakers, proper names, and code-switching. If one annotator writes “twenty one” and another writes “21,” the evaluation adds noise unrelated to ASR behavior. Preserve verbatim speech for audit, then derive a normalized comparison field.

Intent needs its own contract. Exact label accuracy works when the taxonomy is stable and mutually understood. Hierarchical systems may need both a broad family such as delivery_change and a leaf such as cancel_delivery. Multi-intent turns require a set or sequence, not one label. Entity checks should include value, type, and source span when available.

Confusion matrices are more actionable than a single accuracy figure. A confusion between track_delivery and delivery_status may be tolerable if both route to the same read-only flow. Confusing cancel_delivery with reschedule_delivery is not. Give high-risk pairs explicit assertions even if they are rare in the corpus.

Latency also belongs beside correctness. Measure timestamps at boundaries you control, such as end of captured speech, final ASR result received, intent result produced, and first response audio sent. Name whether the measurement is client-side or server-side. Do not compare timings from clocks that have not been correlated, and do not invent a latency target because another product published one.

Use failures that force the right diagnosis

The delivery example is a critical-entity substitution. The broad intent is correct, so intent accuracy alone passes. WER may look modest because only one word changed. The oracle must assert date=tomorrow or an equivalent resolved date derived under a pinned clock and timezone. It should also confirm that the action payload uses that value.

The first fix is not automatically “improve ASR.” Replay the exact recognizer input. If the word is clear but consistently misrecognized, add representative cases and evaluate recognizer or vocabulary changes. If the audio clips the first syllable, inspect voice activity detection and turn-boundary handling. If the transcript says “tomorrow” but the resolved date is today, fix entity resolution. Each layer can create the same customer symptom.

A second case involves negation during barge-in. The agent asks, “Should I cancel reservation Q7?” The caller interrupts with, “No, don't cancel it.” The microphone also captures the agent's own word “cancel.” A transcript may contain “cancel it” after echo suppression, overlap handling, or segmentation loses “No, don't.”

The evidence must include separate timing for agent playback and caller audio, the mixed or post-processed audio sent to ASR, partial transcripts, the final transcript, and the turn selected for classification. A clean offline recording of the caller will pass because it does not reproduce the duplex interaction. This is a live media and turn-taking case.

Possible fixes include tuning capture constraints where supported, stopping playback sooner on barge-in, improving echo handling, changing endpointing, or requiring confirmation for destructive intents. Each choice has a cost. Aggressive endpointing can cut off quiet speech. Waiting longer adds response latency. Confirmation adds a turn and may frustrate users, but it is often the right safety boundary for cancellation or payment.

A third case uses an alphanumeric identifier: “Open claim B7K9.” The transcript reads “B seven K nine,” which is linguistically faithful but does not exactly match the stored claim ID. WER can penalize the formatting even while entity normalization succeeds. The opposite can also happen: the transcript looks plausible, but normalization converts “B7K9” to “B7K8” and opens the wrong record.

Test the raw transcript and the canonical entity separately. Build a deterministic normalizer for the identifier grammar, then use exact matching against known fixtures. Do not apply fuzzy matching to account, claim, booking, or one-time-code identifiers without a product-approved confirmation rule. A “close” identifier is usually a different customer's data.

A fourth case is a taxonomy migration. The transcript remains identical, but the intent service changes change_address to update_delivery_address. Old fixtures fail every row after deployment. That resembles a model collapse in the dashboard, yet the recognizer and classifier may both be behaving as designed under different schemas.

Version the taxonomy in expected and observed records. During migration, map old and new labels only where product owners confirm equivalence. Keep the raw label as evidence. A permanent compatibility map can conceal a router that still expects the retired name, so test the downstream action as well.

Synthetic voices help cover phrasing, rates, and systematic audio transformations, but they rarely reproduce the full distribution of production speech. Include consented recordings across microphones, distance, room acoustics, accents, speaking styles, ages where appropriate and lawful, interruptions, and code-switching relevant to the product. Report slices separately. A broad average can improve while one user group gets worse.

Collect evidence from audio to action

Start triage by proving that the audio artifact is the one evaluated. Store a digest at the ASR boundary. Check duration, channel count, sample rate, and sample width for formats where those concepts apply. Listen through an approved secure workflow when policy permits. A transcript cannot reveal a dropout, clipped opening, or leaked playback.

For uncompressed PCM WAV fixtures, Python's wave module can expose the header and frames. This diagnostic checks a narrow 16-bit PCM contract and calculates peak and root-mean-square amplitude. Those amplitude values help find silence or clipping, but they do not measure intelligibility:

from array import array
from dataclasses import dataclass
from pathlib import Path
import math
import sys
import wave


@dataclass(frozen=True)
class PcmFacts:
    sample_rate: int
    channels: int
    duration_seconds: float
    peak: int
    rms: float


def inspect_pcm16_wav(path: Path) -> PcmFacts:
    with wave.open(str(path), "rb") as stream:
        if stream.getcomptype() != "NONE":
            raise ValueError("fixture must contain uncompressed PCM")
        if stream.getsampwidth() != 2:
            raise ValueError("fixture must use 16-bit samples")

        channels = stream.getnchannels()
        sample_rate = stream.getframerate()
        frame_count = stream.getnframes()
        payload = stream.readframes(frame_count)

    samples = array("h")
    samples.frombytes(payload)
    if sys.byteorder != "little":
        samples.byteswap()
    if not samples:
        raise ValueError("fixture contains no audio samples")

    peak = max(abs(value) for value in samples)
    rms = math.sqrt(sum(value * value for value in samples) / len(samples))
    return PcmFacts(
        sample_rate=sample_rate,
        channels=channels,
        duration_seconds=frame_count / sample_rate,
        peak=peak,
        rms=rms,
    )


if __name__ == "__main__":
    facts = inspect_pcm16_wav(Path("fixtures/audio/cancel-tomorrow.wav"))
    assert facts.sample_rate == 16000
    assert facts.channels == 1
    print(facts)

Do not run that function on Opus, MP3, or arbitrary files and interpret the exception as bad speech. Container and codec support belong in fixture metadata and decoding setup. Transcoding everything to PCM can make offline comparison convenient, but retain the original encoded artifact because the decoder or resampler may be the bug.

Next, compare the client-side and server-side recordings. If their digests differ, that can be expected when encoding occurs, so decode them to a documented common representation before waveform analysis. Do not expect byte equality across codecs. Instead, check duration, gaps, channel routing, and listenable alignment with tools your team has validated.

Then inspect the trace in order:

• Did speech start before capture was ready?
• Did the recorded utterance include the final word?
• Were partial transcripts replaced by a final transcript, or accidentally concatenated?
• Did normalization remove negation, punctuation, or identifier boundaries?
• Which conversation turns were passed to the intent classifier?
• Did the action layer receive the same intent and entities shown in the evaluator?

A common instrumentation bug scores a partial transcript. Streaming ASR may emit evolving hypotheses, and an adapter can accidentally send an early string to intent classification while the report displays the final string. Record event identifiers and monotonic sequence numbers supplied by your own adapter. The visible final transcript is not proof that the classifier saw it.

This illustrative failure output shows the evidence a test should expose:

$ python -m pytest -q tests/voice/test_voice_contract.py -k cancel_tomorrow
F                                                                        [100%]
=================================== FAILURES ===================================
________________________ test_voice_case[cancel_tomorrow] ________________
audio: fixtures/audio/cancel-tomorrow.wav
reference transcript: "cancel tomorrow's delivery"
observed transcript:  "cancel today's delivery"
expected entities: {"delivery_day": "tomorrow"}
observed entities: {"delivery_day": "today"}
expected action: cancel_delivery(delivery_day="tomorrow")
observed action: cancel_delivery(delivery_day="today")
1 failed in 0.11s

The numbers and output above are illustrative. The important feature is the causal trail. It tells an engineer not to celebrate the matching intent label and not to start with the action executor, because the wrong day is already present in the transcript.

When a live failure cannot be reproduced offline, inspect media statistics over the failed interval. Packet loss, discarded packets, jitter-buffer behavior, or concealed samples can support a transport hypothesis in WebRTC. Absence of those signals does not prove the microphone input was clean. Device processing, permissions, acoustic echo, and clipping can occur before RTP statistics begin.

Separate a silent caller from a discarded caller channel

An empty transcript and a fallback intent can arise from two failures that look identical in the final test log. The captured utterance may truly contain no caller speech because permission, device selection, or capture startup failed. A stereo or multi-channel recording may instead contain clear caller speech on one channel while an adapter sends only another channel to ASR. The recognizer receives silence in both situations. Prompting, vocabulary, and intent tuning cannot distinguish or repair them.

Inspect audio by boundary and by channel. Begin with the client capture, then the encoded upload, decoded server representation, and the exact audio passed to the recognizer. For each artifact, report its digest, format, duration, channel count, and channel-specific signal summary produced by a validated audio tool. Pair those facts with speech intervals or voice-activity decisions when the pipeline records them. If every client channel lacks the utterance, investigate capture readiness and device input. If one client or server channel contains the words but the ASR-bound representation does not, find the conversion or channel-selection step where that channel disappears. If the recognizer input contains clear speech and its completed result is empty, move the handoff to the ASR integration or provider investigation.

A healthy diagnostic row shows the same case identifier and media lineage at every boundary, a duration that covers the utterance, caller energy on the channel actually submitted, a completed recognition result, and a transcript sequence that feeds the recorded intent event. A broken channel-routing row shows speech on a retained artifact but near-silence on the recognizer input. A broken capture row is already silent at the first trustworthy client artifact. The final empty transcript is therefore the symptom, not the separating field.

Aggregate amplitude is a misleading value for this case. Loud agent playback on the left channel can produce a normal overall root-mean-square value while quiet caller speech on the right channel is dropped. File size is also misleading because encoded silence occupies bytes, and a plausible duration says nothing about which channel carried the caller. Even listening to a player-generated downmix can conceal the defect if the player combines channels differently from the production adapter. Listen to or visualize each channel under the approved secure workflow, and retain the actual recognizer input.

The same ownership test applies when voice activity detection labels the caller segment as silence. Preserve the audio before and after that decision and the interval selected for recognition. If the pre-decision artifact contains the utterance and the selected interval excludes it, the recognizer never had a chance to succeed. If the interval is correct but the completed transcript remains empty, the evidence crosses into a different team's boundary.

Roll out a gate that stays honest

Build cases from incidents and product risks, then balance them across intent, critical entities, acoustic condition, device path, language, and channel. Do not create fifty paraphrases of “check my balance” and call the suite representative. Include refusals, ambiguity, silence, background speakers, overlapping speech, corrections, and confirmation turns.

The following pytest example keeps transcript, intent, entity, and action checks visible. Its limits and results are illustrative case data, not claimed production measurements:

import pytest

from wer import word_error_rate


CASES = [
    {
        "id": "cancel_tomorrow",
        "reference": "cancel tomorrow's delivery",
        "transcript": "cancel tomorrow's delivery",
        "maximum_wer": 0.0,
        "expected_intent": "cancel_delivery",
        "observed_intent": "cancel_delivery",
        "expected_entities": {"delivery_day": "tomorrow"},
        "observed_entities": {"delivery_day": "tomorrow"},
        "expected_action": "cancel_delivery",
        "observed_action": "cancel_delivery",
    },
    {
        "id": "claim_identifier_spoken",
        "reference": "open claim B seven K nine",
        "transcript": "open claim B seven K nine",
        "maximum_wer": 0.0,
        "expected_intent": "open_claim",
        "observed_intent": "open_claim",
        "expected_entities": {"claim_id": "B7K9"},
        "observed_entities": {"claim_id": "B7K9"},
        "expected_action": "get_claim",
        "observed_action": "get_claim",
    },
]


@pytest.mark.parametrize("case", CASES, ids=lambda case: case["id"])
def test_voice_case(case: dict[str, object]) -> None:
    wer = word_error_rate(
        str(case["reference"]),
        str(case["transcript"]),
    )
    assert wer.rate <= float(case["maximum_wer"])
    assert case["observed_intent"] == case["expected_intent"]
    assert case["observed_entities"] == case["expected_entities"]
    assert case["observed_action"] == case["expected_action"]

Production code should load records from a versioned manifest and write actual pipeline output to a separate immutable result file. Keeping expected and observed fields separate prevents a fixture generator from copying the answer into the result. Validate that every selected case produced a result; an empty parameter list must not become a green job.

Start CI with a deterministic offline smoke set. Pin the audio, ASR configuration, normalizer, intent model, taxonomy, and orchestration code. Fail immediately on missing artifacts, critical intent reversal, wrong protected entity, or unsafe action. Publish WER and confusion counts without gating until baseline variance and labels have been reviewed.

Add a scheduled live-media job for browser or telephony paths. Live infrastructure can fail for reasons unrelated to a code change, so preserve media evidence and separate product regressions from environment outages. Repeatedly retrying until green erases the very variability the job exists to find. If a retry is allowed, report the first result and reason for retry.

This CI wiring runs only the offline contract on pull requests and uploads its evidence:

name: voice-agent-evaluation

on:
  pull_request:
    paths:
      - "voice_agent/**"
      - "evals/voice/**"
      - "requirements.lock"

jobs:
  offline-voice-contract:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: requirements.lock
      - run: python -m pip install -r requirements.lock
      - run: mkdir -p artifacts
      - run: python -m pytest -q tests/voice --junitxml=artifacts/voice.xml
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: voice-evaluation-evidence
          path: artifacts/

Give the pip cache an explicit dependency path. It searches **/requirements.txt and **/pyproject.toml by default, and a repository pinned through requirements.lock matches neither, which fails the setup step long before any audio is transcribed.

Voice artifacts require stricter handling than ordinary screenshots. They can contain biometric characteristics, names, account details, health information, and background conversations. Use consented data, least-privilege access, encryption and retention controls appropriate to the product, and redacted derived reports. Do not upload sensitive audio to a generally accessible CI artifact store just because the YAML makes it convenient.

An existing transcript-only suite needs a compatibility path before it can become an audio-to-action gate. First add versioned case manifests, stable case identifiers, reference-transcription rules, and explicit states for missing, pending, and failed observations. Keep publishing the current transcript metric while new readers learn those states. Historical rows without source audio should remain useful for normalizer or intent tests, but they cannot be promoted into acoustic evidence by attaching guessed metadata.

Next, run the legacy assertion and new trace reader over one frozen bridge corpus. Require the same selected case identifiers, reference-token denominator, hypothesis text, and old WER result before interpreting any new fields. A disagreement at this stage is a reader, normalization, or aggregation migration defect, not an ASR change. Give every legacy row a declared evidence level so a transcript-only case remains eligible for text and intent checks while audio-dependent checks report “not available” instead of silently passing. Switch the release consumer only after the new path accounts for every bridge row and reproduces the old decision where both paths claim to answer the same question.

The first things to break are often fixtures and report assumptions. Old records may have one text field but no distinction between raw and normalized text. Dashboard code may treat no action as a successful refusal even when orchestration never ran. Test runners may time out while uploading audio after a failure. Retention jobs may delete the pre-ASR artifact while keeping only a derived WAV, removing the evidence needed to diagnose codec or channel conversion. Exercise those paths with forced failures before trusting the new gate.

This rollout has a concrete cost. Retaining both client-side and ASR-bound audio keeps two sensitive media artifacts for one case, increases encrypted storage and upload work, and doubles the locations that deletion and access controls must cover. End-to-end action validation also requires isolated test accounts or a safe simulator, which adds environment maintenance. Running live-media coverage outside the pull-request smoke path contains latency, but it delays discovery of device-specific failures until that scheduled job completes.

Ownership follows the first changed representation. The client or telephony team owns missing speech before transport. The media adapter owns a channel, codec, resampling, or segmentation loss before ASR. The recognition team owns a wrong completed transcript over correct input. The language-understanding team owns a wrong intent or entity from the correct normalized words. The orchestration team owns a correct intent transformed into the wrong action. Privacy and security owners define who may access the artifacts during every handoff. Send the case revision, consented fixture provenance, boundary digests and metadata, raw and normalized transcripts, event ordering, taxonomy version, entity values, action payload, and the first boundary where expected evidence changes. A clipped audio attachment without that chain merely moves the investigation.

Promote the gate slice by slice. Read-only intents can tolerate different risk than cancellation, transfer, or identity workflows. Languages and channel types need their own denominators. Require enough evaluated cases to make the report meaningful, and fail the job if a whole slice disappears due to collection or routing errors.

When changing ASR providers or models, replay the same frozen audio first. When changing capture or codecs, run the live path as well. When changing intent taxonomy, hold transcripts fixed. This controlled separation gives each migration a fair comparison and keeps teams from blaming whichever model name changed most recently.

Know when not to use one combined score

Do not gate a voice agent on WER alone. It weights common and critical words through the same edit count. Keep explicit assertions for negation, amounts, dates, destinations, identities, consent, and any field that controls a side effect.

Avoid intent-only evaluation when the taxonomy discards information needed by the action. transfer_money is not enough without source, destination, amount, currency, and confirmation state. A correct label with a wrong slot is not partial success for the customer.

Do not use synthetic speech as the only acoustic evidence. It is excellent for repeatable coverage and weak at representing the messy interaction between people, rooms, devices, networks, and duplex audio. Its cleanliness can make a fragile pipeline look mature.

Skip end-to-end live calls when diagnosing a deterministic normalizer. They add cost and noise without exercising the suspected rule more directly. Feed frozen transcripts to the normalizer, prove the bug, then keep a smaller end-to-end case to confirm wiring.

Finally, do not record or retain real callers merely to improve a benchmark. Privacy, consent, security, and deletion obligations set the boundary. A smaller lawful dataset with clear provenance is better engineering evidence than a huge corpus nobody can safely inspect when a release fails.

Correct transcription and intent do not prove who spoke. A replayed recording or unauthorized person can produce the expected words, entities, and action payload, so this evaluation may pass while authentication or consent has failed. Speaker verification, session authentication, liveness defenses where appropriate, and confirmation policy need their own tests. Do not treat semantic accuracy as proof of caller authority.