When a helpful AI conversation still leaves the job unfinished

A support bot says a refund is complete, but it never tells the customer which card will receive the money or when to expect it. The final message sounds polished, the sentiment score is high, and the user still has to ask another question. That is an incomplete conversation hiding inside a plausible answer.

The difficult part is not spotting an obviously truncated sentence. It is proving which user outcomes were left open across several turns, without rewarding verbosity or punishing a concise answer. A useful evaluation follows the work, not the word count.

Why a fluent final turn can still leave work open

Conversation completeness is a property of the whole interaction. The final assistant message matters, but it is only one piece of evidence. A booking flow might collect a date in turn two, confirm a location in turn four, and issue a reservation identifier in turn six. Looking only at turn six loses the facts that made the outcome valid.

Start with obligations. An obligation is something the system must establish before the conversation may be called complete. It might be a fact delivered to the user, a choice obtained from the user, a successful tool result, a warning shown before an irreversible action, or an honest statement that the task cannot continue. Each obligation needs an identifier and a rule for what counts as evidence.

That last point prevents a common testing mistake. The sentence “Your refund is handled” is not evidence that a refund tool succeeded. It is only evidence that the assistant claimed success. If the contract requires a successful transaction, the evaluator must inspect the recorded tool outcome or an application event. If the contract only requires an explanation of policy, text may be enough.

Completeness also has allowed exits. Suppose a user asks to cancel an order that has already shipped. The bot cannot satisfy the original cancellation outcome, but it can still complete the conversation by explaining the limitation and offering the supported return path. Marking every blocked task incomplete would confuse product constraints with assistant failures. The case contract should name acceptable terminal states such as completed, blocked_with_recovery, declined_safely, or needs_user_input.

Do not collapse correctness and completeness into one score. A response can cover every requested item and give the wrong values. It can also give one correct value while omitting two others. Those defects need different owners. Retrieval, tool integration, and domain logic often cause correctness failures. Prompt structure, state management, stopping rules, and response assembly often cause completeness failures.

Consider three conversations that all end with “Done”:

1. A refund tool succeeds, and the bot provides the amount, destination, and expected processing window. The conversation is complete if those are the contracted outcomes.
2. The tool succeeds, but the response omits the destination. The action may be correct while the user-facing conversation remains incomplete.
3. No tool call occurs, but the response claims success. This is not merely incomplete. It is an unsupported action claim, and the product should usually treat it as a higher-severity defect.

The mechanism is therefore a small state machine. User intent opens obligations. Later turns satisfy, replace, or invalidate them. Tool events can supply stronger evidence than prose. A terminal decision is valid only when no required obligation remains open and the chosen exit state is permitted.

Turn product requirements into observable obligations

Write the case before writing the grader. Product language such as “help the customer finish a return” is too broad for a repeatable test. A better case names the expected branch and the observable outcomes for that branch. It does not prescribe exact wording unless exact wording is itself a legal or safety requirement.

The following TypeScript is a complete deterministic evaluator for explicit evidence labels. It uses no model API. The conversation runner or a human annotation step supplies the evidence labels, and the evaluator decides whether the case is complete.

type ExitState =
  | "completed"
  | "blocked_with_recovery"
  | "declined_safely"
  | "needs_user_input";

type Obligation = {
  id: string;
  description: string;
  requiredFor: ExitState[];
};

type ConversationCase = {
  id: string;
  expectedExit: ExitState;
  obligations: Obligation[];
};

type ObservedConversation = {
  caseId: string;
  exitState: ExitState;
  evidenceIds: string[];
};

type CompletenessResult = {
  caseId: string;
  passed: boolean;
  missing: string[];
  unexpectedExit: boolean;
};

export function evaluateCompleteness(
  testCase: ConversationCase,
  observed: ObservedConversation,
): CompletenessResult {
  if (testCase.id !== observed.caseId) {
    throw new Error(`Case mismatch: ${testCase.id} != ${observed.caseId}`);
  }

  const required = testCase.obligations.filter((obligation) =>
    obligation.requiredFor.includes(observed.exitState),
  );
  const seen = new Set(observed.evidenceIds);
  const missing = required
    .filter((obligation) => !seen.has(obligation.id))
    .map((obligation) => obligation.id);
  const unexpectedExit = observed.exitState !== testCase.expectedExit;

  return {
    caseId: testCase.id,
    passed: !unexpectedExit && missing.length === 0,
    missing,
    unexpectedExit,
  };
}

const refundCase: ConversationCase = {
  id: "refund-approved-card",
  expectedExit: "completed",
  obligations: [
    {
      id: "refund_amount",
      description: "User receives the approved amount",
      requiredFor: ["completed"],
    },
    {
      id: "refund_destination",
      description: "User knows which payment method receives the refund",
      requiredFor: ["completed"],
    },
    {
      id: "refund_timing",
      description: "User receives the supported processing window",
      requiredFor: ["completed"],
    },
  ],
};

const result = evaluateCompleteness(refundCase, {
  caseId: "refund-approved-card",
  exitState: "completed",
  evidenceIds: ["refund_amount", "refund_timing"],
});

console.log(JSON.stringify(result, null, 2));
process.exitCode = result.passed ? 0 : 1;

The example fails with refund_destination in missing. That output is useful because it names the unsatisfied contract, not a vague score of 0.67. The fraction can be reported for trends, but the identifier is what helps an engineer reproduce and fix the defect.

Evidence labels need their own rules. For refund_timing, decide whether “soon” qualifies. It usually should not. A test case can require a supported range, while allowing different natural-language forms of that range. For refund_destination, a masked card description may qualify, but a generic phrase such as “your account” may not. Those decisions belong in the rubric and calibration examples, not in a judge prompt assembled during the run.

Use semantic judgment only where string or event checks cannot express the requirement. An evaluator may need to decide whether a troubleshooting explanation clearly tells the user how to verify the fix. If so, ask a judge one narrow question about that obligation. Do not ask it to assign an overall impression of completeness. Narrow graders are easier to calibrate, and their disagreements tell you which requirement is ambiguous.

A judge result should retain at least the case ID, obligation ID, binary or small-label decision, reason, grader version, and the exact text it saw. Without those fields, a score change cannot be separated from a product change, transcript assembly bug, or grader update.

Diagnose the missing outcome before changing the prompt

When a case fails, inspect the evidence path in order. First confirm that the expected branch is correct. Then check whether the required fact or event exists anywhere in the trace. Finally check whether the evaluator observed it. This order avoids “fixing” a prompt when the real defect is a broken event collector.

A diagnostic record should make that path visible. The next script reads newline-delimited evaluation results, prints only failures, and distinguishes an unexpected exit from missing obligations. It is runnable with Node after TypeScript compilation or directly with a TypeScript runner.

import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";

type DiagnosticRow = {
  caseId: string;
  expectedExit: string;
  observedExit: string;
  requiredEvidence: string[];
  observedEvidence: string[];
  transcriptId: string;
  graderVersion: string;
};

const input = process.argv[2];
if (!input) throw new Error("Usage: diagnose-completeness <results.ndjson>");

const lines = createInterface({
  input: createReadStream(input, "utf8"),
  crlfDelay: Infinity,
});

let failures = 0;
for await (const line of lines) {
  if (!line.trim()) continue;
  const row = JSON.parse(line) as DiagnosticRow;
  const seen = new Set(row.observedEvidence);
  const missing = row.requiredEvidence.filter((id) => !seen.has(id));
  const exitMismatch = row.expectedExit !== row.observedExit;
  if (!exitMismatch && missing.length === 0) continue;

  failures += 1;
  console.error(
    JSON.stringify({
      caseId: row.caseId,
      transcriptId: row.transcriptId,
      exitMismatch,
      expectedExit: row.expectedExit,
      observedExit: row.observedExit,
      missing,
      graderVersion: row.graderVersion,
    }),
  );
}

if (failures > 0) {
  console.error(`COMPLETENESS_GATE failed=${failures}`);
  process.exitCode = 1;
}

For the refund fixture above, a useful console record looks like this. These values are illustrative output from the shown data shape, not measurements from a production system.

$ npx tsx scripts/diagnose-completeness.ts artifacts/completeness.ndjson
{"caseId":"refund-approved-card","transcriptId":"tr_1042","exitMismatch":false,"expectedExit":"completed","observedExit":"completed","missing":["refund_destination"],"graderVersion":"obligation-rubric-3"}
COMPLETENESS_GATE failed=1

Open the transcript and trace for tr_1042. If the destination never appears, the system response is incomplete. If the destination appears in a tool result but not in the assistant message, response assembly or prompt instructions are likely involved. If it appears in the assistant message but observedEvidence omits it, the evaluator or redaction pipeline is wrong. The same missing ID leads to three different fixes.

Worked example: troubleshooting stops before verification

A Wi-Fi support flow asks the user to restart a router. The user replies that the lights are back, and the assistant says, “Great, that should do it.” Product requirements say the flow is complete only after the user verifies connectivity or receives a concrete verification step.

The open obligation is verify_connectivity. No tool integration is necessary to detect it if the transcript clearly lacks a check. The fix may be a stopping-condition change that keeps the conversation active until the user confirms a result. The trade-off is another turn, more latency, and a chance that the user abandons the chat. That cost is still preferable if the product currently records unverified attempts as successful resolutions.

Now change the transcript: the assistant says, “Open any website to confirm the connection.” The user leaves. This may be a valid needs_user_input exit rather than an incomplete completed exit. The expected state depends on how the product defines session timeout. A test that forces completed would report a false defect.

Worked example: scheduling loses a constraint

A user asks for “Tuesday after 3 PM with Dr. Mehta.” The system books Tuesday at 2 PM with the requested clinician. All major entities appear in the transcript, but one constraint was violated. This is not primarily completeness. The conversation captured the requested constraints; execution selected an invalid slot. A constraint-correctness check should fail, and the completeness evaluator should report that the outcome was present but invalid rather than claim it was missing.

That distinction changes the investigation. Check the tool arguments and returned availability, not the final response length. If the booking tool received after=15:00 and returned 14:00, the integration or upstream service is suspect. If the tool received no time boundary, state extraction or argument construction lost it.

Worked example: a safe refusal is complete

A user asks a financial assistant to transfer money without completing the required verification. A short refusal that explains the blocked condition and gives the supported verification path can be complete. Adding more prose does not improve the outcome. A naive rubric that expects the transfer confirmation will mislabel safe behavior as failure and pressure the team toward a dangerous change.

Model each policy branch separately. The permitted refusal case should require verification_required and recovery_path, while explicitly forbidding transfer_confirmed. Completeness is not obedience to the initial request. It is closure under the product’s allowed behavior.

Separate completeness from failures that look similar

The closest near-miss is state loss. Both defects produce a missing item in the final answer, but the evidence differs. With a response-planning failure, the necessary fact remains in the assembled model input or application state and is omitted from the output. With state loss, the fact disappears before generation.

Inspect the exact input representation captured for the failing turn. If the user’s constraint is absent there, changing the response prompt can only mask the pipeline defect. Check transcript truncation, summarization, serialization, session keys, and event ordering. If the constraint is present but ignored, then prompt structure, model behavior, or output validation becomes a reasonable target.

Truncation has a recognizable boundary. Failures cluster in long conversations, and early-turn obligations disappear together. A summarizer defect looks different: the summary remains present but changes a value or drops a category selectively. A session-mix-up may insert evidence from another conversation. Record turn IDs and source event IDs so those patterns are visible.

Another near-miss is evaluator blindness caused by redaction. Suppose the compliance pipeline replaces a card suffix with [PAYMENT_TOKEN], while the semantic grader expects a phrase such as “card ending 4242.” The user may have received the correct destination, yet the post-redaction evaluator cannot prove it. Either emit a non-sensitive evidence event before redaction or teach the rubric what the placeholder means. Do not disable redaction to make the test pass.

Tool latency can also resemble incompleteness. If the transcript ends while a tool is still pending, the captured sample is not a completed conversation. Mark it incomplete_capture and exclude it from product-quality denominators. Otherwise infrastructure timeouts will appear as response-quality regressions. Keep a separate operational metric for abandoned or timed-out runs because those still affect users.

A clarification turn creates another boundary case. When the user says, “Book the usual place,” and no reliable preference exists, asking for the location is progress, not a completed booking and not a defect. Record which obligation the question is trying to resolve, such as booking_location, and use needs_user_input while that obligation remains open. If the assistant asks again after the user already supplied the location, the trace points to state loss. If it guesses a location and books it, the problem is an unauthorized assumption. Those three outcomes can end with similar short messages, so the exit state and stored evidence matter more than the surface wording.

Judge drift is visible when deterministic evidence stays constant but semantic obligation labels change after a grader update. Rerun a fixed calibration set before accepting the new judge version. Compare per-obligation disagreements, not only the average score. A two-point overall change may come from one newly strict rubric category rather than a broad product regression.

Finally, distinguish optional helpfulness from required closure. A travel assistant might add weather advice, baggage tips, or a map link. Those can improve experience, but making all of them required creates a verbosity ratchet. Obligations should trace to user intent, policy, or an explicit product promise. Nice extras belong in a separate helpfulness evaluation.

Tell an incomplete reply from the wrong evaluated attempt

The most deceptive look-alike produces the exact diagnostic already shown: the expected and observed exits are both completed, and missing contains refund_destination. In one run, the delivered assistant turn genuinely omitted the destination. In another, the assistant produced a first draft without it, retried after a tool result, and delivered a corrected turn, while the evaluation pipeline assembled the first attempt. The user experiences a complete conversation in the second case, but the report points at the same missing obligation.

Separate the two by joining three artifacts, not by rereading the score alone. Identify the assistant turn the channel actually delivered, the generation attempt that produced it, and the transcript snapshot the evaluator received. Their turn identity and safe content digest should agree. The signature of the selection defect is a delivery record pointing to the corrected attempt while the evaluation package contains the earlier draft. A shared conversation ID is not precise enough when one conversation can contain drafts, retries, or regenerated turns. Preserve the attempt relationship at capture time because reconstructing it later from timestamps is unreliable when work overlaps.

The diagnostic fields each answer a narrower question. caseId selects the obligation contract. transcriptId locates the evidence package, but it is healthy only if that package identifies the delivered attempt. expectedExit and observedExit expose a branch disagreement. missing names the open obligations under the observed branch. graderVersion identifies the interpretation applied to the evidence. A healthy completed refund has matching exits, an empty missing list, and matching evaluated and delivered attempt identities. A broken reply keeps the identity match but has a nonempty missing list. A misleading result has the same nonempty list while the attempt identities disagree. In that last case, missing may accurately describe the wrong draft, which is why changing the obligation rubric would be the wrong repair.

This fault should change owners as evidence moves. The conversation runtime team owns an evaluator snapshot tied to the wrong attempt. The response or prompt owner takes the issue only when the correct state reached generation and the delivered reply still omitted the outcome. The evidence-extraction owner takes a label missed in matching text. The product owner decides whether the obligation and exit branch are valid. The handoff needs the case-contract version, conversation and delivered-turn identities, generation-attempt identity, tool-result identities, redacted delivered and evaluated text, extracted evidence IDs, grader version, and the first point where those artifacts disagree. Sending only missing=["refund_destination"] invites each team to reproduce a different conversation.

There is a specific maintenance cost to this evidence model. Retry-aware capture must retain the relationship among drafts, delivered turns, tool events, and evaluation snapshots instead of storing one flattened transcript. Each additional allowed exit also needs its own required-obligation mapping and reviewed examples. For flows with many policy and availability branches, fixture work grows with the supported branches even if the conversational wording stays short. That cost buys a defensible answer to which job was actually completed, but it should be reserved for workflows with a meaningful terminal outcome.

Server-side completeness does not catch a delivery defect after the transcript is recorded. A web client can hide the last paragraph, a voice channel can cut off before the timing statement, or an accessibility layer can omit a status update while the stored assistant turn remains complete. Cover that boundary with channel delivery evidence and client-level tests. A green transcript evaluator only proves closure in the artifact it received.

Roll the check into CI without creating a brittle gate

Begin in report-only mode. Run the evaluator on a small, reviewed dataset and publish the missing obligation IDs. For the first rollout, do not block merges. The team needs to discover mislabeled branches, unstable judge rules, missing telemetry, and duplicated cases before trusting the signal.

Next, split cases by oracle strength. Deterministic obligations backed by events or exact structured fields can gate earlier. Semantic obligations should gate only after calibration against human labels and after the team sets an acceptable disagreement policy. A single aggregate threshold hides which group failed, so report both case-level pass rate and counts by obligation.

The CI job below assumes the repository owns scripts named in the commands. It stores results even when the gate fails, which matters because a red job without its evidence is hard to debug.

name: conversation-completeness

on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/conversation/**"
      - "evals/conversation/**"
  workflow_dispatch:

jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - name: Run representative conversations
        run: npm run eval:conversation -- --out artifacts/completeness.ndjson
      - name: Check required outcomes
        run: npx tsx scripts/diagnose-completeness.ts artifacts/completeness.ndjson
      - name: Upload evaluation evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: conversation-completeness-results
          path: artifacts/completeness.ndjson

Use pull-request evaluation for a compact, high-signal set. Run broader slices on a schedule or before a release candidate. A suite that calls models hundreds of times on every documentation change wastes money and delays unrelated work. Path filters help, but they are not a security boundary. A scheduled full run should still catch dependencies the filter misses.

Set release policy at the case level for critical obligations. “No regression in consent confirmation cases” is more defensible than “overall completeness must exceed 92 percent.” The latter can pass after a severe case fails if several easy cases improve. For lower-risk categories, a slice threshold with a minimum sample count may be appropriate.

Version the case contract, evidence extractor, and semantic grader separately. Store all three versions in the result row. When a test changes, rerun the previous product candidate with the new evaluator if possible. Comparing candidate A under rubric 2 with candidate B under rubric 3 does not isolate the product change.

Migration from an existing exact-match suite works best in layers. Keep current assertions while adding obligation IDs beside them. Replace only the assertions that reject legitimate paraphrases. Run old and new decisions together, review every disagreement, and remove the old rule after its unique catches are either represented in the new contract or deliberately retired.

The first migration failures usually expose identity and lifecycle gaps, not weak prose. Older fixtures may not say which assistant turn was delivered, may label a timed-out capture as completed, or may combine tool results from a rerun with text from the original attempt. Land stable case, turn, attempt, and evidence relationships before adding semantic obligation judgments. Next, encode deterministic event-backed obligations and allowed exits, then run the evidence extractor in shadow mode. Promote a rule only after the old assertion and new contract disagreements have an owner and resolution. The rollout is working when every evaluated transcript maps to one delivered terminal attempt, repeated evaluation of the same frozen artifact gives the same deterministic labels, and each remaining semantic disagreement links to a calibration example rather than an unexplained aggregate score.

Accept the cost, and know when not to use this gate

Better completeness evidence is not free. Event-backed checks require instrumentation and stable identifiers. Semantic checks add model calls, latency, and grader maintenance. Multi-turn fixtures take longer to run than single prompts. Human calibration consumes reviewer time, especially when product requirements were vague to begin with.

The most expensive mistake is evaluating every possible nice-to-have as an obligation. The suite grows, responses become bloated to satisfy it, and reviewers spend time debating low-value omissions. Keep the contract small. Add an obligation when its absence changes whether the user can finish the task, understand the outcome, recover safely, or make an informed decision.

Do not use a completeness gate for open-ended brainstorming. There may be no finite set of required outcomes, and forcing one can reward formulaic responses. Evaluate relevance, diversity, factuality, or user preference instead. The same warning applies to casual conversation, where graceful continuation may matter more than terminal closure.

Avoid the gate when the trace is partial. If privacy controls remove the very evidence needed to decide an obligation, first create a safe structured signal or accept human review. Guessing from a damaged transcript produces confident but unauditable labels.

Do not let this metric approve unsafe or incorrect behavior. Completeness is one release dimension. Safety, correctness, policy adherence, latency, and tool execution need their own checks. A fully complete explanation of the wrong refund amount is still wrong.

Pause automatic blocking when the product contract is changing faster than the dataset. During a redesigned flow, report per-obligation results and review them with product owners. Re-enable the gate after expected branches and allowed exits stabilize. A stale gate does not preserve quality; it preserves yesterday’s workflow.

The practical standard is simple: every failed result should tell an engineer which user outcome stayed open, where the supporting evidence disappeared, and which rule made that outcome required. If the evaluation cannot answer those three questions, improve the evidence model before tightening the threshold.