Chatbot Testing Checklist and 20 Essential Test Cases

A chatbot testing checklist follows the user goal into the system of record. A greeting cannot compensate for a forgotten constraint, fabricated policy, or duplicate transaction. Use it for features and every model, prompt, or knowledge change.

For metric definitions, read how to test a chatbot.

Pre-Test Setup Checklist

• Define supported users, goals, channels, languages, and escalation paths.
• List prohibited content, actions, data, and claims.
• Create safe accounts and resettable backend state.
• Label intent utterances, including ambiguous and out-of-scope input.
• Version prompts, dialogue rules, models, knowledge, connectors, tools, and evaluators.
• Protect or remove PII from production-derived conversations.
• Define task success in the backend, not only in the final message.
• Set separate blocking rules for safety, privacy, authorization, and critical outcomes.

Twenty Chatbot Test Cases

Each case needs an ID, initial state, turns, accepted and prohibited behavior, tool effects, severity, and cleanup. Accept valid paraphrases, using exact strings only for fixed wording or identifiers.

Execution Evidence Flow

1. 01 / case

   Versioned Case

   Goal, state, turns, risk, and oracle.

   run

2. 02 / conversation

   Conversation

   Capture messages, memory, and timing.

   observe

3. 03 / effects

   Tool Effects

   Verify calls, permissions, and backend state.

   score

4. 04 / evaluation

   Evaluation

   Check task, context, fallback, and safety.

   gate

5. 05 / decision

   Decision

   Gate severe failures independently.

Intent and Entity Checklist

• Report confusion, not only overall intent accuracy.
• Include fragments, typos, negation, overlaps, and out-of-scope input.
• Test missing, ambiguous, repeated, corrected, and boundary entities.
• Prevent invented high-impact values.
• Segment by intent, locale, channel, and severity.

Context and Golden-Dialogue Checklist

• Assert state after relevant turns.
• Test corrections, topic changes, returns, timeouts, and handoffs.
• Prove session isolation.
• Store accepted and prohibited behavior, not one transcript.
• Add minimized production failures and freeze comparisons.

DeepEval provides conversational cases and metrics. promptfoo supports declarative evals and simulated users. See the DeepEval tutorial and promptfoo tutorial. Cyara Botium is alive as an enterprise platform.

Fallback and Safety Checklist

• Test nonsense, out-of-scope questions, unavailable services, and unauthorized goals.
• Verify recovery, escalation, and handoff without apology loops.
• Exercise toxicity, jailbreaks, injection, PII, secrets, and cross-account requests.
• Inspect tool calls and route severe disputes to human review.

Use deterministic checks for schemas, secrets, permissions, identifiers, and side effects. Use calibrated, versioned metrics for meaning and check them against human labels.

Latency and Task-Success Checklist

• Measure first output, turn time, tool time, and total resolution.
• Report percentiles and errors with task success.
• Verify backend state, authorization, idempotency, and confirmation.
• Test timeouts around tool calls and preserve correlation IDs.

Release Checklist

• Run critical cases on every relevant change and broad suites on schedule.
• Review risk slices, not only aggregates.
• Block severe authorization, privacy, safety, and task failures.
• Require owner and expiry for exceptions.
• Preserve original failures and every component version.

Frequently Asked Questions

What belongs in a chatbot testing checklist?

Cover supported goals, intent and entity handling, multi-turn context, out-of-scope fallback, safety, privacy, authorization, latency, backend task success, session isolation, channel behavior, and versioned golden-dialogue regression.

How do I write expected results for a generative chatbot?

Define required facts, supported evidence, prohibited claims, tone or policy boundaries, tool effects, and final task state. Avoid one exact reference sentence unless the wording itself is contractually fixed.

How should chatbot latency be tested?

Measure each turn, first visible response when streaming, tool-call time, total time to resolution, error rate, and task success. Report percentiles by model, connector, channel, and outcome rather than one average.

Can Cyara Botium test chatbots?

Yes. Botium is alive as Cyara Botium and is an enterprise conversational AI testing option. Validate its current connectors and workflow against your channels, bot engine, data controls, and release process.

Which chatbot failures should block release?

Block unauthorized actions, PII or secret leakage, severe safety violations, wrong high-impact task outcomes, cross-session data exposure, and regressions in critical golden dialogues. Do not let average quality offset them.

Keep the Checklist Risk-Based

Use this checklist to protect high-value goals, dangerous transitions, production failures, and the evidence chain to verified outcome.