Usability Testing Guide: Plan, Run, and Report Better Tests

Six operations managers successfully exported a monthly report, yet four exported the wrong month. They clicked the prominent “Download” button before noticing that the date filter retained last month’s value. Functional tests passed because the file downloaded and matched the selected filter. The usability decision was different: could a busy manager understand the current scope before creating evidence for an audit?

Usability testing observes decisions, hesitation, errors, and recovery while representative people attempt realistic work. It is not a guided demo, a preference poll, or a substitute for accessibility testing.

Frame a decision the study can answer

Start with a product decision, such as whether the revised reporting flow is ready for general release or which of two filter summaries better prevents wrong-period exports. Name the target users, context, workflow, and risk.

A focused plan might state:

Decision: release the new monthly export flow to all operations managers
Users: managers who export at least once a month, mix of new and experienced
Context: desktop browser during a time-limited reporting task
Primary question: can participants select and verify the intended month before export?
Critical failure: participant submits a wrong period without recognizing it

The text block constrains recruitment, tasks, observation, and severity. “See whether users like the page” would not support a release decision.

Recruit for behavior, not convenience

Write screening criteria from actual product use: role, frequency, domain knowledge, device, and any workflow-specific responsibility. Avoid recruiting only teammates who know the design vocabulary. Internal users can help debug a script, but they are weak evidence for first-time comprehension.

Use a small focused round to discover clear problems, then run another round after changes. Do not claim population percentages from a tiny qualitative sample. If comparing variants quantitatively or segmenting several user groups, plan a larger study with appropriate research expertise.

For the export study, recruit participants across two meaningful dimensions: managers who export every month and managers who do it only at quarter end. Record relevant experience, not names in the report. Include accessibility needs in recruitment when people with those needs belong to the audience, and provide the accommodations required for genuine participation.

Write tasks without giving away the path

A task should describe a believable goal and the information a participant would have, without naming controls or copying interface labels. Instead of “Set the date filter to June and click Download,” say:

The task allows observation of how the participant discovers scope, verifies it, and decides completion. Supply realistic account state and data. Remove dead ends caused by unfinished prototypes unless those limitations are part of the study.

Pilot every task with someone outside the product team. Check that instructions are understandable, test data exists, links work, recording captures the relevant screen, and completion criteria can be observed. Pilot behavior does not belong in final results if the script then changes materially.

Define evidence before the first session

Create operational measures with unambiguous scoring:

Time alone is not quality. A quick wrong export is worse than a slower verified one. Count meaningful errors, backtracks, repeated clicks, and help. Capture quotations sparingly and with consent, using them to explain observed behavior rather than as decoration.

Set severity criteria in advance. One participant encountering a severe data-loss path may outweigh several cosmetic complaints.

Facilitate without teaching the interface

Use the same welcome, consent, recording explanation, and task wording for each session. Make clear that the product is being tested, not the participant. Ask participants to share what they are looking for, but accept that constant think-aloud can change timing and behavior.

When someone pauses, wait. Neutral prompts include “What are you thinking?” and “What would you do next?” Avoid “Did you see the date at the top?” because it directs attention and invalidates discovery evidence. If help is necessary to continue, give the smallest planned prompt and record assistance.

Do not defend the design or explain its intended behavior during the task. Save follow-up questions for afterward: what did the participant believe the date represented, when did they feel ready to export, and what would they verify before sending the file?

For remote sessions, have a recovery plan for dropped calls, blocked screen sharing, and private notifications. Ask participants to close unrelated windows and never request passwords or real sensitive data.

Capture observations as a timeline

Use timestamps and separate behavior from interpretation:

The table preserves what happened before the team decides why. Tag the recording at each critical moment and retain prototype version, browser, task order, participant segment, completion, assistance, errors, and duration.

Have a second observer take notes when possible. The facilitator cannot watch cursor, expression, task state, and timestamps equally well. Conduct a short debrief immediately after each session while context is fresh, but wait for cross-session evidence before redesigning between participants.

Diagnose patterns, not isolated opinions

Group observations by underlying breakdown: unclear system state, misleading hierarchy, terminology mismatch, hidden dependency, weak error recovery, or absent confirmation. A participant may ask for a larger button, but the behavior may show they never understood which month was selected. Fix the information problem, not necessarily the requested pixel change.

Build an evidence matrix:

Repeated behavior increases confidence, but frequency is not the only severity input. Consider task consequence, recoverability, visibility, and whether the user can detect the error before harm.

Connect usability findings to functional evidence

QA can strengthen a finding by checking the resulting artifact and system state. For each export, inspect file metadata, filter summary, row date range, and audit event. This distinguishes misunderstanding from a backend defect.

If a participant selected June but received May, file a functional defect with request and data evidence. If the participant never noticed May remained selected, report a usability finding. If the control changed but the summary did not, both are involved. Preserve these distinctions because owners and verification methods differ.

Also record positive evidence. If every participant correctly interpreted a new inline summary, the team should protect that behavior during redesign rather than treating only problems as findings.

Prioritize and report for action

A useful finding states the affected users and task, observed behavior, consequence, evidence, likely interaction breakdown, and a testable recommendation. Avoid claiming a root cause the study did not prove.

For example: “Four of six managers exported a retained month before noticing the current filter. All four expected the report to reset for a new session. Add a persistent pre-export scope summary adjacent to the action and require confirmation when filters came from an earlier session. Retest the same task with monthly and quarterly users.”

Include participant characteristics, method, prototype version, tasks, operational definitions, results by task, limitations, clips, and unresolved questions. Report counts as observations from this study, not market estimates. Protect identities and follow the agreed retention policy.

Retest the behavior and make the decision

Convert the critical finding into acceptance evidence. In the revised flow, participants should identify June before submission, the exported file should contain only June records, and no moderator prompt should be needed. Keep task goal and scoring stable enough to compare, while acknowledging different participants and learning effects.

A release decision can require zero unrecognized wrong-period exports, successful recovery from an intentionally stale filter, and no severe accessibility blocker found in the complementary accessibility review. Document any residual hesitation and why it is acceptable.

Usability work is complete when the team can connect an observed behavior to user consequence, choose a change, and verify the changed behavior. A collection of opinions or polished clips is not that evidence.