Keyword Driven vs Data Driven Testing: Clear Guide

A regression team moved 240 login cases into a spreadsheet so analysts could maintain them without code changes. Six weeks later, a button rename broke every row because each test also stored the same click sequence. The team called the framework data driven, but the real source of duplication was workflow. Another group solved that problem with keywords, then created one keyword per test and merely moved the scripts into YAML.

The keyword driven vs data driven testing decision is not a choice between two competing tools. Data driving varies inputs and expected results for one behavior. Keyword driving composes behavior from a controlled vocabulary of actions. A suite can use either, both, or neither. The right design follows the kind of change the suite must absorb.

Separate the two design axes

Data-driven tests keep the procedure in code and supply rows from an array, file, database, or generator. One password policy test might run for empty, short, valid, and overlong values. The framework is responsible for parsing, validating, naming, and isolating each row.

Keyword-driven tests represent a procedure as named operations such as Create customer, Add item, and Verify total. An engine maps each keyword to implementation code. The vocabulary becomes an interface between test authors and automation maintainers.

Neither approach guarantees maintainability. Externalizing values can hide intent, and externalizing actions can create a programming language with poor tooling. Compare the ongoing authoring model, not the first demo.

The distinction also clarifies reviews. A changed data row should prompt questions about coverage and expected results. A changed keyword should prompt questions about every workflow that calls it. Treating both as ordinary spreadsheet edits hides their very different blast radii.

Build a data-driven test with typed rows

Keep nearby data in code when developers and automation engineers own it. Types, review diffs, and editor support are usually better than a spreadsheet. This Playwright example validates login outcomes without sharing a browser session between rows:

import { expect, test } from "@playwright/test";

type LoginCase = {
  name: string;
  email: string;
  password: string;
  outcome: "dashboard" | "invalid-credentials" | "locked-account";
};

const cases: LoginCase[] = [
  {
    name: "active customer",
    email: "active@example.test",
    password: "Valid-Password-7",
    outcome: "dashboard"
  },
  {
    name: "wrong password",
    email: "active@example.test",
    password: "wrong",
    outcome: "invalid-credentials"
  },
  {
    name: "locked customer",
    email: "locked@example.test",
    password: "Valid-Password-7",
    outcome: "locked-account"
  }
];

for (const row of cases) {
  test(`login: ${row.name}`, async ({ page }) => {
    await page.goto("/login");
    await page.getByLabel("Email").fill(row.email);
    await page.getByLabel("Password").fill(row.password);
    await page.getByRole("button", { name: "Sign in" }).click();

    if (row.outcome === "dashboard") {
      await expect(page).toHaveURL(/\/dashboard$/);
    } else {
      await expect(page.getByTestId("login-error")).toHaveAttribute(
        "data-code",
        row.outcome
      );
    }
  });
}

The row has a diagnostic name and a constrained outcome instead of an arbitrary expected message. UI copy can change without weakening the domain assertion. Each generated test gets normal setup, reporting, retries, and an independent result.

External files make sense when data has a different owner, changes on a different schedule, or is generated by another system. Add schema validation at load time. A CSV cell is always text, so false, an empty string, and a missing value can be confused unless parsing rules are explicit.

Choose cases by risk, not by available rows

Data-driven suites often grow through copying. Ten valid countries become two hundred combinations of countries, currencies, devices, and account types, even though most combinations exercise the same branch. Execution cost rises while defect detection barely changes.

Start from equivalence classes, boundaries, decision tables, and production risks. Give each row a reason. If a tax rule depends on country and customer category, cover the combinations that change the decision. Use pairwise generation only after defining constraints and reviewing what interactions matter. Generated data should include a seed or persisted failing case so a failure can be reproduced.

Keep secrets and personal data out of datasets. Reference a credential alias resolved by the test environment instead of storing a password in CSV. Synthetic records should still obey realistic formats and relationships. A test with an impossible postal code may pass validation paths that real data never reaches.

Verify a dataset by intentionally corrupting a required field, duplicating a case ID, and adding an unsupported expected outcome. Loading must fail before browser or API actions begin. Silent coercion is a framework defect because it can make a malformed case appear green.

Design keywords as a stable domain API

Good keywords describe capabilities, not individual clicks. Submit refund can change from three UI interactions to one API call while preserving its meaning. Click blue button exposes presentation details and cannot explain business intent.

A compact YAML workflow could look like this:

name: refund a captured card payment
steps:
  - keyword: create_customer
    args:
      alias: buyer
  - keyword: capture_payment
    args:
      customer: buyer
      amount_minor: 2500
      currency: INR
  - keyword: request_refund
    args:
      payment: last_payment
      amount_minor: 2500
  - keyword: verify_payment_status
    args:
      payment: last_payment
      expected: refunded

The vocabulary is intentionally small and domain focused. Aliases let later operations refer to created entities without exposing generated IDs to the author. Monetary units are explicit, which avoids decimal and currency ambiguity.

Treat keyword names and arguments as a versioned public interface. Define required fields, types, defaults, produced context values, and failure behavior. Reject unknown arguments. If misspelling amount_minor is ignored, the engine may use a default and report a dangerous false pass.

Implement a transparent keyword engine

The dispatcher should be boring. It validates the document, finds a registered operation, executes it with scenario-scoped context, and records structured evidence. Avoid eval, reflection over arbitrary method names, or code embedded in cells.

type Args = Record<string, unknown>;
type Context = {
  values: Map<string, string>;
  api: { post(path: string, body: unknown): Promise<Response> };
};
type Keyword = (args: Args, context: Context) => Promise<void>;

const keywords: Record<string, Keyword> = {
  create_customer: async (args, context) => {
    if (typeof args.alias !== "string") {
      throw new Error("create_customer.alias must be a string");
    }
    const response = await context.api.post("/test-support/customers", {});
    if (response.status !== 201) throw new Error(`customer setup returned ${response.status}`);
    const body = await response.json() as { id: string };
    context.values.set(args.alias, body.id);
  },
  verify_payment_status: async (args, context) => {
    if (typeof args.expected !== "string" || typeof args.payment !== "string") {
      throw new Error("verify_payment_status requires payment and expected");
    }
    const actual = context.values.get(`${args.payment}:status`);
    if (actual !== args.expected) {
      throw new Error(`expected payment status ${args.expected}, received ${actual}`);
    }
  }
};

export async function runStep(
  step: { keyword: string; args?: Args },
  context: Context
): Promise<void> {
  const operation = keywords[step.keyword];
  if (!operation) throw new Error(`unknown keyword: ${step.keyword}`);
  await operation(step.args ?? {}, context);
}

In a real engine, validate the complete YAML against a schema before execution and add timing, attachments, and redaction around runStep. Report the workflow name, step index, keyword, sanitized arguments, and underlying cause. Authors should not need to search runner logs to learn which operation failed.

Keyword frameworks fail when they reproduce a general-purpose language badly. Conditions, loops, variables, imports, and exception handling quickly create programs without a compiler or debugger. Keep branching inside tested keyword implementations. When authors need complex control flow, ordinary test code is the more honest tool.

Combine approaches only where each adds value

A hybrid framework can run the same workflow against several meaningful datasets. For example, a refund workflow may use card and wallet rows with different settlement expectations. The keyword sequence stays stable while provider, currency, amount, and expected status vary.

Do not build a Cartesian product automatically. A workflow with twelve steps and fifty rows produces six hundred operation executions, often through a slow UI. Decide which variations need the full workflow and which belong in API or unit tests. One UI case can prove wiring while lower layers cover calculation boundaries.

Keep workflow definitions and datasets independently identifiable. A result named refund.yaml / wallet-delayed-settlement is actionable. row 18 failed at step 9 is not. Reports should record the keyword library version and dataset revision so reruns use the same inputs and behavior.

Hybrid designs also need ownership boundaries. Domain analysts may propose rows and review workflows, but automation maintainers should own keyword implementation, schemas, and runner releases. Without that separation, an apparently harmless table edit can alter execution semantics.

Decide using repository evidence

Choose data driving when one procedure is stable and defects cluster around input partitions, boundaries, locales, roles, or expected results. Choose keyword driving when a genuine non-coding author group needs to compose a limited set of recurring domain workflows and the organization can maintain the vocabulary as a product.

Stay with direct test code when engineers own the suite, workflows are unique, refactoring support matters, or the proposed keyword layer merely renames library calls. A page object or API client already provides reuse without adding a separate file format.

Run a pilot with five representative tests, including setup failure and a product defect. Measure review clarity, time to diagnose, required code changes, and whether intended authors can safely modify cases. Also make a UI or API change that should be absorbed by one abstraction. The location and size of the repair reveal whether the design actually isolates change.

Govern the suite after adoption

For data-driven tests, review row purpose, schema validity, uniqueness, sensitive fields, and execution growth. Remove redundant cases when the same rule is covered more cheaply elsewhere. Assign stable case IDs if results feed release reporting.

For keywords, maintain a searchable catalog with argument contracts and examples. Deprecate operations before removing them, find all workflow references in CI, and prevent synonyms from multiplying. A pull request that adds check_order, verify_order, and assert_order should trigger a vocabulary discussion.

Both styles need isolated state, deterministic cleanup, useful artifacts, and clear failure ownership. Neither should conceal flaky waits or environment coupling behind a friendly format. Test the framework itself: parser errors, invalid documents, duplicate IDs, missing keywords, cleanup failure, secret redaction, and parallel execution all deserve automated checks.

The strongest comparison is a maintenance exercise. Change a locator, add a required API field, revise a business rule, and inspect how many artifacts must change. If data varies without duplicating procedure, the data-driven layer is earning its cost. If workflows remain readable while keyword internals absorb technical change, the keyword layer is doing its job. If both changes ripple through spreadsheets and engine code, simplify the architecture.