How to Build a Test Automation Framework from Scratch

If you need to build a test automation framework from scratch, do not start by creating twenty abstract base classes. Start by deciding what the framework must make easy: writing reliable tests, running them in CI, diagnosing failures, and changing the product without rewriting the suite. A framework is not a trophy architecture. It is a set of conventions and shared tools that keep automation valuable after the demo day glow fades.

This guide shows a practical way to design a hybrid, scalable test automation framework: goals, layers, folder structure, design patterns, configuration, reporting, logging, CI integration, and the mistakes that create unmaintainable platforms. The examples lean toward Playwright and TypeScript, but the architecture applies to Selenium, Cypress, REST clients, and mixed suites.

What a Test Automation Framework Really Is

A test automation framework is the reusable structure around your tests. It usually includes:

• A test runner and assertion library
• Project layout and naming conventions
• Shared setup and teardown
• UI page objects or API clients
• Test data strategy
• Configuration for environments and secrets
• Logging, screenshots, traces, and reports
• CI commands and artifact publishing

It is not merely "Selenium plus TestNG" or "Playwright plus a utils folder." Tools are ingredients. The framework is how your team cooks with them every day.

Framework vs Test Suite vs Tool

Term	Meaning	Example
Tool	Browser or protocol driver	Playwright, Selenium, REST client
Suite	The tests you own	Smoke, regression, API contract packs
Framework	Shared architecture and conventions	Fixtures, POM, config, reporters, CI glue

Teams often blame tools for framework problems. Switching from Selenium to Playwright will not fix random waits, shared mutable test users, or missing reports.

Decide Goals Before Structure

Write the non negotiable outcomes first.

Typical goals:

1. Critical user journeys are covered by reliable end to end checks.
2. Most regression signal comes from fast API or component tests.
3. Any engineer can add a test without copying brittle setup.
4. Failures in CI are diagnosable within minutes.
5. Tests can run in parallel against multiple environments.
6. Secrets never live in the repository.

Constraints matter as much as goals:

• Language the team already knows
• App architecture (monolith, microservices, SPA)
• Release cadence
• Browser matrix requirements
• Compliance or audit needs

If you skip this step, you will invent abstractions for imaginary future teams and under invest in the pain you have today.

Choose the Right Framework Shape

Linear Scripts

Fine for spikes and throwaway bug repros. Not a framework.

Modular Framework

Shared libraries for login, navigation, and assertions, but limited formal layering. Good early stage choice.

Data Driven Framework

Same flow, many data rows. Excellent for validation matrices and calculation heavy features.

Keyword Driven Framework

Business readable keywords map to actions. Powerful for some enterprises, heavy for most product teams.

Hybrid Framework

Combines page objects or API clients, data factories, fixtures, and selective data driven tables. This is the shape most modern product teams should target.

Shape	Strength	Weakness	Best use
Modular	Fast to start	Can become messy	Small teams
Data driven	High input coverage	Weak for complex journeys	Forms, pricing, rules
Keyword driven	Non engineer authoring	High framework cost	Large manual heavy orgs
Hybrid	Balanced speed and scale	Needs discipline	Most product engineering teams

The Layers of a Test Automation Framework

A clean hybrid architecture usually has these layers.

1. Spec Layer

Contains scenarios and expectations. Specs should read like product behavior.

test("buyer can checkout with valid card", async ({ buyer, shop }) => {
  await shop.addItem("Trail Runner");
  await shop.checkoutWithSavedCard();
  await expect(shop.confirmation).toBeVisible();
});

2. Interaction Layer

UI page objects, component objects, and API clients live here. This layer knows how to talk to the system. See the full pattern guide on page object model.

3. Fixture and Lifecycle Layer

Creates authenticated contexts, browser state, temporary users, seeded records, and cleanup hooks.

4. Test Data Layer

Builders and factories produce realistic, unique data. Avoid a single shared spreadsheet user for parallel runs.

5. Configuration Layer

Environment URLs, feature flags, timeouts, browser projects, and secret injection.

6. Support Utilities

Date helpers, money formatting matchers, retry wrappers for known third party flakes, file download helpers.

When downloads are part of product value, design those helpers with automated file download testing in mind instead of leaving them as ad hoc assertions.

7. Reporting and Observability Layer

HTML reports, JUnit output, traces, screenshots, logs correlated by test name.

8. Execution Layer

Local scripts, CI workflows, sharding, and artifact upload. Wire this early with CI/CD for test automation with GitHub Actions.

Starter Architecture You Can Copy

automation/
  package.json
  playwright.config.ts
  .env.example
  src/
    config/
      env.ts
    pages/
      login.page.ts
      checkout.page.ts
    api/
      users.client.ts
      orders.client.ts
    fixtures/
      index.ts
      auth.fixture.ts
    data/
      builders/
        user.builder.ts
        product.builder.ts
    utils/
      money.ts
      wait.ts
  tests/
    smoke/
      login.smoke.spec.ts
    regression/
      checkout.spec.ts
    api/
      orders.api.spec.ts
  reports/
  test-results/

Keep production app code and automation code boundaries clear. Some monorepos co-locate tests beside features. That can work if ownership is explicit. For many QA owned suites, a dedicated automation package remains easier to version and deploy in CI.

Step by Step: Build the Framework

Step 1: Create a Thin Runnable Skeleton

Install the runner, add one passing smoke test, and run it in CI on day one. A framework that only runs on laptops is incomplete.

Proof goals for day one:

• npm test works locally
• One workflow runs on pull requests
• Failure artifacts are stored

Step 2: Centralize Configuration

// src/config/env.ts
export const env = {
  baseURL: process.env.BASE_URL ?? "http://localhost:3000",
  apiURL: process.env.API_URL ?? "http://localhost:3000/api",
  adminEmail: process.env.ADMIN_EMAIL ?? "",
  adminPassword: process.env.ADMIN_PASSWORD ?? "",
};

Rules:

• No production secrets in git
• .env.example documents required keys
• Config fails fast when mandatory values are missing in CI

Step 3: Add Fixtures Before Utilities Bloat

Fixtures beat ad hoc setup helpers because they compose cleanly and clean up predictably.

import { test as base } from "@playwright/test";
import { LoginPage } from "../pages/login.page";

type Fixtures = {
  loginPage: LoginPage;
  buyerStorageState: string;
};

export const test = base.extend<Fixtures>({
  loginPage: async ({ page }, use) => {
    await use(new LoginPage(page));
  },
});

Later you can add authenticated contexts, API clients, and feature flags without rewriting every spec.

Step 4: Introduce Interaction Objects

Start with the highest traffic screens and APIs:

• Login
• Navigation shell
• Checkout
• User creation API
• Order seeding API

Do not page object the entire app on week one. Extract when duplication appears twice, not when a blog post says every button needs a class.

Step 5: Build Data Factories

let counter = 0;

export function buildUser(overrides: Partial<User> = {}): User {
  counter += 1;
  return {
    email: `buyer.${Date.now()}.${counter}@example.test`,
    password: "ValidPass#2026",
    role: "buyer",
    ...overrides,
  };
}

Unique data is essential for parallel CI. Shared static users create cross test pollution and mysterious flakes.

Step 6: Add Hybrid Setup Paths

Use API setup for speed:

test("order history shows completed order", async ({ page, api, buyer }) => {
  const order = await api.orders.createCompletedOrder(buyer.id);
  await page.goto(`/orders/${order.id}`);
  await expect(page.getByText("Completed")).toBeVisible();
});

Use UI only when the UI itself is under test. This hybrid approach is the practical core of a scalable automation framework for CI/CD.

Step 7: Standardize Reporting and Logging

Minimum useful set:

• HTML report for humans
• JUnit or JSON for CI annotations
• Screenshots on failure
• Traces on failure for UI tests
• Request and response logging for API tests

Name artifacts by test title and timestamp so debugging does not require psychic powers.

Step 8: Codify Conventions in Docs and Lint Rules

Write a short CONTRIBUTING-TESTS.md:

• When to write UI vs API tests
• Locator strategy
• Data cleanup expectations
• How to quarantine flaky tests
• Folder placement rules

Conventions that live only in one senior engineer's head are not framework features.

Design Patterns That Earn Their Keep

Page Object Model

Best default for UI structure. Keep methods intent based. Full guide: page object model.

Facade for APIs

export class ShopApi {
  constructor(private readonly request: APIRequestContext) {}

  users = new UsersClient(this.request);
  orders = new OrdersClient(this.request);
}

Specs depend on shopApi.orders.create(), not on scattered raw request calls.

Builder / Factory for Data

Readable overrides beat giant optional parameter lists.

Strategy for Environment Behavior

Local may use seeded Docker services. Staging may use shared integrations. Production smoke may be read only. Encode those differences in strategy modules instead of if (env === "prod") soup inside tests.

Object Mother for Domain Setup

When creating a "buyer with two past orders and an expired card" becomes common, give that recipe a name. Named recipes reduce setup mistakes.

Framework Design With Reporting and Logging

Reporting is part of the product quality loop, not an afterthought.

What Good Failure Output Includes

• Test name and file
• Environment and browser or runtime
• Assertion message
• Screenshot or response body
• Trace or log correlation id
• Seeded entity ids when relevant

Logging Guidelines

• Log arrange steps at info level
• Log sensitive tokens only in redacted form
• Prefer structured logs over decorative print lines
• Attach logs to the test result, not only console history

Report Consumers

Different people need different views:

Consumer	Needs
Author of the failing PR	Exact assertion and reproduction path
QA lead	Trend of failures by suite and area
Engineering manager	Signal reliability and runtime cost
On call engineer	Whether smoke is red in production checks

If your report only says "expected true, received false," your framework is incomplete.

Making the Framework CI Native

Local green is not enough. Design for pipeline realities.

Parallel Safety

• Unique test data
• No shared writable accounts
• Stateless tests where possible
• Explicit cleanup or disposable tenants

Runtime Budgets

Track:

• Total suite duration
• P95 test duration
• Failure rate
• Flake rate

A framework that cannot answer "are we getting slower?" will eventually become shelfware.

Sharding and Selection

Support commands like:

npm run test:smoke
npm run test:api
npm run test:ui -- --shard=1/4

Pull requests may run smoke plus affected packs. Nightly runs may run full regression. Fail fast and full suite strategies both have a place. Details live in the GitHub Actions guide linked above.

Sample playwright.config.ts Backbone

import { defineConfig, devices } from "@playwright/test";
import { env } from "./src/config/env";

export default defineConfig({
  testDir: "./tests",
  timeout: 60_000,
  expect: { timeout: 10_000 },
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 4 : undefined,
  reporter: [
    ["list"],
    ["html", { open: "never" }],
    ["junit", { outputFile: "reports/junit.xml" }],
  ],
  use: {
    baseURL: env.baseURL,
    trace: "on-first-retry",
    screenshot: "only-on-failure",
    video: "retain-on-failure",
  },
  projects: [
    { name: "chromium", use: { ...devices["Desktop Chrome"] } },
  ],
});

This is not the whole framework, but it encodes important policy: parallel by default, limited CI retries, and artifacts on failure.

Hybrid Test Automation Framework Architecture

A realistic hybrid model:

                  +-------------------+
                  |   CI Workflows    |
                  +---------+---------+
                            |
                            v
                  +-------------------+
                  |  Test Selection   |
                  | smoke/api/ui/full |
                  +---------+---------+
                            |
            +---------------+---------------+
            |                               |
            v                               v
   +----------------+               +----------------+
   |  API Specs     |               |  UI Specs      |
   +--------+-------+               +--------+-------+
            |                                |
            v                                v
   +----------------+               +----------------+
   | API Facades    |               | Page Objects   |
   +--------+-------+               +--------+-------+
            |                                |
            +---------------+----------------+
                            |
                            v
                  +-------------------+
                  | Fixtures / Data   |
                  | Factories / Env   |
                  +-------------------+

API and UI share config, data, auth, and reporting. They do not share bad habits. UI tests should not become API tests with a browser open for decoration.

Ownership, Quality Gates, and Team Process

Technology fails without ownership.

Define:

• Who approves framework changes
• Who triages nightly failures
• How flaky tests are quarantined
• What "done" means for a new automated check

A useful definition of done for an automated test:

1. Protects a real risk
2. Has stable data setup
3. Passes three consecutive CI runs
4. Produces clear artifacts on failure
5. Is documented enough for another engineer to update

Common Mistakes When You Build a Test Automation Framework

Mistake 1: Abstract Base Class Theatre

Deep inheritance trees look enterprise grade and age poorly. Prefer composition with fixtures and small helpers.

Mistake 2: Automating Everything Through the UI

End to end coverage is expensive. Push validation left to API, component, and unit layers when those layers can catch the risk faster.

Mistake 3: No Data Strategy

Hardcoded users and mutable shared state are the top cause of "framework works on my machine" syndrome.

Mistake 4: Silent Failures Without Artifacts

If CI only prints a stack trace, people stop trusting and eventually stop running the suite.

Mistake 5: Infinite Utility Folders

utils/helpers/common/misc.ts becomes a junk drawer. Name modules by purpose and delete dead code aggressively.

Mistake 6: Treating Retries as a Fix

Retries can reduce noise while you investigate, but they are not architecture. Root cause work belongs in how to fix flaky tests.

Mistake 7: Building for Imaginary Scale

You do not need multi region device farms and a custom DSL in month one. You need trustworthy smoke tests and a path to grow.

Implementation Checklist

Use this checklist while building from scratch:

• Goals and constraints written down
• Runner chosen with team skill fit
• One smoke test green locally and in CI
• Environment config centralized
• Secrets excluded from git
• Folder conventions documented
• Fixtures for auth and common setup
• Page objects or API clients for top flows
• Unique data factories
• HTML plus machine readable reports
• Failure artifacts uploaded in CI
• Smoke, regression, and API commands defined
• Parallel safety reviewed
• Flake triage process defined
• Contribution guide for new tests

30 / 60 / 90 Day Build Plan

Days 1-30

• Skeleton project
• CI smoke
• Login and one critical journey
• Basic page objects and env config
• Failure screenshots

Days 31-60

• API setup helpers
• Data builders
• Expanded regression pack
• Report publishing
• Initial parallelization

Days 61-90

• Test selection strategy for PRs vs nightly
• Flake dashboard or simple flake log
• Component or API depth where UI is overkill
• Onboarding docs and sample tests
• Runtime budget tracking

This plan produces value continuously. It avoids the six month framework program that ships no product protection.

How to Evaluate Whether Your Framework Is Working

Ask monthly:

1. Do developers trust red builds?
2. How long does a PR suite take?
3. What percentage of failures are real defects vs flakes or env issues?
4. How long does a new engineer need to add a test?
5. Are high risk product areas covered?

If trust is low, improve stability and diagnostics before adding more tests. Volume without trust is liability.

Practice: Build a Mini Framework on a Real App

Pick a small application, even a QABattle arena app under test, and practice the whole loop:

1. Create a repo skeleton.
2. Add two smoke tests.
3. Extract one page object.
4. Add an API or fixture based setup.
5. Run in a pipeline.
6. Break a selector on purpose and confirm the failure is obvious.

Then open QABattle tracks and use automation challenges to pressure test your structure against messy UI reality. Frameworks that only work on perfect demo apps are not ready.

Naming Conventions That Reduce Friction

Frameworks fail socially when nobody can find anything. Agree on names early.

Spec Names

checkout.guest.happy.spec.ts
checkout.coupon.invalid.spec.ts
orders.list.pagination.api.spec.ts

Include product area, scenario intent, and layer. Avoid test1.spec.ts and final_final_v3.spec.ts.

Page and Client Names

LoginPage
CheckoutPage
OrdersApiClient
BillingComponent

Match product language. If the business says "workspace," do not invent "tenant console page" unless that is the real domain term.

Tag Names

@smoke
@regression
@api
@ui
@checkout
@flaky

Tags power CI selection. Keep the vocabulary short enough that people remember it.

Security and Secrets in Framework Design

Automation often needs privileged credentials. Design for least privilege.

• Separate admin setup credentials from buyer journey credentials
• Rotate secrets used in CI
• Redact auth headers in logs and HTML reports
• Avoid printing full response bodies when they contain PII
• Provide a fake payment mode for non production environments

A framework that leaks tokens in artifacts is not "done," no matter how elegant the folder tree looks.

Example: Adding a New Feature Without Chaos

Suppose the product adds "save for later" on cart items. Framework workflow:

1. Identify risk: item state transitions, inventory, logged in vs guest.
2. Decide layers: API tests for state transitions, one UI smoke for the drawer.
3. Add data builder support if new entities appear.
4. Extend CartPage with saveForLater(itemName) and moveToCart(itemName).
5. Add API client methods for setup if UI arrange is slow.
6. Tag the critical path @smoke and deeper cases @regression.
7. Confirm artifacts on intentional failure.
8. Document the new methods in the contribution guide if the pattern is new.

That is framework use in the wild. The architecture exists so this process stays boring and repeatable.

Final Takeaway

To build a test automation framework from scratch, optimize for team speed and signal quality, not for pattern count. Define goals, create a thin runnable core, layer specs above interaction objects, invest in fixtures and unique data, and make CI reporting excellent. Grow hybrid API plus UI capabilities as soon as setup time becomes the bottleneck.

A great framework feels boring in the best way: new tests are easy to add, failures are easy to read, and product changes do not require archaeological refactors. Start small, keep the layers honest, and let real suite pain decide the next abstraction.