Flaky Tests: Causes and How to Fix Them

A suite that is green, red, then green again with no product change trains teams to ignore failures. How to fix flaky tests is less about stacking retries and more about finding the real source of non-determinism: races, shared state, time, networks, and order-dependent data.

This guide covers the main causes of flaky automation, a practical debugging workflow, Playwright friendly fixes, quarantine strategy, CI policy, and the mistakes that keep unstable suites unstable. Use it as a playbook for root cause analysis on unstable UI tests and for building a culture that treats flake debt as real debt.

What Is a Flaky Test?

A flaky test is a test that both passes and fails under the same product revision and supposedly the same conditions. The failure is intermittent. Rerunning the job may "fix" it without any code change.

Important distinctions:

Category	Meaning	Response
Deterministic failure	Fails consistently	Fix product or fix incorrect test
Flaky failure	Passes and fails intermittently	Remove non determinism
Environment failure	Infra or dependency outage	Fix env, isolate dependency, or mark accordingly
Product race	Real intermittent product bug	File defect, do not silence

Not every intermittent failure is a bad test. Sometimes the product has a race condition and the test is the first honest witness. Your job is to separate signal from noise, then act.

Why Flaky Tests Are So Expensive

A single flake looks minor. At suite scale it becomes cultural damage.

Costs include:

• Engineers rerun pipelines instead of investigating
• Real regressions hide in noisy histories
• Release trains stall on false alarms
• New automation is distrusted before it is even reviewed
• Teams add sleeps and retries until runtime explodes

A flaky smoke suite is worse than a small reliable smoke suite. Trust is the product.

Common Causes of Flaky Tests

1. Race Conditions and Timing Assumptions

The test clicks before the page is ready, asserts before a table finishes rendering, or navigates away while a request is in flight.

Symptoms:

• Fails more on slower CI runners
• Passes locally on a warm machine
• Fails when parallel load increases

2. Hard Coded Sleeps

waitForTimeout(3000) is a coin flip. Sometimes three seconds is enough. Sometimes it is not. Sometimes it is far too much and the suite crawls.

3. Unstable Locators

Selectors tied to auto generated classes, absolute XPath, or visual order break when the design system shifts. See CSS selectors vs XPath and prefer roles, labels, and test ids inside a solid page object model.

4. Shared Mutable Test Data

All tests log in as the same user, edit the same record, or depend on a single coupon code. Parallelism turns that into random collisions.

5. Order Dependent Tests

Test B passes only if test A created state and never cleaned it up. Sharding and reordering expose the landmine.

6. Third Party Dependencies

Maps widgets, payment sandboxes, chat beacons, and analytics scripts introduce latency and outages you do not control.

7. Animation and Transition Timing

Buttons may be visible but not stable. Modals may still be animating. Click interception errors are a classic symptom.

8. Clock and Time Zone Issues

Tests that depend on "today," local midnight, or relative timestamps fail around day boundaries or in different CI regions.

9. Resource Contention

CPU starved CI agents, limited browser workers, or database locks create timeouts that look like product bugs.

10. Over specified Assertions

Asserting exact full page HTML, precise pixel layouts, or complete unsorted JSON blobs creates noise when irrelevant fields change.

A Practical Workflow for How to Fix Flaky Tests

Use the same disciplined loop every time. Random tweaks create new flakes.

Step 1: Confirm It Is Actually Flaky

Run the test repeatedly on the same commit:

npx playwright test tests/checkout.spec.ts --repeat-each=20

If it fails inconsistently, continue. If it fails every time, treat it as a deterministic bug.

Step 2: Gather Artifacts Before Changing Code

Collect:

• Trace files
• Screenshots
• Videos
• Network logs
• Server logs around the timestamp
• Seeded entity ids
• Browser console errors

Without artifacts, people guess. Guessing is how sleep(5000) becomes architecture.

For UI failures, standardize evidence capture with screenshots in Playwright so every intermittent failure has visual context.

Step 3: Classify the Failure Mode

Ask:

1. Did the element never appear?
2. Did it appear late?
3. Did the wrong record load?
4. Did another test interfere?
5. Did a dependency 500?
6. Did the assertion expect the wrong thing?

Classification prevents "fix everything" pull requests.

Step 4: Reproduce Under Stress

Try:

• Slow network throttling
• Lower CI class machines
• Higher parallelism
• Fresh browser context each run
• Different order with shard settings

Flakes that only appear under load often point to shared data or race conditions.

Step 5: Fix the Root Cause

Change the design of the test or the product contract. Do not only widen timeouts unless the environment truly needs a policy change.

Step 6: Prove the Fix

Re-run enough times to be confident. One green local run after a sleep increase is not proof.

Step 7: Record the Learning

Add a short note to the PR or team log: cause, fix, and how to avoid repeats. Flake patterns are usually family groups, not unique snowflakes.

Root Cause Analysis for Unstable UI Tests

When a UI test is intermittent, inspect these layers in order.

Layer A: Waiting Strategy

Prefer condition based waits:

await expect(page.getByRole("heading", { name: "Orders" })).toBeVisible();
await page.getByRole("button", { name: "Create order" }).click();

Avoid:

await page.waitForTimeout(5000);
await page.click("#create");

Playwright auto waiting helps, but you still must assert the state that makes the next action valid. Waiting for a spinner to disappear, a network response, or a heading can be the missing condition.

Layer B: Navigation and Network

await Promise.all([
  page.waitForURL(/\/orders\/\d+/),
  page.getByRole("button", { name: "Save" }).click(),
]);

Or wait for a specific API response when the UI depends on it:

const responsePromise = page.waitForResponse(
  (res) => res.url().includes("/api/orders") && res.status() === 200,
);
await page.getByRole("button", { name: "Refresh" }).click();
await responsePromise;

Layer C: Data Isolation

const user = await api.users.createUniqueBuyer();
await loginAs(page, user);

Never rely on "whatever is currently in staging."

Layer D: Locator Quality

// brittle
page.locator("div.sc-aBcDeF > button:nth-child(2)");

// stronger
page.getByTestId("create-order");
page.getByRole("button", { name: "Create order" });

Layer E: Environment Hygiene

Confirm:

• Feature flags match expectations
• Migrations finished
• Seed jobs completed
• Required services are healthy
• Rate limits are not throttling CI

How to Reduce Flaky Playwright Tests

Playwright gives you strong defaults. Teams still create flakes by fighting those defaults.

Do More of This

• Use getByRole, getByLabel, getByTestId
• Assert with expect auto retries
• Keep tests independent
• Seed via API
• Capture trace on first retry
• Fail on test.only in CI

Do Less of This

• Manual sleeps
• CSS chains based on layout
• Shared storage state mutated by every test
• Hidden try/catch that swallows errors
• Giant end to end setups for unit sized risks

Example: Stabilizing a Table Refresh

Flaky version:

await page.click("text=Refresh");
await page.waitForTimeout(2000);
await expect(page.locator("table tr")).toHaveCount(3);

Stable version:

const rows = page.getByTestId("orders-table").getByRole("row");
await Promise.all([
  page.waitForResponse((res) => res.url().includes("/api/orders") && res.ok()),
  page.getByRole("button", { name: "Refresh" }).click(),
]);
await expect(rows).toHaveCount(3);

The stable version ties progress to a real condition.

Should You Retry Flaky Tests in CI?

Yes, carefully. No, not forever.

A Reasonable Policy

Suite	Retries	Why
PR smoke	0 or 1	Fast feedback, high trust needed
Full UI nightly	1	Some infra noise possible
Known quarantine lane	0	Do not fake stability

Retries help when:

• Infrastructure blips exist
• You need temporary signal while fixing a hard flake
• The cost of a false red is very high

Retries hurt when:

• They hide systemic data races
• They make every job slower
• They become the only "fix" process

Track retry success as a metric. If a test passes only on retry often, it is still flaky.

Quarantine Flaky Tests in the Pipeline

Quarantine is a visible, temporary holding area for unstable tests that should not block the team while ownership is assigned.

Good Quarantine Rules

1. A named owner is required.
2. A ticket or issue link is required.
3. A max age exists, such as 14 days.
4. Quarantined tests still run in a non blocking job.
5. Results are reviewed, not ignored.

Example Tagging Approach

test("checkout with coupon @flaky", async ({ page }) => {
  // ...
});

CI can run @flaky in a separate workflow that reports without failing the merge gate. The main gate stays strict.

Bad Quarantine

• Skipping silently with no owner
• Leaving tests skipped for months
• Quarantining whole folders because triage is hard

Quarantine is a scalpel. If you use it as a broom, your suite becomes fiction.

Debugging Checklist for Intermittent Failures

When a test fails intermittently, walk this list:

1. Does it fail on a clean retry of the same commit?
2. Are traces and screenshots available?
3. Is the locator unique and stable?
4. Is the wait condition explicit and meaningful?
5. Does the test create unique data?
6. Does it depend on another test's side effects?
7. Are third party calls involved?
8. Does it only fail under parallel execution?
9. Are timeouts close to the real needed duration, not arbitrary?
10. Is the assertion checking a user observable outcome?

If you answer these thoroughly, most flakes reveal themselves.

Product Bugs That Look Like Test Flakes

Do not automatically blame the suite.

Examples of real product races:

• Button enabled before server accepts submits, causing double orders
• Search results arriving out of order without request sequencing
• Websocket counters updating late without loading states
• Cache invalidation lag after writes

When you find these, file them as product defects with evidence. A test that catches an intermittent product bug is valuable. Silencing it to keep CI green is a business risk.

Organizational Practices That Keep Suites Healthy

Own the Nightly Build

Someone reviews failures every working day. Unowned nightlies become archaeology.

Budget Flake Rate

Publish a simple number weekly:

• Smoke flake rate
• Regression flake rate
• Top 5 offenders

What gets measured gets fixed.

Block New Patterns That Cause Flakes

Code review should reject:

• New hard sleeps without justification
• Shared credentials in new tests
• Assertions on entire document bodies
• UI setup that could be API setup

Separate Fast and Slow Lanes

PR suites should be small and strict. Nightly suites can be broader. This architecture is covered more in CI/CD for test automation with GitHub Actions and framework layering in build a test automation framework.

Common Mistakes When Handling Flaky Tests

Mistake 1: Increasing Timeouts Forever

A longer timeout can reduce failure frequency without removing the race. The suite becomes slow and still unstable under worse conditions.

Mistake 2: Deleting the Test

Deleting removes pain and coverage at the same time. Prefer quarantine with ownership, then repair or intentionally retire with a documented risk acceptance.

Mistake 3: Fixing the Symptom Only on One Browser

If Chromium is flaky and Firefox is ignored, you will rediscover the issue later. Validate the fix across the matrix you claim to support.

Mistake 4: Global State Cleanup in After Hooks That Can Fail Silently

Cleanup must be reliable. Failed teardown is a future flake factory.

Mistake 5: Using Production Accounts

Rate limits, MFA prompts, and real data drift will punish you.

Mistake 6: No Reproduction Notes

"Fixed flake" with no cause leaves the next engineer blind when the sibling test fails next week.

Mistake 7: Treating All Retries as Equivalent

A retry after a network blip is different from a retry after a wrong selector. Metrics should distinguish infra noise from test design failure when possible.

Worked Example: Flaky Login Redirect

Symptoms

Test fails about one in eight runs with:

expect(page).toHaveURL(/dashboard/)
// still on /login

Investigation

Trace shows click on Sign in occurred while a feature flag request was pending. Sometimes the app accepted login, sometimes an error toast flashed too quickly for the assertion path.

Bad Fix

await page.waitForTimeout(5000);

Better Fix

await loginPage.loginAs(user.email, user.password);
await expect(page.getByRole("heading", { name: "Dashboard" })).toBeVisible();
await expect(page).toHaveURL(/\/dashboard/);

And if the app can legitimately recover slowly:

await expect
  .poll(async () => page.url(), { timeout: 15_000 })
  .toMatch(/\/dashboard/);

Better still: ensure the app disables the submit button until ready and returns a stable authenticated landing state. Test and product quality rise together.

Metrics and Dashboards Worth Tracking

You do not need fancy tooling on day one. A spreadsheet works initially.

Track:

• Failures by test name over 14 days
• Pass on retry counts
• Median and P95 duration
• Environment related failures
• Age of quarantined tests

Promote the worst offenders into weekly triage. Fixing the top ten often improves perceived suite health more than writing twenty new tests.

When the Framework Design Is the Flake Source

Some suites are flaky because architecture is wrong:

• Everything is UI level
• No fixtures
• No test data service
• Global singleton browser state
• Hidden dependency on run order

If that is your situation, local patches will not hold. Step up a level and repair structure with a proper hybrid approach from the framework guide linked earlier. Pattern level fixes beat endless tactical sleeps.

Practice Loop

Take one flaky test in your suite, or intentionally create a race in a practice app:

1. Capture a failure artifact.
2. Classify the cause.
3. Remove sleeps.
4. Isolate data.
5. Re-run twenty times.
6. Document the cause in one paragraph.

Then practice failure analysis on live challenges in QABattle battles. The skill of reading intermittent failures is as important as writing the first green test.

Flaky Network and Third Party Patterns

Third party widgets are frequent flake factories: payments, maps, chat, feature flag consoles, analytics.

Mitigations:

1. Stub or mock non critical third parties in test environments.
2. Use vendor sandbox accounts dedicated to CI.
3. Wait for specific callbacks or network responses rather than fixed sleeps.
4. Split tests so optional widgets are not on the critical smoke path.
5. Record known vendor outages separately from product failures.

Example of isolating analytics noise:

await page.route("**/analytics/**", (route) => route.fulfill({ status: 204, body: "" }));

Only stub what the test does not intend to prove. If the test is about payment confirmation, do not stub the payment confirmation endpoint.

Parallel Execution and Flake Amplification

A suite can look stable in serial local runs and explode under CI sharding.

Typical parallel-only failures:

• Two tests edit the same user profile
• Coupon code has a single use global limit
• Search index lag differs under load
• Database unique constraints collide on fixed emails

Fixes:

• Unique data per test
• Separate tenants or workspaces per worker when the product supports it
• Prefer create and delete over reuse
• Avoid depending on global ranking positions in lists

If you need to keep a shared user, make the test read only. Shared writable fixtures and parallel CI do not mix.

Communication Templates for Flake Work

Good triage notes save days.

Bug ticket for product race

Title: Intermittent double submit creates two orders
Evidence: Playwright trace, two POST /orders within 120ms, UI showed one spinner
Impact: Duplicate charges risk
Repro rate: 4/20 under slow 3G throttle

Ticket for test design flake

Title: checkout.spec depends on shared staging coupon SAVE20
Evidence: fails when another suite redeems the code first
Fix plan: create unique coupon via admin API in test setup
Owner: QA platform

Clear classification prevents the wrong team from being paged.

Assertion Design That Avoids False Reds

Some flakes are assertion quality problems, not wait problems.

Fragile:

await expect(page.locator("body")).toHaveText(/Welcome Jane Doe to the dashboard of Acme Corp/);

Stronger:

await expect(page.getByRole("heading", { name: "Dashboard" })).toBeVisible();
await expect(page.getByTestId("user-menu")).toContainText("Jane");

Assert the outcomes the product guarantees. Avoid asserting incidental copy, ads, or timestamps unless those fields are the requirement.

Also prefer auto retrying assertions over reading text into a variable and comparing once. One shot reads lose the race to late rendering.

A 14 Day Flake Burn Down Plan

Day 1-2: Measure top offenders and flake rate.

Day 3-5: Fix or quarantine the top five blockers for PR smoke.

Day 6-8: Remove hard sleeps from the smoke pack.

Day 9-11: Break shared account dependencies in the next ten offenders.

Day 12-13: Improve artifacts where diagnosis is still slow.

Day 14: Publish before and after metrics to the team.

You will not reach zero forever. You will restore trust, which is the real goal.

Smoke Suites Deserve Higher Stability Standards

Not every suite needs the same flake tolerance. Smoke packs that gate merges should be boringly reliable. A flake in an experimental visual pack is annoying. A flake in smoke blocks the company and trains people to click re-run.

Protect smoke by:

• Keeping it small and high signal
• Banning shared writable accounts
• Preferring API arrange plus short UI assert paths
• Reviewing every smoke failure the same day
• Refusing to add known unstable coverage to the gate

If a check is valuable but not yet stable, keep it outside the required gate until it earns trust.

Final Takeaway

Knowing how to fix flaky tests is a core automation skill. Confirm flakiness, gather artifacts, classify the failure, remove non determinism at the root, and prove the repair under stress. Use retries only as a temporary cushion. Quarantine with owners and deadlines. Measure flake rate like you measure build time.

Stable automation is not luck. It is intentional waiting, isolated data, resilient locators, honest CI policy, and a team that refuses to normalize the rerun button as a quality process.