Stop HAR refreshes from approving backend drift

Yesterday's HAR replay failed after the catalog API added a field. Today it passes, but only because the test ran with update: true and replaced its own fixture. No reviewer decided whether the new response was correct.

That is the dangerous side of convenient recording. A HAR is executable test data, not a disposable log. When a test refreshes its expected network responses during the same run that judges the application, upstream drift can approve itself.

Understand what update mode does to the test boundary

browserContext.routeFromHAR() and page.routeFromHAR() have two distinct operating modes. In replay mode, matching browser requests are served from an existing HAR. In update mode, Playwright sends requests to the real network and records the actual traffic into the given path instead of serving responses from that file. The updated file is written when the browser context closes.

That last detail explains several confusing results. Code can finish the page journey, read the HAR immediately, and conclude that Playwright ignored the update. The bytes are still old because the context remains open. The opposite failure is also possible: an assertion fails, fixture teardown closes the context, and Playwright writes the captured traffic anyway. The failed refresh has changed a file even though the journey never earned promotion.

Teams often use the phrase Playwright HAR update modes for three separate choices:

1. update: true chooses live capture instead of replay.
2. updateMode: 'minimal' or 'full' controls how much HAR metadata is retained during that capture.
3. updateContent: 'embed' or 'attach' controls where resource bodies are stored.

Those options solve different problems. Minimal mode keeps what Playwright needs to route later. Official API documentation says it omits sizes, timing, page, cookies, security, and other information not used for replay. Full mode is closer to a general HTTP archive. It is larger, changes more often, and can retain more sensitive or environment-specific detail. A replay fixture normally benefits from minimal mode. A network investigation that needs timing or security information may need full mode, but that diagnostic archive should not automatically become the fixture used by UI tests.

Embedded content stays inside the HAR. JSON and other text bodies are straightforward to review because the request entry and response payload change together. Attached content is written as separate resources, or as separate entries when the path ends in .zip. That avoids enormous inline binary strings, but a reviewer now has to follow companion files. Deleting or failing to publish one attachment can leave a HAR entry whose body cannot be replayed.

The url option defines the capture and replay scope. Without it, a journey can record documents, identity traffic, analytics, fonts, feature-flag requests, and unrelated APIs. A wide recording increases fixture size and raises the chance of storing cookies, tokens, personal information, or internal hostnames. A filter that is too narrow is not safer. Update mode may let the rest of the page use the live network, so the journey passes while the candidate lacks a required request.

HAR matching is strict where it matters. Playwright matches the URL and HTTP method, and POST requests also use their payload for matching. If several entries match, header similarity helps choose one. Random request IDs, timestamps, tenant hosts, GraphQL variables, and idempotency keys can therefore turn a newly recorded fixture into a future replay miss. Refreshing more often does not make unstable inputs deterministic.

There is no assertion inside update: true that says the captured response was appropriate. A 401 login response, a 503 maintenance page, an empty catalog, or a response from the wrong tenant can all be real network information. The recorder did its job. Governance has to decide whether those bytes are acceptable test data.

Build a refresh path that cannot run by accident

Separate capture from replay at the file level. The routine test should contain no environment branch that can enable updates. A dedicated refresh test should require an explicit guard, use one worker, write to a candidate path, and close its own context before inspecting the result. The authoritative fixture remains untouched until validation and human review finish.

The following local application makes the workflow runnable without an external API. It serves a small page and a catalog endpoint on a stable port. The CATALOG_NAME environment variable lets us change the live response between capture and replay, while apiHits proves whether the browser reached the live API.

Save it as tests/helpers/catalog-app.ts:

import { once } from 'node:events';
import { createServer, type ServerResponse } from 'node:http';

const port = 41739;

function send(
  response: ServerResponse,
  contentType: string,
  body: string,
): void {
  response.statusCode = 200;
  response.setHeader('content-type', contentType);
  response.end(body);
}

export async function startCatalogApp() {
  let apiHits = 0;

  const server = createServer((request, response) => {
    const url = new URL(
      request.url ?? '/',
      `http://${request.headers.host}`,
    );

    if (url.pathname === '/') {
      send(
        response,
        'text/html; charset=utf-8',
        `<!doctype html>
        <html>
          <body>
            <h1>Catalog</h1>
            <ul aria-label="Products"></ul>
            <script>
              fetch('/api/catalog?region=test')
                .then(response => response.json())
                .then(products => {
                  const list = document.querySelector('ul');
                  for (const product of products) {
                    const item = document.createElement('li');
                    item.textContent = product.name;
                    list.appendChild(item);
                  }
                });
            </script>
          </body>
        </html>`,
      );
      return;
    }

    if (url.pathname === '/api/catalog') {
      apiHits += 1;
      const name = process.env.CATALOG_NAME ?? 'Field Recorder';
      send(
        response,
        'application/json',
        JSON.stringify([{ id: 7, name, region: url.searchParams.get('region') }]),
      );
      return;
    }

    response.statusCode = 404;
    response.end('Not found');
  });

  server.listen(port, '127.0.0.1');
  await once(server, 'listening');

  return {
    origin: `http://127.0.0.1:${port}`,
    get apiHits() {
      return apiHits;
    },
    close: async () => {
      server.close();
      await once(server, 'close');
    },
  };
}

The refresh test writes catalog.candidate.har, not catalog.har. It creates a context from the browser fixture because the test must control exactly when that context closes. Save it as tests/tools/refresh-catalog-har.spec.ts:

import { existsSync, mkdirSync, rmSync } from 'node:fs';
import path from 'node:path';
import { expect, test } from '@playwright/test';
import { startCatalogApp } from '../helpers/catalog-app';

const harDirectory = path.resolve(process.cwd(), 'tests/hars');
const candidateHar = path.join(harDirectory, 'catalog.candidate.har');

test.describe.configure({ mode: 'serial', retries: 0 });

test('records a catalog HAR candidate', async ({ browser }) => {
  test.skip(
    process.env.REFRESH_HAR !== 'catalog',
    'Set REFRESH_HAR=catalog to run the controlled refresh',
  );

  mkdirSync(harDirectory, { recursive: true });
  rmSync(candidateHar, { force: true });

  const app = await startCatalogApp();
  const context = await browser.newContext({ serviceWorkers: 'block' });

  try {
    await context.routeFromHAR(candidateHar, {
      url: `${app.origin}/api/**`,
      update: true,
      updateMode: 'minimal',
      updateContent: 'embed',
    });

    const page = await context.newPage();
    await page.goto(app.origin);
    await expect(page.getByRole('listitem')).toHaveText([
      'Field Recorder',
    ]);
    expect(app.apiHits).toBe(1);
  } finally {
    await context.close();
    await app.close();
  }

  expect(existsSync(candidateHar)).toBe(true);
});

Run the capture alone:

REFRESH_HAR=catalog CATALOG_NAME='Field Recorder' npx playwright test tests/tools/refresh-catalog-har.spec.ts --workers=1

The guard prevents a broad npx playwright test from updating anything. The dedicated filename prevents the refresh from approving itself. Serial mode, zero retries, and one CLI worker reduce the chance of multiple contexts writing the same candidate. The fixed source value creates a reviewable payload. In a real environment, use a dedicated account and a known data seed rather than today's shared staging state.

The finally block closes the context even when the UI assertion fails, so a failed attempt may still leave a candidate. That is intentional. The candidate is evidence, not an approved fixture. Validation must reject it, and the canonical HAR has not changed.

Prove the candidate before promoting it

A useful HAR validator checks the boundary the replay depends on. At minimum, require the expected origin, method, path, successful status, response shape, and absence of forbidden headers. Counting entries catches the common case where a URL glob matched nothing. Parsing the response catches a 200 HTML login page that would otherwise look successful by status alone.

This standalone Node script validates the embedded candidate produced above. Save it as scripts/validate-catalog-har.mjs:

import { readFile } from 'node:fs/promises';

const [harPath] = process.argv.slice(2);
if (!harPath) {
  throw new Error('Usage: node scripts/validate-catalog-har.mjs <har-path>');
}

const har = JSON.parse(await readFile(harPath, 'utf8'));
const entries = har?.log?.entries;
if (!Array.isArray(entries)) {
  throw new Error('HAR does not contain log.entries');
}

const errors = [];
const forbiddenHeaders = new Set([
  'authorization',
  'cookie',
  'set-cookie',
  'x-api-key',
]);

if (entries.length !== 1) {
  errors.push(`expected 1 captured entry, found ${entries.length}`);
}

for (const [index, entry] of entries.entries()) {
  const requestUrl = new URL(entry.request.url);

  if (requestUrl.origin !== 'http://127.0.0.1:41739') {
    errors.push(`entry ${index}: unexpected origin ${requestUrl.origin}`);
  }
  if (entry.request.method !== 'GET') {
    errors.push(`entry ${index}: unexpected method ${entry.request.method}`);
  }
  if (
    requestUrl.pathname !== '/api/catalog' ||
    requestUrl.searchParams.get('region') !== 'test'
  ) {
    errors.push(`entry ${index}: unexpected catalog URL ${requestUrl.href}`);
  }
  if (entry.response.status !== 200) {
    errors.push(`entry ${index}: response status ${entry.response.status}`);
  }

  const headers = [
    ...(entry.request.headers ?? []),
    ...(entry.response.headers ?? []),
  ];
  for (const header of headers) {
    if (forbiddenHeaders.has(String(header.name).toLowerCase())) {
      errors.push(`entry ${index}: forbidden header ${header.name}`);
    }
  }

  const content = entry.response.content ?? {};
  let text = content.text;
  if (text && content.encoding === 'base64') {
    text = Buffer.from(text, 'base64').toString('utf8');
  }

  try {
    const products = JSON.parse(text ?? '');
    if (
      !Array.isArray(products) ||
      products.length !== 1 ||
      products[0].id !== 7 ||
      products[0].name !== 'Field Recorder' ||
      products[0].region !== 'test'
    ) {
      errors.push(`entry ${index}: catalog response has the wrong shape`);
    }
  } catch {
    errors.push(`entry ${index}: catalog response is not valid JSON`);
  }
}

if (errors.length > 0) {
  console.error('HAR validation failed:');
  for (const error of errors) console.error(`- ${error}`);
  process.exitCode = 1;
} else {
  console.log(`HAR validation passed: ${entries.length} entry`);
}

Run the validator before comparing the candidate with the canonical fixture:

node scripts/validate-catalog-har.mjs tests/hars/catalog.candidate.har
diff -u tests/hars/catalog.har tests/hars/catalog.candidate.har

The validator encodes the test's actual contract. A generic JSON schema check would not catch an unexpected host or a leaked cookie. A secret scanner alone would not catch a valid-looking 503. Add organization-specific checks for access tokens in query strings, personal data in bodies, internal domains, and prohibited response fields. Prefer an allowlist for the small fixture you intend to keep.

Review the semantic changes, not only the fact that JSON parses. A new optional response field may be harmless. A renamed field can indicate a UI and API contract change. A different product ID may mean the data seed drifted. A new request can represent a legitimate feature or an analytics call that does not belong in this HAR. Promotion is the moment to decide which of those claims the test should freeze.

For a new workflow, capture twice in sequence from the same seeded environment and write two different candidate paths. Compare them before reviewing either against the canonical HAR. Any difference between the two candidates is nondeterminism in the capture inputs, source data, or response metadata. Find that cause before promotion. Otherwise reviewers will repeatedly approve random timestamps, identifiers, ordering, or feature allocation as if each were a contract change. This check doubles live traffic and setup time, so do not use it against billable or non-idempotent journeys without a disposable account and owned cleanup. Once the source is stable, a periodic two-capture audit is enough; every ordinary refresh does not need to pay that cost.

After acceptance, replace the canonical fixture with the candidate in an ordinary reviewed change. Then delete the candidate or keep it outside the routine test paths. Do not make the refresh test perform that promotion. The extra manual step costs time, but it preserves separation between producing evidence and approving expected behavior.

The final proof is replay while the live API would return something else. Save this routine test as tests/catalog.har.spec.ts:

import path from 'node:path';
import { expect, test } from '@playwright/test';
import { startCatalogApp } from './helpers/catalog-app';

const catalogHar = path.resolve(
  process.cwd(),
  'tests/hars/catalog.har',
);

test('replays the reviewed catalog without calling the API', async ({
  browser,
}) => {
  const app = await startCatalogApp();
  const context = await browser.newContext({ serviceWorkers: 'block' });

  try {
    await context.routeFromHAR(catalogHar, {
      url: `${app.origin}/api/**`,
      notFound: 'abort',
    });

    const page = await context.newPage();
    await page.goto(app.origin);
    await expect(page.getByRole('listitem')).toHaveText([
      'Field Recorder',
    ]);
    expect(app.apiHits).toBe(0);
  } finally {
    await context.close();
    await app.close();
  }
});

Run it with a deliberately different live value:

CATALOG_NAME='Changed upstream' npx playwright test tests/catalog.har.spec.ts --workers=1 --trace on

The page should still render Field Recorder, and the server counter must stay at zero. The UI assertion alone is insufficient because the live API could coincidentally return the same text. The zero-hit assertion proves this local example used the reviewed recording.

Diagnose a bad refresh from the first conflicting evidence

When update mode misbehaves, identify whether the failure happened during selection, live capture, file writing, content storage, or later replay. Those stages produce different evidence.

The candidate does not exist. Confirm the code reached routeFromHAR() and that the manually created context closed. The official API contract writes on context closure, not after the last request. A test timeout, forced process termination, or forgotten await context.close() can prevent the write. If the Test Runner owns the default context, checking the file inside the test is too early because fixture teardown has not happened yet.

The candidate exists but has zero relevant entries. Print the observed request URLs and compare them with the url filter character for character. Scheme, host, port, path prefix, and query placement all matter to your pattern. The page can still pass in update mode because traffic outside the recording scope may continue to the real network. A server access log showing the API call alongside a validator count of zero points to capture selection, not an application failure.

The candidate contains login HTML. Inspect the final URL, response content type, status, and the beginning of the decoded body. A redirect can finish on a sign-in page with 200. The source account may have expired, the storage state may belong to another environment, or the API may require credentials that the refresh context did not load. Do not promote the login page and then change the UI assertion to match it.

The HAR stays byte-for-byte unchanged after a known contract change. Verify that the refresh used the candidate path you inspected. Then compare the source environment and tenant with the intended target. Check the live service log for the request. A stale environment, wrong port, too-narrow filter, Service Worker interception, or reading before context closure can all produce the same observation.

The diff changes hundreds of unrelated lines. Look at updateMode first. Full mode includes information that minimal replay does not need. Next inspect the URL scope and test data. Timestamps, trace IDs, random IDs, rotating cookies, response dates, and shared account state create churn. Stabilize the input or narrow the fixture. Do not normalize away a field that the browser genuinely uses to make a decision.

Replay fails with an aborted request after a clean refresh. In the trace Network view, compare the failing request's URL, method, POST body, and relevant headers with the candidate entry. A new query parameter or changed POST payload prevents strict matching. If notFound is abort, the browser commonly reports a failed request rather than an HTTP response. If notFound is fallback, the same miss can reach the live API and hide the fixture gap.

The server was called during a supposed replay. Search the routine test for update: true and notFound: 'fallback'. Check for a more specific page route registered after the HAR route, because route ordering can change which handler receives traffic. Also check Service Workers. Playwright documents that requests intercepted by a Service Worker are not served from the HAR and recommends blocking Service Workers when routing is required. A PWA test that must exercise its worker needs a different test boundary.

The response body is missing. With attached content, verify every companion resource or ZIP entry traveled with the HAR. A plain .har extracted from an archive may reference content stored relative to it. A clean-environment replay is valuable because a developer machine can have an untracked attachment that CI lacks.

Two refreshes produce alternating results. Check worker count, test retries, matrix jobs, and manual job concurrency. Every updating context writes when it closes. Two writers aimed at one path create a last-closer-wins race, even if both individual tests pass. One worker inside one job is not enough when two CI runs can overlap. Add job-level concurrency control and use a unique candidate artifact per run.

A practical failure report might read:

HAR validation failed:
- expected 1 captured entry, found 3
- entry 0: forbidden header authorization
- entry 1: unexpected origin https://identity.test.example
- entry 2: response status 503

That output describes a broad capture from an unhealthy authenticated journey. Regenerating it again without changing scope, credentials, or source health will only produce another unsafe candidate.

Roll the workflow out without creating a fixture factory

Start with an inventory rather than a bulk refresh. For every existing HAR, record the owning test, captured URL family, source environment, data identity, content storage choice, refresh command, and reviewer. Files with no owner should remain read-only until someone can state what behavior they preserve.

Move any test that toggles update from an environment variable into separate replay and refresh files. The replay file gets notFound: 'abort' for the in-scope URLs and never mentions update mode. The refresh file writes a candidate and requires a named guard. This duplicates a small amount of route setup, but it makes routine execution safe by inspection.

Introduce validation one fixture family at a time. Begin with expected entry count, origins, paths, methods, statuses, content types, and forbidden headers. Add response-shape checks for fields the UI consumes. Avoid a universal validator that assumes all HARs contain JSON GET requests. Upload workflows, redirects, GraphQL POSTs, and binary resources have different contracts.

Give every refresh run a unique output before promotion. A job ID or temporary artifact path prevents simultaneous runs from sharing a destination. Configure the refresh job with one worker and no retries. Apply CI concurrency at the workflow level so two manually triggered refreshes for the same fixture cannot overlap. This adds queue time, which is preferable to a fixture whose provenance depends on teardown order.

Sanitize before the candidate enters normal source control review. Use a dedicated low-privilege account and synthetic records. Search request and response headers, cookies, query strings, POST bodies, and response bodies. HAR is plain captured data. Minimal mode reduces unrelated fields but is not a redaction feature.

Require a replay proof after promotion. Ideally, make the captured API unavailable for the in-scope route, or assert a server-side hit counter in a controlled environment. A green replay while the source remains reachable does not prove the HAR served every request, especially when fallback is allowed.

Keep a small live contract test beside HAR replay. The HAR proves the UI handles a reviewed conversation. It cannot prove that today's deployed API still returns that conversation. Contract or integration coverage should report drift without rewriting the UI fixture. When drift is intentional, the owner decides whether to update the app, fixture, assertions, or all three.

Measure maintenance cost after the first few refreshes. If a three-entry fixture changes every week because request bodies contain timestamps, a hand-written route may be clearer. If a large multi-request journey changes twice a year and the payloads must remain internally consistent, a governed HAR can save substantial mock code. The right answer depends on stability and reviewability, not recording speed.

The main trade-offs are concrete. Candidate promotion adds an artifact and a review step. Minimal mode gives small replay-focused files but removes diagnostic timing and security fields. Full mode retains more evidence but creates larger diffs and a larger sensitive-data surface. Embedded bodies improve text review but bloat a single file. Attached bodies handle binary resources better but can go missing. Strict abort behavior exposes drift but increases maintenance when harmless requests are added.

Know when updating the HAR is the wrong response

Do not refresh because a replay test failed. First decide whether the changed request or response is intended. If the UI renamed customerId to customer_id by mistake, recording the broken request makes the regression permanent. If the backend removed a required field without coordination, a new HAR hides the contract break that the old fixture exposed.

Do not use HAR for one small response that a route handler can express in ten readable lines. A hand-written page.route() handler makes status, headers, and body obvious, and it is easier to create 400, 500, latency, or malformed JSON variants. HAR earns its maintenance cost when a coherent multi-request exchange is difficult to model manually.

Do not use replay to claim current backend coverage. The point of replay is to avoid the live dependency. A green test can show that this UI build works with captured traffic, not that the deployed service is healthy or compatible. Keep a live path for that claim.

Do not refresh highly dynamic POST conversations until you can stabilize the inputs. HAR matching includes POST data. A timestamp, nonce, generated ID, or signed payload can make every future request distinct. Replacing those values after capture may also invalidate a signature or change the behavior under test.

Do not treat a replay fixture as performance evidence. Minimal mode intentionally omits timing and size information that routing does not need. Replay also removes real server work and network distance. Use a live, purpose-built performance setup for latency, caching, compression, connection reuse, and capacity claims.

Do not capture a production journey merely because it has realistic data. HAR can contain authorization material, cookies, personal data, internal URLs, request bodies, and full responses. A sanitized synthetic environment is cheaper than investigating a credential or privacy leak in repository history.

Do not block Service Workers in a test whose purpose is to verify the Service Worker. Playwright routing cannot serve requests already intercepted by the worker. Move the mock below that boundary, control the worker's data source, or keep the scenario live. Changing the product behavior to make the test tool convenient invalidates the coverage.

Do not update all browser projects into the same path. Chromium, Firefox, and WebKit can produce different headers or request sequences, and parallel contexts can race at teardown. Choose one recording project for a shared protocol fixture, or keep clearly named project-specific fixtures when the difference is part of the contract.

Finally, do not schedule automatic refreshes that open pull requests with no human decision. A calendar can remind an owner to review a fixture, but elapsed time does not prove the captured backend state is correct. Refresh when there is a named contract change, a controlled data update, or a deliberate expansion of the covered journey. The cost of that discipline is slower fixture maintenance. The benefit is that a green test still means someone approved the behavior it replays.