Build Meaningful Visual Regression Tests with Playwright

Playwright's toHaveScreenshot() creates a reference image on the first run and compares subsequent renders with it. This comparison can catch changes to layout, color, typography, and responsive placement at once. It becomes untrustworthy, however, when data or execution environments drift, or when updated baselines are approved without review.

This guide uses apps/web/app/(tech)/visual.e2e.test.ts to explain what to verify, how to classify a failure, and when to update a reference image.

Verify user contracts before screenshots

A visual test contains three distinct contracts. Do not make one screenshot stand in for all of them.

Judge meaningfulness by the product risk a failure explains, not by the number of assertions.

• Make the test name reveal the relevant starting state, user action, and expected outcome
• Let one case protect one user risk, and group assertions only when they describe the same outcome
• Verify roles, accessible names, and visible results instead of CSS classes or DOM depth
• Keep the result unchanged when the test runs alone or in a different order by avoiding dependencies on another test's cookies, storage, or data
• Be able to explain which regression would become invisible if the test were deleted

Playwright best practices recommend testing user-visible behavior instead of implementation details and using locators and web-first assertions with automatic waiting and retries. The test isolation guide also explains that every test runs in an isolated browser context to prevent cascading failures and order dependencies. Let each test prepare its own preconditions, then assert that the page's essential meaning is ready immediately before taking a screenshot.

test("visual: overview-wide-light", async ({ page }) => {
  await page.setViewportSize({ width: 1440, height: 1000 });
  await page.addInitScript(() => {
    localStorage.setItem("tech-theme", "light");
  });
  await page.goto("/en");

  await expect(
    page.getByRole("heading", { level: 1, name: "Engineering" }),
  ).toBeVisible();
  await expect(
    page.getByRole("navigation", { name: "Editorial navigation" }),
  ).toBeVisible();

  await expect(page).toHaveScreenshot("overview-wide-light.png", {
    fullPage: true,
  });
});

• Treat a heading or navigation assertion failure as a content, accessibility, or routing problem
• Treat a screenshot assertion failure as an appearance or rendering-environment problem
• Separating the failures avoids guessing whether a feature works from a pixel diff alone

Select cases by risk instead of screen count

Capturing the product of every page and viewport increases storage and review cost without necessarily increasing confidence. Every case should explain the regression it catches in one sentence.

Require the following when adding a case.

• It represents a user risk that differs from existing cases
• Its viewport width clearly represents one side of a real breakpoint
• Its light, dark, and locale combination represents a distinct layout or token risk instead of every permutation
• Its name makes the intended route, screen size, and theme discoverable
• The regression-detection capability lost by deleting it can be explained

Use a full-page screenshot when navigation through footer form one product contract. Use a locator screenshot to reduce the diff surface when only an independent component or region matters.

const article = page.getByRole("article");

await expect(article).toBeVisible();
await expect(article).toHaveScreenshot("document-article.png");

Make inputs and rendering deterministic before capture

The official visual comparison guide warns that rendering can vary with the OS, browser version, settings, and hardware, and recommends verifying screenshots in the same environment that generated the baseline. Apply the same principle to page state.

1. Fix the environment before navigation

   Set values that affect the initial render, such as theme, locale cookies, reduced motion, and authentication state, before page.goto().

   await page.addInitScript(() => {
     localStorage.setItem("tech-theme", "dark");
   });
   await page.goto("/en");

2. Control time and data

   When dates or relative time appear on screen, fix time with page.clock.setFixedTime(). When API responses change the screen, return a small fixed fixture with page.route(), or use deterministic build-time data.

   await page.clock.setFixedTime(new Date("2026-08-28T00:00:00Z"));
   await page.route("**/api/articles", async (route) => {
     await route.fulfill({
       json: [{ id: "visual-contract", title: "Visual contract" }],
     });
   });

   Separate tests of real backend integration from visual fixture tests. Do not make the current state of an external API part of the visual baseline.

3. Wait for observable readiness

   Use web-first assertions on user-visible elements instead of fixed timeouts. When fonts and image dimensions affect layout, wait for those resources as well.

   await expect(page.getByRole("main")).toBeVisible();
   await page.evaluate(() => document.fonts.ready);
   
   await page.evaluate(async () => {
     await Promise.all(
       [...document.images].map((image) =>
         image.decode().catch(() => undefined),
       ),
     );
   });

   Avoid waitForTimeout() and arbitrary long delays because they increase runtime without proving readiness.

4. Remove only irrelevant volatility

   Disable animations and the caret in screenshot configuration. Control volatile regions that are not under test, such as ads or random avatars, with mask or stylePath.

   If a real timestamp, error state, or loading indicator is under test, fix its input instead of hiding it. Broad masks can conceal real regressions.

Manage tolerances as contracts, not noise controls

maxDiffPixelRatio, maxDiffPixels, and threshold control different concerns.

• Stabilize data, fonts, animations, and the execution environment first
• Measure repeated non-product differences before choosing the smallest tolerance
• Do not immediately increase a global tolerance just to make a failure pass
• Document the reason and removal condition next to an assertion that needs an exception
• Treat a different image height or moved primary component as a layout change, not a tolerance problem

The current Web configuration disables animations, hides carets, uses CSS pixel scale, and permits a maximum diff ratio of 0.001. This value is a boundary that produces a diff for human review, not an automatic approval rule.

Classify failures with expected, actual, and diff together

A visual failure can mean either a product bug or an invalid test environment. Narrow the cause in this order.

1. Confirm that behavior assertions and the URL passed first
2. Compare the width and height of expected and actual images
3. Determine whether the diff is localized to one component or spreads across the page
4. Confirm that fonts, images, dates, locale, and API data match
5. Inspect navigation, console, and network failures in the trace
6. Accept a baseline candidate only when the product change explains the diff

Playwright retries screenshot assertions until two consecutive captures match, then compares the last capture with the baseline. Therefore, captured a stable screenshot means that two browser captures matched each other; it does not prove that data and environment conform to the product contract.

The current trace: "on-first-retry" setting, failure screenshot, and video provide evidence for CI failures. Following the official CI debugging guidance, inspect the trace's DOM snapshots, action log, and network activity together.

Treat baselines as reviewed changes, not generated output

Updating a reference image approves a new UI; it does not fix a test. Narrow the update to a failing case instead of overwriting every baseline.

1. Run the target without updating it

   From the repository root, first confirm what the current baseline detects.

   pnpm --filter @jongminchung/web exec playwright test \
     visual.e2e.test.ts \
     --project=tech-chromium \
     --grep "visual: overview-wide-light"

2. Compare failure evidence with the product change

   Inspect *-expected.png, *-actual.png, *-diff.png, and the trace under apps/web/test-results. If regions outside the change request differ, do not update the baseline; fix the cause.

3. Update one case in the CI-equivalent environment

   Generate Linux CI baselines with the same Playwright and Chromium versions in a Linux environment.

   pnpm --filter @jongminchung/web exec playwright test \
     visual.e2e.test.ts \
     --project=tech-chromium \
     --grep "visual: overview-wide-light" \
     --update-snapshots=changed

   Do not replace a Linux CI baseline with a *-darwin.png generated on macOS.

4. Verify again without the update option

   Run the same command without --update-snapshots. Then run all Tech visual tests to confirm that one baseline update did not hide a regression in another screen.

   pnpm --filter @jongminchung/web exec playwright test \
     visual.e2e.test.ts \
     --project=tech-chromium

5. Review the baseline lifecycle in the Git diff

   git status --short -- \
     'apps/web/app/(tech)/visual.e2e.test.ts' \
     'apps/web/app/(tech)/visual.e2e.test.ts-snapshots'
   git diff --check

   Confirm that every added case has a baseline and that deleted or renamed cases leave no orphan image. Do not approve a PNG merely because it changed; record the meaning of its visual diff in review.

Make CI guarantee reproducibility and failure evidence

The Playwright CI guide installs browser and system dependencies before running tests and recommends worker 1 when CI stability takes priority. Maintain the following contracts for visual regression CI.

• Pin Playwright and Chromium versions with the lockfile
• Match the Linux baseline-generation and verification environments
• If parallel execution produces intermittent diffs, measure resource contention and reduce CI workers
• Preserve playwright-report and test-results as failure artifacts
• Track a visual test that passes only on retry as a flaky signal
• Review PNG changes and product code changes together in a pull request

Completion criteria

• Every screenshot case explains the user risk it protects
• Web-first assertions verify user behavior and essential structure before capture
• Dates, API data, theme, locale, fonts, and images reach deterministic states
• Full-page and locator screenshot scopes match their verification intent
• Tolerances and masks do not hide real product regressions
• Baselines are generated with the same OS and browser environment as CI
• Expected, actual, diff, and trace are reviewed before a baseline is updated
• Targeted and full tests pass without --update-snapshots after an update
• No required baseline is missing and no deleted case leaves an orphan image

Official documentation

• Writing tests
• Visual comparisons
• Best practices
• Isolation
• Locators
• Assertions
• toHaveScreenshot() API
• Clock
• Mock APIs
• Continuous Integration