Changelog

A test marked test.fail() that fails is a pass to Playwright, but the report and the terminal summary showed it as a red failure, and showed one that unexpectedly passed as green. They now agree with Playwright.

• An expected failure reads as a pass, with an "expected to fail" badge. Its error is still shown, under "Expected failure".
• A test.fail() that passed is a failure that reads "Expected to fail, but passed."
• The counts explain themselves. The verdict band and the terminal summary add a line such as "1 failed as expected (test.fail)", so it's clear why their counts differ from report.json. summary.json, report.json and what's uploaded to the cloud count exactly as before.
• A missing baseline shows up in the report. With updateSnapshots: 'none', the mode most CI runs use, a comparison whose baseline doesn't exist failed the test but didn't appear in the report, so you couldn't see which snapshot was missing. It's now listed by name, with its verdict.
• No more boxes around nothing. The report no longer draws a highlight for an element with no width or height, matching the cloud and Studio.

Released as @testrelic/playwright-analytics 2.16.3 and 2.16.4, on @testrelic/core 2.9.3. Report schema moves to 1.9.0: the streaming report's index rows and per-test meta.json now carry expectedStatus whenever it isn't 'passed'. The addition is optional.

When the report directory sat on a different filesystem from where artifacts were written (a Docker volume mount, a RAM disk, or a Windows temp folder on another drive), screenshots, videos and other artifacts could go missing from the report, leaving only a "Failed to move file" line behind. They're now copied across as intended.

Fixed in @testrelic/playwright-analytics 2.16.3, @testrelic/appium-analytics 1.2.2 and @testrelic/maestro-analytics 1.4.2. The latest releases are 2.16.4, 1.2.3 and 1.4.3.

A first run of a visual suite records nothing it can compare — Playwright writes the baselines and reports each test as failed. 2.16.0 listed every one of those assertions twice, one of them green. This release makes the verdicts true and gives a reviewer the path to act on them.

• One assertion, one record, the right word. toMatchVisualBaseline reads the committed file to decide what happened: a first write is new, a rewrite under --update-snapshots is updated (and the rewritten baseline is attached, so the report has a picture of it), and a match is passed. Two assertions resolving at once can no longer report each other's verdict. (Under updateSnapshots: 'none' a missing baseline fails the test and Playwright attaches nothing for it, so the report does not list that comparison — unchanged here.)
• A native toHaveScreenshot failure stays visible even when it shares a name with a toMatchVisualBaseline pass in the same test.
• Each comparison records where its committed baseline lives — baselinePath, relative to your Playwright config's directory — so tr visual accept promotes a render into the file Playwright actually reads, not the per-run copy in the report. (The platform records the decision; only the CLI, which has your checkout, writes the file.)
• Merged shard reports keep their comparisons and their images. A merged report used to claim comparisons nothing could show.
• The report uses the platform's words — Wipe, Pixel diff, This run — so a comparison reads the same in the HTML report, in Studio and in the cloud.

Released as @testrelic/playwright-analytics 2.16.2 and @testrelic/core 2.9.1. Report schema moves to 1.7.0; the addition is optional.

A pixel diff can tell you 80,858 pixels moved. It cannot tell you which card gained a margin. This release captures a DOM snapshot alongside every screenshot and diffs it against a committed baseline, so the report names the elements responsible.

• Visual comparisons appear in the report. If your suite already uses toHaveScreenshot() or toMatchSnapshot(), upgrade the reporter and they show up — baseline, actual and diff, viewable side by side, through a wipe slider, as an onion-skin blend, or as Playwright's own diff image, with the differing-pixel count and the exact ratio. No import to add, no assertion to rewrite. TestRelic does not compare images itself; Playwright already ships the comparator, so this adds nothing to your dependency tree.
• toMatchVisualBaseline records comparisons that passed. Native toHaveScreenshot attaches nothing when images match, which makes a green visual check invisible to any reporter. The new matcher runs the same comparator with the same options and additionally records the baseline it asserted against.
• DOM comparison names what changed. Element geometry, a curated set of computed styles, visible text and identity attributes are captured, diffed, and reported as seven kinds of change — added, removed, moved, resized, restyled, text, and attrs. Hovering a row in the new Elements tab draws that element's box over the screenshot, so the semantic change and the pixels it caused are connected. Needs Playwright 1.51 or later; everything else in the reporter still runs on 1.35.
• Knock-on movement is separated from the actual edit. One card gaining a 24px margin shifts everything below it. A flat list would report eleven equal findings for one change, so elements that only moved or resized are grouped beneath the element that did something.
• DOM baselines are text, and belong in git. A <name>.dom.json sits beside each image baseline, so a restyle arrives in a pull request as margin-top: 24px in a readable diff rather than an opaque binary change. --update-snapshots rewrites both files together, and the mask option you already pass excludes regions from the DOM comparison as well as the pixel one.
• The run overview leads with what broke. A verdict band opens the page, and a Needs attention block promotes failures to the top with their cause, their evidence and the pixel diff itself. Test rows carry evidence chips saying what was captured and a duration bar relative to the slowest test on screen. The test detail is now a page with its own URL and history entry, so a link to a failing test opens that test.
• API assertions are captured again. The documented setup — importing test from /fixture and expect from /api-fixture — passed every assertion through uncaptured. Our own example suite reported 0 assertions; it now reports 126. There is a new Assertions section on the detail page, failures first, each with its expression, expected and received values, source line, and the call it was made against.
• Media that was already on disk is reachable. Screenshots and video in a report opened from disk now load and play. Traces now have a tab of their own, with the npx playwright show-trace invocation.

Released as @testrelic/playwright-analytics 2.16.0 and @testrelic/core 2.9.0. Report schema moves to 1.6.0 — every addition is optional.

See Visual Testing for the full guide, including how to keep a visual suite from going flaky.

• Projects running trace: 'on' wrote trace.zip to disk but never uploaded it — only screenshots and video were collected. Fixed hardest exactly where it mattered most: a beforeEach failure has no screenshot worth having, and a pure timeout previously reached the cloud with no artifact at all.
• Traces are now collected, classified, and uploaded alongside screenshots and video.

Released as @testrelic/playwright-analytics 2.13.0. Note for existing users: every project already running trace: 'on' will start uploading trace archives — they're the largest of the three artifact kinds (~900KB in verified runs).

The API session workspace's Cookies view now shows the request and response cookies captured for each API call, with sensitive values automatically masked before they ever leave your machine.