How to fix flaky Playwright tests: find the real cause, stop the retry roulette

Ask any team what their biggest automation problem is and the answer is almost always the same: flaky tests. A test that passes on retry gets waved through; a test that fails once a week gets ignored; and after a few months of that, a red pipeline stops meaning anything. The suite still runs, but nobody trusts it — which is functionally the same as not having one.

The uncomfortable part is that "flaky" isn't a diagnosis. It's a bucket teams throw failures into when they don't have the data to tell what actually happened. This post is about emptying that bucket: the four real causes behind most flakiness, how to fix each one in Playwright, and why the deciding factor is almost never the individual trace — it's cross-run history.

Retries hide the problem; history exposes it

Playwright's retries setting is a useful circuit breaker, but it's also how flakiness becomes invisible. A test that fails then passes is marked flaky in one run's report — and that report is gone when the next run overwrites it. Nobody is counting how often it happens, so nobody notices that a test has quietly gone from 1% flaky to 20% flaky over a quarter.

The first fix is therefore not a code fix at all: start accumulating results. The @testrelic/playwright-analytics reporter records every run — pass, fail, flaky, skipped — with navigation timelines, per-navigation network stats, and failure diagnostics, and uploads them to a per-repo history in the cloud platform. Once ten runs of the same test sit side by side in the run dashboard, the question "is this flaky or broken?" answers itself:

• Fails every run since Thursday → it's broken, and Thursday's deploy is your suspect.
• Fails ~15% of runs, always on the same navigation, always with a slow third-party request in the network stats → it's flaky, and you now know where.
• Fails only on one CI worker or one environment → it's neither the test nor the app; look at the infrastructure.

The four causes, and what actually fixes them

Nearly every flaky E2E test falls into one of these.

1. Timing and synchronization

The classic: the test asserts before the app is ready. Playwright's auto-waiting removes most of this, so when timing flakiness survives, it's usually one of the hard cases — an assertion racing a background fetch, an animation, or a websocket update. Fixes that hold up:

• Ban page.waitForTimeout() in review. A fixed sleep is either too short (flaky) or too long (slow) — it is never right.
• Assert on the signal, not the symptom: wait for the response (page.waitForResponse) or the resulting UI state (await expect(locator).toBeVisible()), not for time to pass.
• Use TestRelic's navigation timeline to see what the browser was actually doing at failure time. "The expect() fired 300ms before the /api/cart response landed" is a fix; "it fails sometimes" is not.

2. Test isolation

Tests that pass alone but fail in the suite share state: a logged-in session, a database row, a leaked localStorage key. Playwright gives you a fresh browser context per test — the leaks are almost always in your layer (shared accounts, shared fixtures, order-dependent setup). The tell in run history is a test whose failures correlate with which tests ran before it, not with any deploy. (State and data problems are big enough to get their own post.)

3. Environment and infrastructure

Underpowered CI runners, cold caches, rate-limited third parties, DNS hiccups. These failures look random per-run but cluster sharply in history: one worker pool, one time of day, one environment. TestRelic records CI metadata with every run, so you can group failures by branch, environment, or runner in the dashboard instead of guessing — and monitoring views for smoke/regression/nightly cadences make an environment-shaped failure pattern visible at a glance.

4. The app is actually flaky

The cause nobody wants: sometimes the test is faithfully reporting a real race condition in the product. Before deleting a "flaky" test, check whether its failure signature — same navigation, same API error in the captured network data — looks like something a user could hit. A flaky test that's flaky because checkout genuinely 500s under concurrency is the most valuable test you own.

Let the AI read the history

Classifying one test by hand is fine; classifying a 400-test suite isn't. This is where accumulated history pays off twice: Ask AI answers questions like "which tests were flaky this week, and what do their failures have in common?" directly from your run data, and the MCP server brings the same intelligence into your editor — its triage and healing capabilities draft the fix in context, with the failure history as evidence rather than a single trace as anecdote.

A workable order of operations

1. Instrument — add the reporter, upload runs, stop deleting your evidence.
2. Rank — after a week or two of history, list tests by flake rate. Fix the top five; ignore the long tail for now.
3. Diagnose per test — timeline + network stats + CI metadata → one of the four causes above.
4. Verify with data — a fix is proven when the flake rate drops in history, not when it passes three times on your laptop.
5. Hold the line — keep retries at 1, alert on flake-rate regressions through monitoring, and treat a newly flaky test like a newly failing one.

Flakiness never fully goes away — but "we have 3 flaky tests, we know why, and we're watching them" is a completely different place than "the suite is red again, rerun it."