Most teams we talk to already have some Playwright tests. They also have a backlog of flows nobody has gotten around to covering, a CI job that fails one run in five for reasons nobody can explain, and a Slack channel where screenshot diffs go to be ignored. AI tooling gets pitched as the answer to all three.
Some of that pitch holds up. Language models are good at producing the boring parts of a test and at reading large volumes of failure output. They're bad at knowing what your application is supposed to do, and that's the part that makes a test worth having. Here's where we've found the line sits.
Generating test scaffolds
The highest-value use is drafting. Give a model the page's accessibility tree, a short description of the user flow and two or three of your existing tests as style examples, and it will write a spec that uses getByRole and getByLabel locators, follows your fixture conventions and gets the navigation steps mostly right. Playwright's MCP server and the planner, generator and healer agent definitions shipped in recent Playwright releases make this loop easier because the model can drive a real browser while it writes.
What the draft won't have is good assertions. Models tend to assert what they can see on the page right now: a heading is visible, a button exists. That proves the page loaded. It doesn't prove the order total includes tax or that a cancelled subscription stops billing. An engineer who knows the domain still has to decide what the test is for and write those checks.
Our rule is simple. AI writes the steps, a person writes or approves every expect(). Reviewing a generated test takes a few minutes; writing it from scratch takes much longer. That saving is real and it compounds across a backlog of fifty uncovered flows.
Self-healing locators: useful, with a catch
Self-healing means that when a locator stops matching, a tool finds the element that probably replaced it and carries on. It sounds like pure upside. The problem is that a locator failing is sometimes the bug. If the Submit button disappears because a feature flag broke, a healer that clicks the nearest similar button turns a red test green and the regression ships.
| Approach | What happens on a broken locator | Risk |
|---|---|---|
| No healing | Test fails, engineer investigates | Time spent on trivial renames |
| Heal at runtime, silently | Test passes with a guessed element | Real regressions masked |
| Heal at runtime, flag in report | Test passes, warning raised | Warnings get ignored |
| Propose fix as a pull request | Test fails, fix suggested for review | Slower, but nothing hidden |
We use the last option. The run fails, a model reads the trace and the current DOM, and it opens a pull request that updates the locator with a short explanation. A person merges it or rejects it. You keep most of the time saving and none of the blind spots.
The better fix is upstream. Tests built on roles, labels and visible text break far less often than tests built on CSS classes or XPath. If healing is firing every week, your locators are the problem, and adding test ids to unstable components will do more than any tool.
Triage for visual diffs and flaky tests
Playwright's toHaveScreenshot comparison is precise, which is exactly why teams stop looking at it. A font rendering change or a date in the footer produces dozens of diffs, and nobody wants to open forty images to find the one real layout break.
A vision model is a reasonable first filter. Pass it the baseline, the actual image and the diff, and ask it to classify the change: content only, minor rendering, or layout change, with a one-line reason. Sort the review queue by that label. Don't let it approve baselines automatically. Its job is to put the five diffs that matter at the top.
Flaky test analysis is a similar job. Configure retries in CI and Playwright will report a test that failed and then passed as flaky. Record traces on the first retry so you have evidence. Then feed a model the failing trace, the error and the passing trace, and ask what differs. It's often good at spotting the usual suspects:
- A click that happens before a network request settles, where an expect on the resulting state would wait properly.
- Tests sharing a user account and changing each other's data when run in parallel.
- Assertions on timestamps, random ids or sort orders that aren't stable.
- Hard-coded waitForTimeout calls that are long enough locally and too short on a slower CI runner.
Treat the model's diagnosis as a hypothesis. It reads traces quickly, but it will also produce a confident explanation for a failure it doesn't understand. Confirm by reproducing with repeat-each before you change anything.
CI setup that keeps the suite honest
None of the AI tooling helps if the pipeline itself is shaky. The basics we put in place first:
- Run in the official Playwright Docker image pinned to the same version as your package, so browsers and fonts match between local and CI. That alone removes a lot of screenshot noise.
- Shard across machines once the suite passes ten minutes, and merge the blob reports into one HTML report.
- Keep retries at one or two in CI and zero locally. More retries than that just hide flakiness.
- Save traces, videos and screenshots on failure as build artifacts, and keep them for a couple of weeks so trends are visible.
- Run a fast smoke subset on every pull request and the full suite on merge and nightly.
Test data is still your problem
AI won't fix test data, and test data causes more flakiness than locators do. Each test should create what it needs through an API call or a database seed, rather than relying on a shared fixture account that drifts over time. Use Playwright's storageState to log in once per worker instead of filling the login form in every test. Give parallel workers separate users so they can't collide.
Models are handy for generating realistic input variety, such as names with diacritics, long German addresses or Japanese text, which is useful for catching layout and encoding bugs. Generate those sets once, review them, and commit them as fixtures. Don't call a model during the test run. It adds latency and cost, and it makes the test non-deterministic.
Measuring whether it's worth it
Count of tests is a poor metric, and AI makes it worse because generated tests are cheap. Track things that relate to outcomes instead: escaped defects found in production for flows that have coverage, mean time from a red build to a diagnosis, the flaky rate per week, and how long the suite takes on a pull request.
Take a baseline for a month before introducing AI tooling, then compare. If generated tests raise coverage but the escaped defect rate doesn't move, they're asserting the wrong things. If triage time falls but flakiness stays flat, you're getting faster at reading failures without fixing their causes. Both are useful to know, and neither shows up in a test count.
Count the costs too. Model usage for drafting and triage is usually small next to engineering salaries, but review time for generated tests isn't, and a suite that doubles in size also doubles CI minutes. A smaller suite of tests that each guard something important beats a large one that nobody reads when it fails.
Questions we get asked
- Can AI write Playwright tests for me?
- It can draft the navigation steps and locators well, especially with access to a live browser. A person who knows the product should still write or approve the assertions.
- Are self-healing tests a good idea?
- Silent healing at runtime can mask real regressions. Letting the test fail and having a tool propose a locator fix for review keeps most of the benefit without that risk.
- How do I reduce flaky Playwright tests?
- Use role-based locators, rely on auto-waiting assertions instead of fixed timeouts, isolate test data per worker, pin the browser version in CI, and record traces on retry to diagnose failures.
- Should tests call an LLM at runtime?
- Generally no. It adds cost, latency and non-determinism. Use models to generate fixtures or analyze results outside the run, and commit what they produce.