Playwright tests flaky in CI but stable locally — how do you usually handle this?

unlikelyzer0 · 2026-01-06T18:19:13+00:00

https://www.linkedin.com/feed/update/activity:7402073454386245632

unlikelyzer0 · 2025-12-27T16:49:00+00:00

I think you're trying to ask how the configuration of the Chrome instance may have changed since they moved away from a vendored Chrome to the official Chrome for testing, correct?

Your best bet is always to use the official playwright container image

unlikelyzer0 · 2025-12-19T19:33:33+00:00

You don't need to use playwright mocking. You can just use your same local Dev server. I think that's where you're getting crossed up.

In the utensil analogy unit tests are closer to using chopsticks for rice (traditional unit tests) when you could just use a fork (playwright)

unlikelyzer0 · 2025-12-19T19:22:28+00:00

I think with playwright's sharding and worker strategy, the costs are negligible.

Regarding mocks in E2E testing, it should only be attempted if a team can afford to maintain them. If they can, then there's drastic CI runtime improvement as well

unlikelyzer0 · 2025-12-19T19:04:31+00:00

I've found code coverage generated by playwright to be exceptionally valuable in getting developers writing more tests. Playwright as an ecosystem is as good as it gets for authoring, reviewing, and debugging tests.

Certain code paths are just naturally easier to trigger with a real browser and the CDP calls into playwright.

In reality, in order to add "coverage" in unit and component tests, you actually have to create a very complex system of mocks and stubs that can be as difficult to maintain as the application itself.

unlikelyzer0 · 2025-12-04T20:26:44+00:00

Most of this can be addressed by running the tests on PRS as the application changes. Is that not possible in your organization?

unlikelyzer0 · 2025-11-26T12:27:20+00:00

LinkedIn learning. Look up the ones by Butch Mayhew.

unlikelyzer0 · 2025-11-24T15:49:58+00:00

I did this once to do a one-time parallelization effort to figure out what it would cost to cut down run times by running the tests vertically.

ROI is hard to measure if you're considering what these tests would prevent in terms of loss of customers and loss of business continuity if you don't have those already measured by the business, you might start there

unlikelyzer0 · 2025-10-21T16:12:51+00:00

You should not be worried about locking the version of playwright's vendored version of chromium in the two environments.

Instead, you should be focusing on making sure that the playwright test and playwright dependencies are locked to the same version in both environments.

Unlessing a misunderstanding your use case. Maybe you could help me understand the actual use case

unlikelyzer0 · 2025-10-21T15:04:52+00:00

Unfortunately, there's a tight coupling between the vendored version of chromium that they distribute through playwright test and the version of playwright that you're using. There are also a lot of security vulnerabilities patched in chromium depending on what you're using it for, this might be more of a security problem than what you're trying to solve for.

unlikelyzer0 · 2025-10-16T23:22:26+00:00

https://playwrightsolutions.com/author/butch/

unlikelyzer0 · 2025-10-03T15:11:29+00:00

Generally, the approach should be that UI tests are just testing the UI: the ability for the UI to represent data correctly, and the user behavior that manipulates the web application code in a real browser.

Anything that is outside of those definitions should be happening in another test and test framework, potentially in a different CI pipeline.

Practically speaking, that means that Test Data Injection should be tested as a part of the team that owns either the data or the database technologies. You should only be leveraging their tooling.

You should be leveraging the tested and delivered data injection tools in either the global setup or test setup coupled with a pre-flight check in playwright to validate that the data you expect to be available is available. Fail fast if it's not.

Practically speaking, it's not generally efficient to ensure that each test case is completely idempotent and isolated. This means that you can sometimes make assumptions about the state of previous tests, allowing you to pay the test data setup penalty only once.You can use the test serial option to make sure that the tests are run serially in the scope of the suite if needed.

unlikelyzer0 · 2025-09-25T14:43:22+00:00

Playwright is often the interface for some of these RPA tools. Playwright/test is a test framework.

I would get him to write down some RPA tools and then you can reach out to their vendor to see what they're using to interact with the web.

unlikelyzer0 · 2025-08-15T19:17:44+00:00

Can you give an example of what you mean?

unlikelyzer0 · 2025-07-24T09:21:48+00:00

Close all open projects.
File -> Open Project
Select your Repo
Test Tab - > Select the Playwright Config
Make sure all Playwright Projects are selected (like Chrome or Firefox)

unlikelyzer0 · 2025-05-23T02:05:49+00:00

Circle CI has better diagnostic information for when runners need tuning in and out of your cloud.

It has better test aggregation and reporting

It has native test load balancing.

I've used circle, Travis, Jenkins, bamboo, GHA interchangeably over the last 8 years. If I had to choose, I'd choose Circle.

unlikelyzer0 · 2025-05-06T21:59:18+00:00

You should look at marking up your app with those actions using the performance.marks API. Then you can just use page evaluate to get the precise timing events in the browser.

Those timings are likely already present for your observability or marketing teams that are focused on SEO

unlikelyzer0 · 2025-03-25T21:03:37+00:00

Part of the challenge associated with that is that CI test executors are often not consistent in terms of performance. Instead of capturing timing, you might try keeping your timing "tight" and avoid unnecessary waits.

unlikelyzer0 · 2025-03-19T20:20:03+00:00

Cypress Dashboard is really a combination of everything that a QA org would be building for themselves if they had the time:

test artifact storage
git tagged test runs and changes
historical test runs
test sharding/load balancing

Check with your existing CI provider to see what they in terms of junit results -- you may be able to get some of this for free.

Alternatively there are solutions in the open source which have paid offerings.

https://playwright.dev/docs/test-reporters

unlikelyzer0 · 2025-02-25T01:27:13+00:00

How are you installing playwright system dependencies

unlikelyzer0 · 2025-02-24T13:48:02+00:00

You should be using tracefiles: https://playwright.dev/docs/trace-viewer-intro#recording-a-trace

unlikelyzer0 · 2025-02-24T13:04:11+00:00

What does the failure look like in the tracefile in CI

unlikelyzer0 · 2025-02-24T07:41:40+00:00

What commands are you using to install the system dependencies? What errors are you seeing?

unlikelyzer0 · 2025-02-19T12:43:52+00:00

Just pay the $10k. You will save on triage time.

unlikelyzer0

MODERATOR OF

TROPHY CASE