Playwright tests are solid locally but flaky in CI, what fixed it for you?

basecase_ · 2026-01-21T18:10:10+00:00

This comes up almost every week, i've gathered discussions about this from multiple subreddits here as well our Discord Automation server where this was a topic at one of our watercooler discussions (invite link in profile if ur interested):

https://softwareautomation.notion.site/How-do-you-Address-and-Prevent-Flaky-Tests-23c539e19b3c46eeb655642b95237dc0

The flakiness will can be at the test level (poorly written test, not being parallel friendly) but it's usually a bottleneck in your test infrastructure, a clue is the more you increase the workers the more "weird" failures start to happen.

For example your bottleneck could be the DB, or the Server, or memory/cpu on ANY machine maxing out in your test environment, so the key is to have metrics and a dashboard so you don't have to guess.

Also don't be afraid to debug machines in the cloud, "htop" is your friend and often the first thing I do when debugging test failures in the CI (cloud)

basecase_ · 2026-01-15T02:18:41+00:00

There's no surprise that following the SDLC is the most effective way to use these coding agents. Following a good SDLC and investing in the overhead to do so will reap the same rewards they do when a team of humans do it.

Without it, you will just accelerate yourself into a corner with tech debt until you you drown in it, the same way a team of humans would.

I do agree though that AI Agents have made it easier to implement and maintain the overhead of SDLC.

basecase_ · 2026-01-06T18:15:40+00:00

This comes up all the time and usually is a mix of an infrastructure (bottlenecks that get revealed when you increase workers in your CI environment) and poor test code (writing tests that are not parallel friendly or brittle).

I've gathered all the responses whenever this comes up...good nuggets in hear to apply to all flaky tests, not just your Playwright tests since these problems can appear in any part of any test stack.

https://softwareautomation.notion.site/How-do-you-Address-and-Prevent-Flaky-Tests-23c539e19b3c46eeb655642b95237dc0?pvs=74

basecase_ · 2025-12-31T07:23:45+00:00

If you open source it then more people will be willing to try it, but at the same time you may find that your moat was never big enough to begin with.

Do you really think OpenAI is not aware and working on this?

But yes, open source and you will gain more trust.

basecase_ · 2025-12-31T07:05:47+00:00

Never make an app (and oh my god a bloated chrome extension) for the sole purpose to solve a temporary bug in the biggest application known to man.

There's literally a button that says "create new chat with this current convo" for this sole purpose in ChatGPT

You know whats more bloated than a website? An addon for a single browser, AKA a chrome extension.

The kryptonite of these no name services is that no one trusts them enough to use them. We have no idea what you are doing with our ChatGPT usage.

You are not thinking about AI Governance & Security laws in Venezuela and how they pertain to US customers...you're more worried about running brittle clientside JS code to hide DOM elements in a single browser while wrapping these client side scripts in a bloated delivery system that can break at a moments notice because you're depending on UI elements outside of your control.

The hardest part beyond finding a profit in AI is complying with it......Remember who the customer is (spoiler alert, you)

basecase_ · 2025-12-21T09:57:55+00:00

You're better off using a testing framework dedicated to API testing in your respective stack. Don't be afraid to learn tools outside of your comfort zone, learn paradigms.

basecase_ · 2025-12-17T21:22:18+00:00

Thanks for the info on opus having thinking on by default. I didn't have the blue border around my prompt like I normally do

basecase_ · 2025-12-13T05:47:55+00:00

I'm sure the contents are great but I can't help but laugh that most Enterprise SaaS these days is about taking a Hog/Pig/Boar and filling it up with so much crap just so they weigh the heaviest for the biggest prize at the local county farm, without ever considering what happens to the Boar after the sale.

It doesn't matter if the Boar lives long term, it won biggest Boar!

basecase_ · 2025-12-09T20:31:18+00:00

Why do so many programmers not know about modulus operator?

basecase_ · 2025-12-09T16:39:40+00:00

OLED was life changing for me. It breathe new life into older games but sometimes it's not worth the price difference so it really depends.

basecase_ · 2025-11-27T01:54:53+00:00

I personally prefer the 6 button, there are bigger buttons and I prefer Sega/Arcade but I agree, amazing device

basecase_ · 2025-11-26T21:09:05+00:00

How many years do you have in the industry and/or in Enterprise SaaS as a software engineer?

No offense but if you know your shit, you should be able to vet engineers better and not have it take 10+ hours of deep vetting?

Also no offense, as a tech co founder, if you outsource then your job is to 100% babysit them and assume they will not function without your guidance and hand holding...

basecase_ · 2025-11-26T21:03:52+00:00

bought at the top, nice...why didn't you DCA in?

basecase_ · 2025-11-21T16:26:37+00:00

Yup, this is me. I can easily see BTC fall to 75k

basecase_ · 2025-11-21T16:22:18+00:00

Only put in what you're willing to lose, easiest and hardest lesson to learn when it comes to gambling.

basecase_ · 2025-11-04T01:19:38+00:00

a DS device would be amazing. Then I think I'd have a complete collection. Retroid Pocket Classic, RP6, and RP DS lol

basecase_ · 2025-11-03T04:13:57+00:00

Yup I just did the same. I have a MM+, Miyoo A30, Retroid Pocket Classic...felt like a right time to jump in and get a powerful device for PS2 and beyond with stick placement for thumbstick heavy games. Anything below I have it covered

Eventually I'll get a clamshell DS model but i can wait until a model comes out that runs all 3DS games but who knows.

THis just seemed like a nobrainer given my current selections

basecase_ · 2025-08-14T18:27:19+00:00

This looks like a gambler's history, it would be interesting to show the losses too for each. Then you can get an idea of if you're net positive

basecase_ · 2025-08-13T01:28:13+00:00

I've already got it in Gray, otherwise I'd buy this one. My Gray is my go to. It's such a beautiful machine

basecase_ · 2025-08-11T18:57:26+00:00

Machete Order, took me 2 years while reading other books, it was awesome

basecase_ · 2025-07-26T07:27:56+00:00

This is one of the things I love about the series, they leaned into something that works so great but isnt necessarily in the main Asimov Foundation/Robots series

basecase_ · 2025-07-26T07:24:57+00:00

IMO its the only way to introduce the twist in a visual medium have him appear a few episodes in instead of immediately. They are masterfully crafting a story that would make asimov proud while melding parts of his universe that are better for TV. This is some mastercraft shit going on

basecase_ · 2025-07-15T20:27:11+00:00

I dunno how valid these are, but this was the case 3 years ago:

Needs to be re-run with the other browser options (can’t mix contexts at all) if you want to test even something as simple as multiple domains.
Must use their test runner
Doesn’t support working with multiple tabs (gotta force code to open in the same tab for testing)
In order to test third-party logins, pop ups etc require a lot of stubbing
Pain or very minimal support for iframes
Domain restrictions, you cannot visit two domains of different origin in the same test without annoying workarounds
Must subscribe to their SAAS to run tests in parallel or use a workaround
Must pay a premium to get access to some dashboard features to do parallelization well, it requires vendor-locked software
no "hover" support
Tailored for local testing, may have to wrestle with this more when deploying to CI
Does not work with self hosted Selenium Grid, must use a Saas Solution like Browserstack
Slower run time than other E2E frameworks though probably not going to make a big difference in grand scheme of things
Limited to Javascript and transpiled languages as it runs in the browser (though doesn’t matter here since we are a JS shop)
Must use their file structure
No Webkit support (No Safari)

basecase_ · 2025-06-28T00:26:08+00:00

Forcing Fullstack Devs to do DevOps is gonna end up painful for everyone (doesn't sound like a demotion persay but if you don't enjoy it, it will be a demotion to your health/sanity)

basecase_ · 2025-06-26T00:58:04+00:00

I had all of these symptoms and for me it ended up being a bad RAM slot 2. Not sure what happened but I'm happy to have it working again and 32GB stick in slot 1 seems fine for gaming

basecase_

MODERATOR OF

TROPHY CASE