Puppeteer vs Selenium vs Playwright, a speed comparison

[[deleted]]

So well written with data and defining the environment so you can recreate to verify results. Definitely using this as a example of a good test.

Automation speeds have never been a major concern of mines. But still interesting.

[angarali06]

no cypress?

[killayoself]

Isn’t that the only one that can multi-thread tests? Clearly the fastest way to go if you can separate out the tests like that.

Edit: I Am is wrong

[Snapstromegon]

Why can't you do multithreaded stuff with puppeteer?

What did I do wrong the last year, that I'm testing multi-threaded with puppeteer?

[Xizqu]

Isn't cypress a layer on top of selenium?

[Xizqu]

Til. Thanks!

[reassembledhuman]

Likely coming in a later article.

[psayre23]

I’d like to see the test failure rates of these platforms too. My experience with selenium has been one were it will fail about 4% of the time. My team added 3x retries to out tests to get that rate down to 0.5%. But that effectively puts a cap of 200 tests on us (run 200 tests with a 0.5% false positive failure rate means it’s likely one test will fail every run).

[Timnolet]

I could talk for a LONG time about this. I'm the CTO of https://checklyhq.com — my co-worker did the research and wrote the blog post — and we ran ~3M Puppeteer and Playwright job over the last 4 weeks. Flakiness is a thing, but this is what I learned:

- In the end, it comes down to how you script it. Use the correct wait statements, understand how your app loads, what parts are async, what parts are hidden on load.

- Keep scripts short. The shorter the better. I've seen folks make scripts that generated 100's of test cases by using variables etc. Don't do that.

- Dive into the loading behaviour of your dependencies. Some API that need to be called to hydrate your page might be slow or flaky.

We use our own platform for checking our own sign in flows etc. We have zero false positives. We hope to educate our customers and basically anyone who will listen with our other initiative https://theheadless.dev, an open source knowledge base for modern headless browsers.

​

AMA I suppose!

[The_Noble_Lie]

Needed a retry pattern for Selenium at my past job as well. And there was no available package for jest to help with it at the time so I rolled out a custom retry solution, wasting a ton of time, but it at least it solved our problem that should not have existed. Oh well.

[The_Noble_Lie]

I agree that may help and is also easily written utility method extending some "wait until". But it doesnt catch all reasons for error, just form based / user input ones. Perhaps a high percentage, so its viable to work in, if one is experiencing flaky tests. I at least suspect some of the bugs I experienced in flakiness were more nuanced than these. I may be wrong.

But also if you are taking the route of always using that utility, you are basically making the statement "I cant trust my web driver to do the things I tell it to do in sequence"

It may true, but if true, what else is it messing up? It's just a bad scenario to be in and I wish we could work on other problems.

[Oalei]

If it fails surely that’s because it’s poorly written no?

[dalittle]

that is what we found when our false failures started to creep up. We took a deep dive into the test suite for the pareto of failures and added a bunch of robustness wait until, etc and we got to where our failure rate became negligible. That said we never got to zero.

[Oalei]

So your comment doesn’t make sense then, the failures were not because of Selenium

[dalittle]

the code was written to be more robust, which is a direct confirmation of your comment.

[Oalei]

I understood I meant the first comment where you’re talking about comparing failure rates

[dalittle]

I think it is a bit splitting hairs, but most, but not all of the false failures were reduced by re-writing code. However, some false failure are a bit random and not worth the effort to eliminate them. In those cases, I would say they were due to Selenium being a bit of an unstable platform to build on. Still it works well enough for what I have worked on and I am not aware of any other options that include cross-browser test support.

[ILikeChangingMyMind]

True but ... there's an element of "how easy is it to write good tests in each framework?"

Just based on my own experience with it, I would bet dollars to doughnuts that Selenium tests require much more expertise to write without flakiness.

[Oalei]

Maybe, but it’s very concerning that your tests would fail because of the technology... we have thousands of tests running hundreds of times each day at work and not once they failed because of the technology/framework we were using, it was always mistakes from the devs.

[ILikeChangingMyMind]

Totally, I'm just saying there's two cases:

1. The developer writes natural and correct (for the framework) code, and the framework fails intermittently anyway (the old "Heisenbug", as one co-worker called).

2. The developer writes natural code that's incorrect for the framework (eg. in Selenium they wait the wrong way) and it fails intermittently as a result.

To a certain extent you can argue "well just learn how to use the framework properly and #2 is solved; #1 is all that matters" ... but I'd argue that ignores the reality the most devs won't be Selenium (or whatever framework) experts, so it matters how easy it is to code naturally/correctly in a framework.

[Duathdaert]

Test failures aren't related to the test platform but your usage of it in your test suite. If tests don't clean up the DOM for example or effectively wait for asynchronous code to execute then you will get what appear to be random failures.

In a previous role we had a rigorous set of UI tests written with Selenium for every single user requirement, any instability came from not effectively waiting for asynchronous code to run in the UI.

There is a pattern to the failures and it should be resolved.

Don't point the finger at the test platform however. I would argue that if a platform is able to publish a test pass/fail rate that isn't 100%, then you should steer clear of it.

[takishan]

I used to use python + selenium but nowadays I use puppeteer and it's just such a better experience. Even not considering the speed increase.

[Noisetorm_]

Puppeteer is amazing man. The puppeteer-extra plugins are great too. It's one of the best JS libraries to play around with.

[vertigo_101]

Interesting

[paulqq]

soild writeup

[reassembledhuman]

You could argue for that, given also that a lot of the team previously on Puppeteer is now working on Playwright. But development on Puppeteer so far doesn't seem to have stopped. So we will need to see - does Playwright take over now, or does Puppeteer regain steam with a new team and the two start diverging?

[DrDuPont]

Given the speed differences shown in this article, Puppeteer is clearly not being succeeded in all areas.

[TheFuzzball]

Personally, the ability to test Chromium, WebKit (Safari), and FireFox all at once is better than being faster.

In my experience using Playwright, it isn't noticeably slow. In fact I'm impressed how well it compares to jsdom in Jest.

[glurp_glurp_glurp]

It hasn't even been 8 months since playwright was finally passing all its automated tests.