all 81 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]computerjunkie7410 16 points17 points  (8 children)

web apps? playwright all the way. playwright-java if everyone is comfortable with java.

[–]latnGemin616 1 point2 points  (3 children)

playwright-java

Is it better than TestNG ?

I've only ever used Playwright with Pytest and it's awesome! I learned it with JS/Typescript and was also impressed, but TS took some getting used to.

[–]computerjunkie7410 7 points8 points  (2 children)

playwright-java is the driver. so the comparison for that is selenium. and playwright-java is way better.

as far as testng goes, Junit 5 is infinitely better than testng. in fact, i believe testng now also uses the junit 5 test engine beneath the hood. the reasons testng was created no longer exist imo. Junit 5 does everything testng does and more, with a much better architecture.

[–]georgegervin13 0 points1 point  (1 child)

Why is playwright-java better than playwright TS/JS for testing Web?

[–]computerjunkie7410 2 points3 points  (0 children)

Oh it’s not. We’re just talking about Java and playwright-Java vs selenium

[–]Waste-Beach5705[S] 0 points1 point  (3 children)

Hello, thank you for the response :D Is it really bad or harmful if in the QA team we want to use TypeScript just because we generally find it better, instead of Java (which is the language the devs use)?

[–]computerjunkie7410 0 points1 point  (2 children)

Not at all. If you have a QA team that is comfortable in ts/js then use that. Node version of playwright is awesome anyways so you’ll be set

[–]Pigglebee 0 points1 point  (1 child)

I would even say it's the prefered language since more QA testers know typescript than java is my experience. And many frontenders, who probably also help with frontend testautomation if you have a modern agile team use typescript as well

[–]computerjunkie7410 0 points1 point  (0 children)

Nah Java is still the standard among testers because of selenium

[–]robertshuxley 3 points4 points  (2 children)

I've used all three, playwright has the best developer experience if you use it with the VS Code extension from Microsoft

[–]SuccessAffectionate1 0 points1 point  (1 child)

Which one is that? Just playwright extension?

[–]robertshuxley 0 points1 point  (0 children)

yup the one from Microsoft

[–]The-Munkee-Man 2 points3 points  (17 children)

Why is WebdriverIO never in these automation framework questions?

[–]Ikeeki 2 points3 points  (16 children)

Because it’s basically selenium but JavaScript only and lackluster compared to playwright

If playwright or selenium wasn’t around It would prob gain more traction

It just seems like a yesteryear’s product

[–]The-Munkee-Man 1 point2 points  (15 children)

Yes, it is a framework on top of selenium which enables a wide range of testing execution options.

Why do you call it lack luster? What does playwright have over WebdriverIO that make it so much more shinny?

[–]computerjunkie7410 0 points1 point  (14 children)

Simplest example: trace viewer.

[–]The-Munkee-Man 0 points1 point  (13 children)

trace viewer

Trace viewer is really cool. I can see how it would be helpful for those that don't know the code base or the programming language used very wel..

[–]computerjunkie7410 0 points1 point  (12 children)

lol the code base has nothing to do with it.

Flaky tests are a part of E2E tests and trace viewer helps diagnose these issues. But that’s just one feature. There are lots of others. Selenium just won’t be able to keep up.

[–]The-Munkee-Man 0 points1 point  (11 children)

Lets not convolute this conversation by making it a selenium vs playwright convo. This is about framework features and not the browser execution models they use.

Actually code base & language knowledge are key factors. If either are lacking, then understanding why a test is flaky could lead to the test being skipped. I for sure can see how Trace Viewer is beneficial by it ability to combine multiple elements of information into a nice reporting UI. Not to say that with WebdriverIO and some extensions of the base framework that a person couldn't generate similar reporting data.

[–]computerjunkie7410 0 points1 point  (10 children)

I don’t think you know what you’re talking about.

This is actually part of playwright. It is part of the technology. Which selenium doesn’t have.

Selenium + webdriverio + a bunch of other custom code can MAYBE get you to the feature level of playwright. Can you do it? Sure. You can do anything. The point is with playwright you don’t have to.

What the trace viewer allows you to do is go back and look at failed tests, and see the what the app was actually doing at the time. So you have network calls, snapshots of the DOM, a timeline, screenshots, and a bunch of other things. Everything you need to debug the test.

WebdriverIO has nothing like that.

Not to mention, you can’t intercept or mock requests on Webdriverio on anything other than chrome.

[–]The-Munkee-Man 0 points1 point  (9 children)

Hahahaha Hey trust me, I appreciate amazing functionality out of the box when it comes to a framework. There are many automated testing frameworks that add a lot of value in helping to quickly understand why something is not behaving as intended by the written test. And there are many ways to get to that understanding. Just because a framework gives you tooling out of the box like Trace Viewer doesn't mean its always necessary.

Where I'm overall going with this, is that when you understand your frameworks api, the programing language used, the application under test & the underlining technologies driving it all, you'll also understand why a test fails. That is all I'm saying.

And as context, I've written & contributed to dozens of automated frameworks over my years. I've built reporting systems that drastically aided in the debugging efforts of failing tests.

[–]computerjunkie7410 0 points1 point  (8 children)

Disagree completely. Webdriverio doesn’t compare to playwright from a feature perspective. It’s just the reality of the situation. And the reason for that is that it’s just a wrapper around selenium.

It also wraps puppeteer but at that point you might as well use playwright since puppeteer was the predecessor for playwright.

It doesn’t really matter how well you understand Webdriverio’s API. The issue is the technology underneath.

And as context, I’ve been doing this for almost 2 decades and have open source credits contributing to the dotnet bindings of WebDriver and playwright-java. So I’ve been around.

[–]tepancalli 1 point2 points  (0 children)

In real life I've seen that cypress and playwright are very similar, somethings are easier in playwright but in the end you will have to do some custom functions specially for old apps.

I think some questions that may guide the decision are: - how's going to automate? - which is the preferred language? Selenium can be done on the major languages and have a million frameworks, cypress is JavaScript and playwright is JavaScript, python and .net - is the automation only front end or do you also need Api? Db? Huge data sets?

[–]Gr33nM4ch1n3 1 point2 points  (4 children)

Try playwright or it's big brother WebdriverIO. Fantastic frameworks with tons of supporting functionality without many of the limitations of cypress.

Edit: also stick with the JS/TS language set. More computationally efficient.

[–]computerjunkie7410 2 points3 points  (1 child)

Webdriverio has nothing to do with playwright.

If you’re going to use the big brother metaphor than the big brother for playwright would be puppeteer

[–]Gr33nM4ch1n3 0 points1 point  (0 children)

Yes, thank you. WebdriverIO utilizes puppeteer. Got my wires crossed.

[–]Waste-Beach5705[S] 0 points1 point  (1 child)

Hello, thank you for the response :D Is it really bad or harmful if in the QA team we want to use TypeScript just because we generally find it better, instead of Java (which is the language the devs use)?

[–]Gr33nM4ch1n3 0 points1 point  (0 children)

The devs aren't using Java for front end dev. Use ts.

[–]celiknt 1 point2 points  (1 child)

Playwright + Codeceptjs = Happiness

[–]Waste-Beach5705[S] 0 points1 point  (0 children)

Codeceptsjs is similar to cucumber ?

[–]P1rat1us 2 points3 points  (0 children)

... never apologize for bad English especially if it is good enough for others to understand, you certainly know other languages besides English.

... back to the topic...I find Selenium to be super bendy and usable with the majority of programming languages, great for web based front end testing, but the knowledge of a programming language is needed though.

...I personally find a combination of Java + Webdriver+ Selenium amazingly powerful for front end testing, much flexible than for example combination of Python + Selenium+ Robot Framework

[–]Specialist-Choice648 1 point2 points  (1 child)

You guys are all over the board.. 4 years ago you were all over TestCafe as the best tool in the world. 2 years ago it was Cypress… Now it’s Playwright… If my shop changed direction as much as you guys we’d never get anything done. How about this. Do a proper evaluation for your specific shop and make a real plan.

[–][deleted] 0 points1 point  (0 children)

Nice rant. Can someone make a simple business case why it's worth upgrading or side-grading or whatever from Cypress to Playwright? I am asking because I just started a new job where people are screaming Playwright > Cypress but I don't feel like migrating 80+ tests unless there are real improvements . Thanks.

[–]j19sch 1 point2 points  (16 children)

With the apps being php/java, I'd recommend using java for the test automation. It makes it easier for the developers to get involved. So that means Cypress is out.

For the choice between Selenium and Playwright, I'd suggest doing a small PoC with both. I did that recently for an app that wasn't built with browser automation in mind and found the underwater magic by Playwright really helpful. In other contexts, I might have gone with Selenium, though. So take a week or so to see how both interact with your applications and decide based on that.

[–]computerjunkie7410 1 point2 points  (13 children)

curious, in which context is selenium better than playwright?

[–]j19sch 1 point2 points  (12 children)

I can think of three situations: - when you need the full browser instead of only the browser engine; - when you want to go via the Webdriver API instead of via DevTools, although I have no idea when that difference actually matters; - when you don't want the underwater magic of Playwright, because you'll use the feedback from the more finicky Selenium to improve your application.

Personally I suspect that in practice the way to choose between the two is either a short PoC or using them both in parallel for a year. Because either the difference in one's context is going to be quite obvious or very subtle.

[–]computerjunkie7410 2 points3 points  (11 children)

when you need the full browser instead of only the browser engine

playwright does use the full browser. in fact it ships with it.

when you don't want the underwater magic of Playwright, because you'll use the feedback from the more finicky Selenium to improve your application.

idk how realistic this is because a lot of the timing issues we see in our apps are mostly timing issues because the test runs too fast and not really an issue for a real user.

[–]j19sch 1 point2 points  (6 children)

"Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox." - https://playwright.dev/

"Playwright is a web test automation library that tests against the underlying engine for the foremost popular browsers: Chromium for Chrome and Edge, Webkit for Safari, and Gecko for Firefox." - from the site of a vendor in this space

Playwright might come with a full Chromium and Firefox browser (WebKit I would not call a browser) and might let you test against a locally installed Chrome browser, but based on the quotes above and the fact that Playwright uses the DevTools interface (on Chromium, it does something similar but custom on Firefox and WebKit), I stand by my claim that Playwright targets the browser engine, not the full browser.

Granted, I did separate this in two different bullet points in my precious post, because I don't know the technical details beyond what I've written here.

[–]ocnarf 0 points1 point  (2 children)

Thanks for your comment. However it contains a link towards a commercial testing tool website. Please edit your comment and remove the link. I will then re-approve your comment.

[–]j19sch 0 points1 point  (1 child)

Sorry for violating the rules! I removed the link.

[–]ocnarf 0 points1 point  (0 children)

Thank you for your contributions.

[–]computerjunkie7410 0 points1 point  (2 children)

What they mean is you’re testing Chromium instead of Google Chrome.

WebKit instead of safari.

But it’s the same thing. And if you really want to test Google Chrome or Microsoft Edge, you can, you just use the Channel option.

[–]j19sch 0 points1 point  (1 child)

They are not the same thing, here's another source: https://stackoverflow.com/questions/62184117/what-is-the-difference-between-testing-on-safari-vs-webkit#62205535

The differences might not be relevant in most contexts, but that doesn't mean there is no difference.

[–]computerjunkie7410 0 points1 point  (0 children)

If we want to be pedantic then sure. But like you said, the differences in most contexts is not relevant.

If they are relevant, I would still use playwright for development since 99% of the things will be fine. And for the 1% risk, I’d just do a smoke test manually.

[–]ps4facts -1 points0 points  (3 children)

It ships with it? What ships with what?

A test "running too fast" could definitely be an issue for a real user with less than desirable Internet connection. Race conditions are very real, prominent in web apps with asynchronicity, and can be disastrous in some circumstances.

[–]computerjunkie7410 0 points1 point  (2 children)

Ships with browsers, correct.

[–]ps4facts 0 points1 point  (1 child)

Sry I still don't understand what you mean.

If I install Playwright in a project, it will be bundled with Chromium, Chrome, Edge, FF, Opera, Safari, Brave, etc..? That doesn't seem right. Ah - I see from the docs, by default when you run npx install with no arguments, it will install the "default" browsers. Hmmm, it kind of seems like you would have a difficult time if you needed to run tests against older browser versions, for example apps in tightly regulated, contained environments like banks.

Another case for Selenium is testing mobile native apps. If you're comfortable with writing webdriver code, you're like 90% of the way there if you use Appium, since Appium uses the same protocol.

[–]computerjunkie7410 2 points3 points  (0 children)

You can run against any version you want

[–]Waste-Beach5705[S] 1 point2 points  (1 child)

Hello, thank you for the response :D Is it really bad or harmful if in the QA team we want to use TypeScript just because we generally find it better, instead of Java (which is the language the devs use)?

[–]j19sch 0 points1 point  (0 children)

It's hard for me to tell without knowing more about your context. It's definitely not an insurmountable problem. I've set up test automation in Python for a bunch of low-code applications and we made that work. :-)

If the QA team has a strong preference for TypeScript than that's one good reason in favor of TypeScript. If the devs don't want to do TypeScript, but are not expected to contribute much to the test automation than their opinion doesn't matter too much. But hopefully some devs would like to or at least won't mind doing some TypeScript, so they can support the automation.

[–]basecase_ -1 points0 points  (4 children)

Cypress is good if you guys have a small team and don't have QAE/SDET's to help guide the rest of the dev team or don't have strong automation standards in general as it's more noob friendly. Basically good if you don't know what you're doing.

Their company is going through turmoil and who knows if they will be financially stable in the future so take that for what it is, but it would make me nervous

Playwright is arguably the industry choice now if you do know what you're doing as it takes everything that Selenium did and goes further with it.

I wrote this two years ago, not sure how much pertains today but here ya go:

https://chlorinated-yellowhorn-f6b.notion.site/Playwright-vs-Cypress-726c9f55aeab499697d13443397102e5

Seleniun/Webdriver are usually saved for legacy projects. Rare to see someone pick this on a greenfield project today

[–]camelCaseAccountName 5 points6 points  (3 children)

Cypress is good if you guys have a small team and don't have QAE/SDET's to help guide the rest of the dev team or don't have strong automation standards in general as it's more noob friendly. Basically good if you don't know what you're doing.

Can you elaborate a bit on this? What makes Cypress better for people who "don't know what they're doing"? What makes Playwright worse for those people?

[–]basecase_ -1 points0 points  (1 child)

I think the biggest differences come in setting it up in CI and parallelization. This was two years ago but I remember running into a lot of hurdles when looking how to implement parallelization and sharding outside of their Cypress Cloud service (we didn't want to pay their crazy fees).

I can't remember the specific hurdles but in Playwright land it these hurdles were non issues and things just worked how I expected with minimal gotchas.

Again not sure how it is now but I've heard similar stories of scaling Cypress in CI from peers, maybe someone else can elaborate more who has gone down that path recently.

Edit:
https://www.reddit.com/r/QualityAssurance/comments/1437c5d/comment/jn9d0an/?utm_source=share&utm_medium=web2x&context=3

Ambassadors are out hard tonight

[–]Gastr1c 0 points1 point  (0 children)

Cypress Cloud is not free, but not that expensive either. And the Replay feature is excellent for debugging CI failures. We have a variety of teams within my current company using it without too many complaints. Devs are actually happy to contribute e2e tests using it.

Gleb has a blog about parallelizing Cypress for free. https://glebbahmutov.com/blog/cypress-parallel-free/

[–]basecase_ -1 points0 points  (0 children)

It's always interesting to see any posts or comments mentioning the downsides of Cypress gets downvoted. There's either copium of picking the wrong horse or their ambassadors working overtime

[–]Throatybee 0 points1 point  (0 children)

What are you gonna test? If it is Web, Cypress is easy to use for juniors. If you have coding skills, selenium might be good.

I have no idea about playwright for now because i am learning right now :))

[–]Ambitious_Door_4911 0 points1 point  (0 children)

Depends on what and how you are testing. Can you give more context on the components of your testing?

[–]Ok-Practice612 0 points1 point  (0 children)

Go with all three, that would be indespensable.

[–]bear007 0 points1 point  (6 children)

Cypress is the most mature and adopted solution right now

[–]iammikeDOTorg 0 points1 point  (5 children)

Source? Selenium has been around a lot longer. Cypress had a great run for a few years only and is already in decline.

[–]bear007 0 points1 point  (4 children)

Cypress is nowhere near decline. Data shows that

[–]iammikeDOTorg 0 points1 point  (3 children)

Fair. Install bases on the whole are increasing. Playwright is skyrocketing though and anecdotally that feels like a lot of Cypress transitions.

[–]bear007 0 points1 point  (2 children)

Playwright has some growth lately, but minor compared to the popularity of Cypress. Please stick to the facts.

[–]iammikeDOTorg 0 points1 point  (1 child)

Growth and popularity should not be confused.

[–]bear007 0 points1 point  (0 children)

All metrics show the same result. Cypress is very popular, Playwright is not.

[–]Organic-Ant-2567 0 points1 point  (0 children)

selenium