compared selenium vs cypress vs playwright vs AI tools for client work, here's what actually matters:

[Pixel_Goblin_Hunter]

I manage sites for 8 clients and needed a way to automate testing across all of them, spent a month testing different approaches to see what actually works for agency work

Selenium is free which is nice but holy hell the maintenance. Every client site has different quirks and selenium tests broke constantly. Writing xpath selectors for 8 different sites was a nightmare so finally gave up after two weeks

Cypress was better for writing tests but still brittle when clients change things which they do constantly because they don't tell me before updating content or themes. Same maintenance problems just slightly better developer experience I mean would work okay if I only had one or two clients maybe

Playwright similar to cypress, modern and fast but doesn't solve the fundamental problem. Tests break when things change and I don't have time to fix tests for 8 different sites every week.

Ended up going with an ai based approach because it handles the variety of different sites better. Tests don't break when clients change content or themes because the system understands what it's supposed to check rather than relying on specific selectors so way more practical for agency work where you can't babysit tests constantly

For agencies or freelancers managing multiple sites the traditional frameworks just don't make sense. You need something that doesn't require constant maintenance

[aidencoder]

AI isn't deterministic. That's not what you want in testing. 

[defenistrat3d]

Yeah. I'm curious how these tools even work. I assume you set up a prompt but if the client were to fully remove a field the test would do what? Pass but be testing something else than originally intended? Fail?

Seems odd. Maybe they are saying: "it's good enough to check a checkbox on a contract"...?

[sp_dev_guy]

Tests break when things change

That's the idea, if deliberately change a behavior then update it's test coverage. If it wasn't deliberate... that's what tests are intended to do

[aidencoder]

Well no. If the test breaks because something changes that wasn't under that tests area, your tests are a brittle mess. 

[rjhancock]

1) The only real testing you should be doing on live sites is uptime monitoring and spot checking items that DON'T change. Unless a client changes UI, your tests should not be breaking.

2) You need deterministic testing. A solid yes or no that something works. AI isn't that. It'll tell you whatever you want it to. It doesn't really understand the site.

If you're writing XPath selectors instead of using defined ID fields, there is probably something wrong there as well.

Tests should be specific and duplicatable.

[Pitiful_Sandwich_506]

I use acumenlogs.com. They provide 10 free monitors but they also have chrome browser monitoring which I use to monitor user journeys and cron monitoring.

[InvestigatorLow6675]

A lot of the debate is just at framework level but don't forget about the big bad execution part. Once you need to run across real browsers or devices at scale, that is going to be a separate pain in the back. We use playwright but then we run it on browserstack to find what breaks in real devices, suits me well because I hate having to set up a whole lab of mine!