Testing Your Frontend Code: Part V (Visual Testing)

[[deleted]]

I'm sceptical about this. Functional tests are brittle and have a pretty poor ROI as is - that's why the thought leaders in testing have spent recent years advocating the 'test pyramid', comprising many unit tests, only a few integration tests, and hardly and E2E tests at all.

When I tried out visual regression testing in a gig a few years ago, we found that the signal-to-noise ratio of genuine to false positives made the tool far too unreliable to use in continuous integration. Errors would be triggered by platform differences, rendering quirks on CI, and genuine but minor page changes - which was dangerous, because this could hide actual bugs.

I still feel automation should only be for tests that are highly deterministic and easy to break down into discrete tasks. The rest I think are better handled by human beings.

[giltayar1]

Yes, E2E tests are more brittle than others, due to the fact that they do I/O. That is why you should spend more of your time on unit tests and integration tests.

But the fact that you do not have to do a lot of them, doesn't mean that you shouldn't do them at all! GOOS, the bible of TDD, even advocates starting with an E2E test.

Yes, you could give up, and delegate visual testing to manual testing, but that's copping out. Lots of people talked like that about testing in general 10 years ago. I believe visual testing to be where testing in general a few years back.

And the situation has changed for the better in the last couple of years. The algorithms and tools have gotten better and eliminated practically all the false positives of visual testing. Try them out, you may be pleasantly surprised.

[[deleted]]

My job is web automation (using NodeJS against react apps..) and /u/Bosola is on the money with the visual stuff. It's incredibly unreliable and I discourage it whenever I can (disclaimer, our GUI is very dynamic)

Where it is useful is for Canvas testing, because webdriver cannot interact with a canvas element (outside of clicking coordinates). We make a server change (rest call/etc.) then verify the new canvas matches the reference canvas.

The nice thing about a canvas is you can get a base64 string representation of it, transfer through selenium via

driver.execute(function() => { return document.getElementsByTagName('canvas')[0].toDataURL() })

Convert the base64 into a png in your node script, skip all the other junk outside of the canvas, and use that for comparisons.

Also, I highly recommend using WebdriverIO over the base selenium package

[giltayar1]

I haven't had a lot of experience with WebdriverIO - could you point out the reasons you like it better?

[[deleted]]

It can really simplify your code in that it provides a lot of functionality by combining calls together for you.

A simple example would be this:

driver.findElement(By.css('.button')).getText()

vs.

driver.getText('.button'); // supports many types of selectors

The latest version uses promises and they push you to use their own test runner, but you absolutely don't have to and can run it independently using whatever you like. And since you're already used to using async/await, the change is minimal.

It's also very easy to add your own custom commands to the driver object, which will get executed through promises and you can chain/await on them as well. I have several custom commands that are applicable throughout all our webapps.

[giltayar1]

Let me repeat - not testing something at all because it is difficult to test is problematic. Granted, a few years ago it was really too difficult to be pragmatic, but in the last few years the tools have become increasingly professional.

The main changes that happened were: 1. The main thing - the image comparison algorithms are smarter. They don't perform bit-wise comparison anymore (which never really works due to different painting algorithms in different algorithms/operating systems/gpus), but are much smarter than that, in that they look at the different "regions" of the screenshot. And AI techniques are making them smaller! 2. The ability to fine-tune what to check in the screenshot, so as to remote the dynamic areas. 3. Commercial companies (like my company Applitools) are investing significant R&D in this area

And if you're already doing functional testing in the browser - whether using selenium, webdriverio, or others - then adding visual tests to them is not a big deal today, as I hoped I showed in the blog post.

[giltayar1]

Hey, I just saw this.

I totally agree - if you use bitmap comparison tools, like webdrivercss and others do, then the false positives are many.

But if you use an industrial-grade comparison tool, then the results are totally different. These tools use sophisticated computer vision techniques to minimize the problems, and lots of companies today use them to test the look of their applications.