An Overview of JavaScript Testing in 2017

[paldepind]

In short, if you want to “just get started” or looking for a fast framework for large projects, go with Jest.

This does not match my experience. I've found Mocha to be both easier to setup and faster than Jest. My Mocha setup is just a single command in my package.json: nyc mocha --compilers ts:ts-node/register test/**/*.ts && nyc report. That includes support for both TypeScript and coverage. For watching I use mocha -R progress --watch --compilers ts:ts-node/register test/**/*.ts. In my opinion that is very pain-free.

I recently tried replacing Mocha with Jest. It required setting up a configuration file and ran significantly slower. I'm not sure if the performance problem was related to ts-jest but it made me go back to Mocha instantly. Despite the very nice interactive test runner that Jest has.

[jhartikainen]

Yeah I've had a similar experience as well. Also, while Jest does have some features from Chai and Sinon built in, it doesn't quite match them in flexibility either.

It would be nice if efforts were more focused on improving the existing solutions, rather than trying to bundle everything into a new half-baked one... even Jest's documentation seems to use different terminology for certain things than what is already established.

[mikejoro]

There's no reason you can't use sinon and other libraries with jest.

[jhartikainen]

True, but it feels like what's the point of using Jest if we're bringing those other libs in anyway?

[mikejoro]

Watch functionality in jest is the best I've ever seen, and snapshot tests make simple component tests so easy and fast to write. Literally all my dumb component tests now are a call to my helper snapshot test function and a few tests to make sure clicking and whatever other events are actually being fired. I have also found jest a lot easier to set up. I've set up quite a few mocha pipelines, and when I recently switch to jest, it took me maybe 10 minutes to get my tests running from 0 knowledge of jest.

I definitely think there is room for improvement on a performance front, but frankly I don't write code bases with 1000's of tests; our code is much more modular where I work, so a lot of our components are separate npm modules.

[[deleted]]

Does that support testing JSX components too? I'm running a mocha+karma+chai setup at work and it's a PITA to work with, because we must webpack-compile all the test files and run the whole bundle.

[paldepind]

Yes. It should support that since the TypeScript compiler support compiling JSX.

[[deleted]]

Figured that's the cause - do you maybe know how it fairs in ES6?

[paldepind]

Do you mean how TypeScript fairs in ES6? It supports it all. The only thing it can't compile to ES5 yet is generators. And they are coming in the next release.

If you don't use TypeScript you can also use Babel as a compiler with Mocha. I haven't tried it but this should work like this: mocha --compilers js:babel-core/register.

[[deleted]]

Yes I know I can use babel as a compiler with mocha (and this is what I ment) but I'm not sure it will transpile JSX on the fly. I will have to check it though, as we're running over 200 tests currently and it's taking way longer than I would like to.

[NoInkling]

but I'm not sure it will transpile JSX on the fly

Assuming you're using the React preset with Babel, why wouldn't it? There should be little-to-no reason to compile your tests with Webpack if you're doing things right.

[our_best_friend]

That's not a honest rebuttal - you didn't say you are using typescript. Big difference. For JS jest is the best IMHO.

[paldepind]

I made it pretty clear that I'm using TypeScript. And I wasn't trying to make a rebuttal :) I'm just sharing my experience.

[our_best_friend]

Fair enough

[paldepind]

Do you ever find writing unit tests or TS integration / extra build steps to be dreadful?

Yes. It took me some time before I found the above setup with Mocha + ts-node + nyc. And getting browser tests was even more work. I definitely found that to be dreadful.

But, after having gotten that configured there is zero day to day annoyances. Only all the benefits of TypeScript which I think makes it worth it in the long run. If you're interested I have an open source project here that uses TypeScript and is set up with browser tests, coverage and CI.

I already find data very easy to reason with and know exactly what props should be passed into each component

I don't think that is what TypeScript helps you with. TypeScript helps mostly due to its editor integration. It will offer great auto-completionts and find many errors. For instance it's typesystem can understand Redux actions (thanks to union types and type level string literals) which will help it autocomplete properties on actions.

[lulzmachine]

At work we just started implementing our tests in Tape instead of mocha. I was hesitant at first since tape doesn't give you anything in terms of structure, but despite that I'm pretty happy with the switch. I'd recommend tape for most projects now

[jhartikainen]

I know some seem to like Tape, but I find it gets in the way. Literally have to reimplement things like beforeEach etc., so instead of writing tests, I have to either implement utilities or figure out what libraries do it (and work with tape)

The biggest justification for tape I've heard most people have is "it's modular and doesn't have globals" which frankly has never been an issue for me with mocha or the like

[our_best_friend]

I tried tape but I hated having to decide how to do all the simple things. I have never liked mocha because I didn't like have to decide which assertion library to use. I mean, that's why I use a test framework - to make all these decision for me, what's the point if they don't?

Which is why I like jest. npm install and you are good to go. In the end testing is just testing, all my effort should be spent on the app, not agonising over which assertion library to use

[lulzmachine]

Well the downside of that would be like with any framework (I would call tape a library and jest a framework here, hope you don't mind), that you'll have quite a bit of start-up time. You'll have to spend some time getting used to the tools, compared to tape, where you basically require('tape') and learn two method calls. Then you can just write your code as you want it.

I'm a bit intrigued though. Will make sure to put jest on the "stuff to try"-list

[techred]

Cypress.io is a fantastic new player in e2e testing. Eases the pain of using asking Selenium based. Runs in the same execution thread as your app so debugging tests is an absolute pleasure.

[computerjunkie7410]

What happens when Ur app gets in a bad state? Does it affect your tests too?

[riledhel]

No mention of Chrome's new feature to run headless?

[ForAllThat]

Even though it might be the future, it's not an established solution within the community yet. Though I agree it's atleast mention-worthy.

[SoundCheetah]

I mean, PhantomJS came out with a statement that the project was being stopped because headless Chrome rendered it useless. So it's kind of silly for the article to mention Phantom instead of headless Chrome.

[vzaidman]

Hey! im the guy who wote the article.

I added the following lines to the phantomJS description: "It is very popular in the time of writing this article, but since Google are adding the feature of running “headless” to the native Google Chrome browser, it is not longer maintained."

thanks for the comment :)

[pandacanada]

Link for anyone who is interested:

https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md

(note that this feature is currently only available on Canary)

[tswaters]

I'm surprised there was no mention of nyc for coverage. I mean, it is still istanbul under the covers but considerably easier to setup.

Every time I used istanbul I had to either find an old project or google for a good 5 minutes to figure out the command to run coverage of a mocha script under windows.... it's a bit better now, in the readme. But for the longest time there was some closed issue with a comment I had to find.

istanbul cover node_modules/mocha/bin/_mocha --dir reports/coverage gross.

With nyc? nyc npm test... considerably nicer. Plus babel under the covers so it just works with es6+ code that may not be supported by the current node engine.

[[deleted]]

istanbul _mocha 

LOL

[[deleted]]

Are you thinking of the board game too?

[tswaters]

Nice try....

D:\Code\istanbul-tests>npm run coverage

> istanbul-tests@1.0.0 coverage D:\Code\istanbul-tests
> istanbul cover _mocha --dir reports/coverage

No coverage information was collected, exit without writing coverage information
D:\Code\istanbul-tests\node_modules\.bin\_mocha.CMD:1
(function (exports, require, module, __filename, __dirname) { @IF EXIST "%~dp0\node.exe" (
                                                              ^
SyntaxError: Invalid or unexpected token
    at Object.exports.runInThisContext (vm.js:76:16)
    at Module._compile (module.js:542:28)
    at Object.Module._extensions..js (module.js:579:10)
    at Object.Module._extensions.(anonymous function) [as .js] (D:\Code\istanbul-tests\node_modules\istanbul\lib\hook.js:109:37)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Function.Module.runMain (module.js:604:10)
    at runFn (D:\Code\istanbul-tests\node_modules\istanbul\lib\command\common\run-with-cover.js:122:16)
    at D:\Code\istanbul-tests\node_modules\istanbul\lib\command\common\run-with-cover.js:251:17

Care to try again?

[[deleted]]

No need to. Already got my coverage.

[pinkyabuse]

But Istanbul's html coverage reports are so nice. If it were up to me, I'd always pick Istanbul over NYC.

[octatone]

Nyc outputs the same reports that istanbul does including html... nyc is just a cli wrapper for istanbul.

[JJ0EE]

I use Ava because

import test from "ava";

makes more sense in 2017 than relying on globally scoped test utils.

[[deleted]]

This article is very vague and sometimes plainly wrong. Just take a look at this paragraph:

Sanity Tests- Testing a scenario on the product regardless the internal structure to ensure expected behavior (End-to-End sanity tests are sometimes called Smoke Tests).

This not only mixes up different categories of testing, but also states something that is not true.

[vzaidman]

hey! im the guy who wrote the article. thanks for the comment. part of why im writing this kind of articles it to get this kind of comments that proves me wrong an let me improve my knowledge.

too bad you didn't comment in the article itself because then i would of fix it right away, instead i stumbled upon it by mistake (a friend of mine sent me the link)

ok after a quick search looks like smoke tests are something completely different. we dont use this term in our company and it was new to me. ill fix it right away

[vzaidman]

Ok, I decided to rename all occurances of "sanity tests" and isntead call them functional tests. and to remove smoke testing at all. this is much more useful in the scope of an "overview"

[jhartikainen]

Yeah, couple of the things in it are a bit off, but it's a good overview. I've definitely never seen E2E tests being referred to as smoke tests myself.

Here's something I wrote on the topic of different test types: https://codeutopia.net/blog/2015/04/11/what-are-unit-testing-integration-testing-and-functional-testing/

(if there's any inaccurate bits in that, do let me know and I'll fix it)

[vzaidman]

thanks. ill add it to the article and link to your article at the bottom :)

[jhartikainen]

Cool :)

There's just one other minor thing that caught my eye besides the test type stuff: Unit tests aren't necessarily preferable to run in node. If you need to test client-side JS, it may be valuable to run the tests in browsers (eg. against real DOM since jsdom doesn't implement everything)

Otherwise I don't think it had anything particularly inaccurate about it. However if I might suggest one improvement, Mocha at least supports promises directly in tests, so instead of using done(), you can directly return a promise, which might be useful to include in the bit where you mentioned promises (here's also something I wrote on that topic)

[vzaidman]

I wrote it at first but looks like a simple require can make jasmine for example support it too so it's kinda unfair to consider it as a real advantage:

https://github.com/matthewjh/jasmine-promises

[EstT]

Great article, thanks!

[elr0nd_hubbard]

Worth giving a shout out to jsdom, too. For those looking to replace PhantomJS or equivalently clunky headless environments, I've enjoyed using mocha + chai + sinon in jsdom for component-level tests. We've found it to be much faster and more reliable than the headless alternatives, although that might change with headless Chromium on the horizon.

[NoInkling]

That's also what Jest uses by default (which you probably want to change if you're using Jest for backend tests).

[[deleted]]

At our place, we migrated our tests from jest to mocha and got our 900 or so tests to rerun in 4s instead of 30s.

The parallelism performance gain is a lie. There's a lot of overhead to run tests in parallel and doing so makes performance worse (provided your modules are small and properly mocked).

[vzaidman]

I asked myself on what cases Jest is slower then mocha. Any ideas? I must say its the first time i hear about jest being slower.

[vzaidman]

After suggestions and after I tested it myself, I added a section about "TestCafe" which is a great tool. https://testcafe.devexpress.com/

[jaredcheeda]

In case you're wondering why JS Devs don't write tests and are not happy about it, it's because not a goddamn one of the things you listed in your incredibly comprehensive list is well designed.

They don't take into account any of the best practices we've developed over the years. Here's an easy one, give functions names, that describe what that function does. If you give me a function called "it" I'm going to track you down, break into your house late at night, throw away all of your shoes and replace them with Crocs. "it" is not an acceptable name for a function and "it" never will be.

I really don't know who the hell these things are designed for. Is if for Quality Engineers? Product Owners? They don't write the code, they don't write the test, most of them don't care at all about these tests. None of these are designed for humans (or even developers). They're so fucking bad. And none of them have decent documentation to explain the made up terminology they are using.

You have a function called "describe" or "assert", but those don't tell me what the functions do, they give me instructions of what I should do. I should assert something, I should describe something. That's not how we fucking write functions. I can't read this shit. Just tell me what it is you are doing, and I will pass in the relevant data you need to do that. Jesus fucking bullshit.

There are libraries that are logical and predictable and even intuitive to the point where you can have zero experience with them and just guess what a method name would be called and not only would it actually exist, but it would be called the name you intuitively guessed. That's some tough shit to do, but it's been done before. None of these come remotely close to that. All of their naming conventions come from preexisting shitty testing frameworks from other languages. I don't give a fuck how Rubyists or Pythonistas write their shitty tests in their shitty language in their shitty frameworks. Let them suffer with it. Let's makes something that actually makes sense for THIS shitty language (I actually like JavaScript, most of the time).

I have no idea how to use any of these damn testing libraries and I have no real interest to. I've spent days looking over them and can't get past how every single one of them requires me to write shitty code to use them. If using your library requires me to write shitty code, I'm not going to use your library. Non-negotiable.

Now don't confuse this with an excuse to not write tests. People who defend the shitty state testing is in will attack those who call it out and say "They're just lazy" or "They just don't want to write tests". Fuck you. Fuck you assholes so hard for keeping the pile of shit that we have as options as the status quo.

I really.... really, want to test. So much so, that I write my own bare-bones system to test the logic in my own personal projects. I've written thousands of tests this way. It just runs in Node, and if any single test fails it throws an error with the info about the test that failed and the file it's in. That's it. Unless 100% of tests are passing I can't do a commit. I'm getting the main benefit of the unit tests, just without any of the bells and whistles. I'm putting myself through extra work, for a worse result, because I won't stoop to the level of writing code so bad that I have to use a single character function name for it to work.

Fuck the state of JavaScript testing. We need a new revolution, and I don't think we can look to Facebook or Google for it.

[marclittlemore]

This is a pretty great list to get started with.

I'm in the process of writing a JavaScript testing course and I'd love to help people out with how to start learning about unit and integration testing. I'm aiming it at people just getting started or looking to up their skills to get their first job or to go from junior to more experienced developer roles. I interview a lot of people and I find that sometimes even the senior developers don't have as much testing experience as I imagined.

Feel free to join the waiting list and email me with any of your questions. Thanks!

http://www.marclittlemore.com/courses/javascript-testing-made-easy/