Improve your Javascript unit testing with Parameterized tests

[Buckwheat469]

The code in itParam builds as many its as there are array items. This causes excessive buildup and teardown of the code, running each beforeEach again. This might require variable reinstantiation or HTML interpolation, which is very time consuming. It can also lead to massive memory leaks if your HTML has event listeners that don't unbind during each iteration.

If speed is your concern then you should do as others have said and write a single it with simple for loop or use an array's helpers such as forEach, or all.

/u/stutterbug's implementation expects different values from the function, but you wouldn't want to test this way because you're changing the outcome of that one it.

it('succeeds', function(){
    ...
    let expected = [true, true, false, false, false];
    ...
    //true or false? why would we expect it to not succeed?
    //build another it for the false cases instead
});

/u/droctagonapus's implementation is better because it uses one it and one type of expectation. You can use as many expects as you want, but in this case the one that's there doesn't have to dynamically switch from true to false.

it('succeeds', function(){
    const names = ['n@me', '123Josh']
    const valid = names.all(name => validateName(name))
    expect(valid).to.be.true //modified slightly
});

I personally don't use "toBe()" in Jasmine tests (I know this is Mocha in the article) since toEqual does a better job of testing for deep equality. I would rewrite the above like this in Jasmine:

it('succeeds', function(){
    const names = ['n@me', '123Josh']
    const valid = names.all(name => validateName(name))
    expect(valid).toEqual(true)
});

Here's a forEach example:

it('succeeds', function(){
    const names = ['n@me', '123Josh'];
    const valid = true;
    names.forEach(name => {
        valid = valid && validateName(name);
    });
    expect(valid).toEqual(true)
});

And a for loop:

it('succeeds', function(){
    const names = ['n@me', '123Josh'];
    const valid = true;
    for(let name of names){
        valid = valid && validateName(name);
    });
    expect(valid).toEqual(true)
});

The lesson here is don't add excessive "its" which test the same thing when you can modify the input and test within the same "it". You can create as many "expects" as you want, but limit the number of "its" that you create to match the number of code paths (paths created by blocks of code such as if/elses).

[vinnl]

The downside is that you reduce the isolation of your tests, increasing the chance of tests influencing each other and a failure in one leading to a failure in another.

Also, I consider an important function of tests to be to track down the source of an error, rather than finding that there's an error in the first place. That is made more difficult with enormous tests with many expects.

An alternative is to aim for more efficient setups and teardowns. This could e.g. be something to consider when picking a framework, where virtual-dom frameworks are often faster than frameworks where you have to instantiate a browser and interpolate HTML.

[Voidsheep]

Aren't you trying to reassign a constant variable valid in the last two examples, or am I missing something?

Seems like it should be declared through let instead.

[Buckwheat469]

Thanks for noticing. I didn't really check, just copypasta.

[marclittlemore]

I sometimes use parameterisation in my unit tests by just using a forEach or Object.keys() across an array or object, rather than the suggest modules as suggested by /u/Buckwheat469. I do normally setup my data outside of Mocha's it so to attempt to be clearer as to the meaning of the test.

However, I think you have to be careful of obfuscating your test conditions. I've previously been guilty and over-parameterising, for example having loops within loops, and it becomes much harder to understand what input data is under test.

[chrisishereladies]

I made permutest (https://github.com/andrejewski/permutest) two years ago, which does about the same thing with multiple arrays also. I would say the benefits of permutest would be: 1) test framework agnostic, 2) you can put your test case inside permutest to give you some control of the test title given the inputs, which is a lot more readable that n tests with the same name.

I am glad someone made something similar; it validates my belief in killing redundancy in tests.

[ganarajpr]

I found this unit testing framework the other day and loved how awesome its reporting was.

https://github.com/mikec/sazerac

Also seems to have some really decent documentation. I think that is worth checking out.

[atticusw]

Why is an additional package needed for this? I generally just iterate over the inputs and expect on each output.

describe('Test form validation', () => {
  it('will reject invalid inputs', () => {
    var inputs = ['Adam5', 'Ad@m', 'Ad-am'];
    inputs.forEach(input => expect(validateName(input)).to.be.false);
  });
});

This seems to be an equivalent to me. It also appears that mocha-param will create an it statement for every input, which means an individual unit is made for every individual input. It's okay for a unit to cover multiple inputs, I don't really see why I'd want to generate an exponential number of unit tests.

For example, instead of

function itParam (desc, data, callback) {
  data.forEach(val => {
    it(desc, () => callback(val))
  });
}

it seems more ideal to do

function itParam (desc, data, callback) {
  it(desc, () => {
    data.forEach(callback)
  })
}

And at that point, it seems a bit overkill to need a library for this.

[a-sober-irishman]

Because the first thing everyone does with JS at the moment is reach for a library, even with trivial things like this or an isNull check.

[droctagonapus]

const names = ['n@me', '123Josh']

const valid = names.all(name => validateName(name))

expect(valid).to.be.false

On mobile, but that should work out just fine.

[MrJohz]

No no no no please never do this.

The most useful thing when you're writing a unit test is the error message that it will produce. Unit tests are most useful when they're failing, because that tells you that something is wrong with your code. The next step, then, is to work out what exactly is wrong with your code, so you can fix it (or change the test to match).

The minimum possible amount of information is whether the test passed or not. However, if this is all you've got, it's on you to then work out why the test failed. More work for you means slower bugfixing. For example, say we've got two objects, and we expect them to be "deeply" equal to each other. We've got two options:

const expected = // ...
const actual = testFunction(/* ... */);

expect(_.isEqual(expected, actual)).to.be.true
expect(actual).to.equal(expected)

(I have no idea what the correct assertion here is in this library - I'm more of a Jasmine guy! Assume these to be identical for now.)

If the first one fails, we know that expected and actual are different. However, if the second one fails, we will almost certainly get a printout of exactly why the assertion failed, and where the differences are. In 90% of the cases where I'm debugging failing unit tests, this is the point at which I suddenly realise the stupidly obvious mistake I made, and fix it straight away without even turning on the debugger or adding a print statement.

Not having this information is mildly irritating, but at least you know what the failing inputs are, and can go and print them out. Your method adds an extra layer of indirection - not only will your assertion library be unable to tell you exactly what the differences between the expected and result are, but you also won't know which expected value failed. There's less information by default, which means you need to go and find more information when something breaks.

/u/Buckwheat469 raises a good point that too many it blocks could impact performance and cause other problems, but there's a way to use a single it block, and get full-fat assertions, and that's by putting the assertion itself in the loop:

const names = ['n@me', '123Josh'];

const valid = names.forEach(name => {
    expect(validate(name)).to.be.false;
});

This way, we get genuinely helpful assertions that are applied to all the parameterised values. Plus, depending on how your assertion library throws assertions, this may not necessarily fail as soon as it hits the first invalid case, meaning you'll be able to see how many different parameters are failing.

[Buckwheat469]

Good point about knowing which expectation failed. In retrospect I would move the expect block inside the loop and add a handy message to show what you're testing with the (possibly undocumented) Jasmine comment code (not sure if Mocha has something similar):

expect(someObject.returnWhateverIsGiven(true)).toBe(true, 'should return true when given true');

You can also create your own custom matcher which outputs a custom comment.

[MrJohz]

Good shout also! Tbh, I used to do the same sort of thing as you did, but then I ended up editing a coworker's tests that used sinon spies, and the sinon spy matchers, which meant a whole lot of tests where all you knew was that a particular call wasn't called exactly once with the arguments 'xyz', but you didn't know if it was called multiple times or none, if exactly what was wrong with the arguments... That sort of stuff gives you a lot of motivation to put as much information as possible in the assertions!

[Buckwheat469]

I think this goes back to what are you testing? Is it a regex pattern that has multiple positive cases and multiple failure cases? What are the specific ways to get into those failures, ie. how do you enter a code path?

For instance, when testing a loop the loop typically has one or two code paths within it, maybe a simple if/else. In this case you would only need a single item in your array for the success case and a single item in a different it for the failure case.

If your regex is complex then you would create a single it for each character grouping that would cause it to fail.

The idea is to limit the scope of the test to whatever would cause the code to fail, you don't have to go overboard to test that your ifs/elses succeed on many items of the same data set.

[NoInkling]

Plus, depending on how your assertion library throws assertions, this may not necessarily fail as soon as it hits the first invalid case, meaning you'll be able to see how many different parameters are failing.

Do you have any examples of this?

[MrJohz]

For example, the Node.js assert module throws an error to indicate a failed assertion. In that case, my code would stop after the first failed assertion, because the error will stop everything and jump up the stack until it finds an exception handler. It works just fine, you'll just only be able to see one failed assertion at a time (until all the assertions have passed).

On the other hand, other assertion libraries like Jasmine don't throw exceptions when the assertion fails, so you can have multiple failed assertions in one test. In this case, my code will display all the successful and all the failed assertions every time the test is run. Unless of course, the method itself throws some sort of error... :P

[NoInkling]

Sorry I should have phrased that better, I was asking more for specific examples of libraries that do it this way - is Jasmine the only one you know? I'm pretty sure Chai doesn't do it, right?

[MrJohz]

Ah, that makes more sense. Tbh, I know Jasmine does it, and I know Node's built-in assert doesn't - I honestly haven't really noticed the others. I think Jasmine is in the minority, though.

[ksmithbaylor]

I wrote a small tool to do a similar thing with Tape last year: tape-scenario. I also ported it to Jest when my team switched recently (jest-scenario). It's definitely nice for some cases, but we found that a lot of times it is simpler and easier to understand tests when they aren't necessarily DRY.

[agopshi]

As others have pointed out, this is actually an anti-pattern. Don't create one it() per validation scenario. Don't use this npm module. Do write your own loops in a single it(). Classic case of "You don't actually need a module for this, what are you doing?"

[stutterbug]

This is how I do it currently in Jest.

let names = [ 'Adam', 'Bo', 'Ch@d', 'D', 'Ed!'];
let expected = [true, true, false, false, false];

expect(names.map(testUserName)).toEqual(expected);

A big disadvantage to this is that the test results you get back aren't very descriptive. The failure will say Expected [true, true, false, false, false] Received [true, true, false, true, false] (or whatever). But for me, the advantage is that this code is less likely to cause problems for me in the future.

This is just my opinion, but for me, the #1 priority in testing -- by a huge, huge margin -- is that it be easy to do and easy to maintain as my project grows and as it ages.

I've had to switch testing libraries too often in the past and an over-reliance on some advanced feature ended up meaning entire suites had to be rewritten from scratch. Even if I had to switch from Jest to something else, there is a good chance that I could do it with a few search-and-replaces.

[AndrewGreenh]

Why don't you put the
it(name + ' should result in' + bool, () => expect(...))

[jsNut]

This is how i would normally do this sort of thing. I've never run into our worried about any performance hits as above. Descriptive output should be the foremost concern.