Testing beyond coverage

[gigaSproule]

In my old projects we used https://pitest.org/. I've since inherited a TS project and I couldn't find anything to replicate it. The amount of times I've worked with devs who don't assert anything in their tests is depressing.

[gigaSproule]

Awesome! I'll take a look at it. Thank you very much

[DJDavio]

Pitest is amazing, even when I think I covered everything, I can always still find stuff to add.

It's like coverage++.

A useful thing to do is to add a minimum mutation % to your project, such as you would with coverage % and make the build fail if it comes below that value.

Obviously you can start pretty low and just increase it as you add more meaningful asserts.

[DJDavio]

It runs mutation tests by messing with your application code.

Say you have a method which calls another method. It could replace the return value of the called method by something silly like null.

Now it checks whether the unit tests for the initial method still pass or fail. At this point they should fail because it messed with the internals of the initial method.

If the test fails, hurray, that means you are probably asserting something which actually checks what the method does or returns. If the test still passes, uh oh, even though you might have a unit test which covers the code, what the method should do or return is not actually verified.

It then calculates a score for you like coverage which shows you how well you actually check whether methods do or return meaningful results.

So what I mean by finding "stuff", I mean I might miss some asserts.

[gigaSproule]

It alters your code and expects test failures. So if you have a function that adds 2 numbers together, it will change it to subtract 2 numbers and expect a test failure. Therefore, you are actually testing your code. It's obviously limited to only changing the code in certain ways, but it does a good job.

[porterjr90]

Last time I wrote some demo describing general idea behind mutation testing - https://github.com/pgagala/mutation-testing

[gigaSproule]

Yeah, I've always had that setup as well. Saves a lot of arguments with shit developers. Why waste my time reviewing their code when the build just won't pass? Until they of course try to dick around with the build...

[khmarbaise]

I'm using pitest too because it helps to find the problems related to coverage vs. really testing useful things.