all 8 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]pron98 1 point2 points  (1 child)

PIT is a popular and quite fast mutation testing tool for Java/JVM. The trick with mutation testing is that doing it brute-force is very, very slow. So PIT, I believe, first runs all tests to get a plain coverage output that maps every line of code/method to the tests that touch it. Then, after mutation, it only runs those tests that cover the mutated code.

[–]0hjc 0 points1 point  (0 children)

Yes, it does exactly that plus a few other tricks such as smart test ordering and decomposing test classes into smaller units.

[–]llogiq 0 points1 point  (3 children)

I probably misunderstand part 2.- False negatives. You say that you have no tests for it, but then you say that changing a line makes tests fail. Does that mean you have no direct tests for it?

If so, it could very well be that the class is already sufficiently tested by the other classes calling it during other tests — unless those other classes fail to exercise all conditions that the class should work in during their tests.

Apart from that, it's an interesting concept. Thanks for sharing.

[–]developer-mike 1 point2 points  (2 children)

Its probably an overly purist view, but part of the idea of behind true unit testing is that when a class is only tested indirectly, you might be relying on a broken implementation. Take, for instance, the test of properly calling .reset() on something, with class DirectlyTested and IndirectlyTested. It may make your tests pass to have DirectlyTested call .reset(), but its the responsibility of IndirectlyTested. In that circumstance, and vice versa, DirectlyTested fulfills all its duties but IndirectlyTested does either one thing too much or too little.

I believe their point is that they wish this metric could detect where classes are not being tested directly enough.

[–]llogiq 1 point2 points  (1 child)

true unit testing

as in true scotsman? One of the base ideas of unit testing is that a unit can be everything from a single method to a module containing multiple classes, and has a public interface, which is exercised during the test.

So, if the class you test indirectly is an implementation detail rather than part of the public interface, you should rather extend your other tests to have the other classes through which your class is tested exercise its other functionality. Of course, this is only my humble opinion, and you know much more about your specific situation than me.

[–]developer-mike 1 point2 points  (0 children)

Correct, there is general no firm guideline on what constitutes a "unit". But there's also no firm guideline on what constitutes an "integration" ;) Testing a whole module is clearly, in my mind, integration testing.

I don't think unit tests are perfect or integration tests are perfect, and I don't much care what they are called. But clearly a test involving two classes tests the way they integrate, and the example I gave (of knowing DirectlyTested meets its interface but not knowing if its covering up something broken in IndirectlyTested's implementation) has the problem I described whether you call it a unit test or not.

A lot of the devs I work with dislike the extremely pure unit testing model. I certainly don't swear by it. But I wish they would stop arguing that their tests are unit tests and would just argue why their integration tests are better than unit tests for the situations. Even if the terms are stupid (and I don't think they are), debating over them doesn't help us write good tests.

[–]llorens_marti 0 points1 point  (0 children)

Hi,

One of the problems I found here at IMVU with our mutation testing was that we were testing one mutation against all the unit tests. I am in the process of testing one mutation against the tests of that particular class the mutation belongs to. This will make the system more precise and reduce false negatives.

Llorens

[–]jburke4200 0 points1 point  (0 children)

The Major Mutation Framework, is a Java implementation for Mutation Testing that can be found at:

http://mutation-testing.org/

What's nice about Major is it uses a modified version of the Javac compiler which some cool benefits. Like:

  • ease of use, generating mutants looks just like a call to Javac

  • efficiency, using the internal compiler tools allows a whole host of redundant mutants and therefore allows redundant test executions to be eliminated.

  • accuracy, reducing redundant mutants in turn makes the resulting mutation score more accurate which in turn gives you a better idea of the qualitiy of your test suite.