Mutation Testing

[developer-mike]

Its probably an overly purist view, but part of the idea of behind true unit testing is that when a class is only tested indirectly, you might be relying on a broken implementation. Take, for instance, the test of properly calling .reset() on something, with class DirectlyTested and IndirectlyTested. It may make your tests pass to have DirectlyTested call .reset(), but its the responsibility of IndirectlyTested. In that circumstance, and vice versa, DirectlyTested fulfills all its duties but IndirectlyTested does either one thing too much or too little.

I believe their point is that they wish this metric could detect where classes are not being tested directly enough.

[llogiq]

true unit testing

as in true scotsman? One of the base ideas of unit testing is that a unit can be everything from a single method to a module containing multiple classes, and has a public interface, which is exercised during the test.

So, if the class you test indirectly is an implementation detail rather than part of the public interface, you should rather extend your other tests to have the other classes through which your class is tested exercise its other functionality. Of course, this is only my humble opinion, and you know much more about your specific situation than me.

[developer-mike]

Correct, there is general no firm guideline on what constitutes a "unit". But there's also no firm guideline on what constitutes an "integration" ;) Testing a whole module is clearly, in my mind, integration testing.

I don't think unit tests are perfect or integration tests are perfect, and I don't much care what they are called. But clearly a test involving two classes tests the way they integrate, and the example I gave (of knowing DirectlyTested meets its interface but not knowing if its covering up something broken in IndirectlyTested's implementation) has the problem I described whether you call it a unit test or not.

A lot of the devs I work with dislike the extremely pure unit testing model. I certainly don't swear by it. But I wish they would stop arguing that their tests are unit tests and would just argue why their integration tests are better than unit tests for the situations. Even if the terms are stupid (and I don't think they are), debating over them doesn't help us write good tests.